ARMO

With the digital world evolving nonstop, keeping your data secure must be one of your top priorities at all times. Shauli Rozen of ARMO explains how they help end-users and developers address their Kubernetes security needs through Kubescape.

In this episode, he explains how their software applies security remediation without breaking the application, bringing DevOps and security together. Shauli also shares the various projects they studied and gathered ideas from in developing Kubescape and how it helps solve the ever-transforming security issues of the open-source industry.

---

In this episode, I'm super interested in speaking to Shauli Rozen who is the CEO and Cofounder of ARMO. Shauli, welcome to the show.

Thank you for having me.

You have a pretty interesting background and the shared connections on LinkedIn are an interesting bunch too. Give us a bit of background of the very potted history of your career and how it got to founding ARMO.

I'm a software engineer by profession. I used to write compression algorithms and stuff like that in the days. I don't know if people remember that before the smartphones you needed to transcode those different stuff. I even have a patent on my name for compressing to different sizes. I did my transition into the business world.

I did my MBA at Wharton in the United States and walked into consulting for something good for a while but eventually, I came back to the region. I came back to technology. I came back to Israel and walked in a few startups. I finally met Ben who is my Cofounder for ARMO. He’s a brilliant guy. He’s much more brilliant than I am. He’s a CTO. He had a great idea and that's how everything started.

Can you give us a very brief background of what ARMO is and what problems it helps you solve?

ARMO in a nutshell is all about solving your Kubernetes security needs. What we believe and what we see in the market is that Kubernetes, specifically Kubernetes security, is a bit of a different animal in many aspects. One is in the aspects of the complexity of the system, the number of microservices that are running in the system, and the number of alerts that it creates.

Noise, false positives, and a lot of things you need to manage become very problematic. We have set a target for ourselves to create contextual Kubernetes security. We are taking your Kubernetes context and workload context and connecting them to your runtime context to pinpoint the top things that you need to manage and to create mediation advice for you based on what's running in your cluster.

What we have seen is that the biggest concern security and DevOps teams have is the ability to apply security remediation without breaking the application. That's a very big point for us. What we have created is using that context to create what we call the Delta remediate mechanism, which you can apply our remediation advice without breaking your application.

Where did the idea for the project come from originally?

It all started with the notion that Kubernetes security needs to be different in many cases and several cases. One of them is the prioritization. Another thing is that Kubernetes security is much more of a cooperation between DevOps and security than other places in the organization and other places in security.

We decided to create a tool that started as an open source that will be welcomed and loved by DevOps and also create a lot of value for security. We launched in late 2021 when we launched Kubescape in our attempt to bridge those two gaps between security and DevOps. It was loved by DevOps. It got 2,000 stars on GitHub, probably the quickest of any project.

We have 9,000 stars in GitHub. Hundreds of thousands of DevOps and security users are using it. It all started with that. Back in the day, it wasn't about prioritization to be very honest. It wasn't about this new technology that we created of remediating without breaking the application and tricking your attack surface in an automated way.

Back then, it was mostly about combining and bridging between DevOps and security. Once it started ramping, we got so many users and people using the platform. All these new levels of security concerns of prioritization and hardening advice came about and that's what we have been focused on for the last few years.

There are quite a few entities going on here that I want to dig into. You said the sequencing of them because I find it interesting. Kubescape as I understand it is a pure open source CNCF sandbox project. It has an APACHE II license. Can you talk through how Kubescape as an entity relates to ARMO and then also the sequencing of incorporating the business starting the Kubescape projects and how those two relate to each other?

Kubescape is independent from ARMO. ARMO is the main maintainer and contributor to the project but we have over 100 contributors. Only 30 of them are ARMO employees. Kubescape is a tool, a project, or a component that you can either install in your cluster as a Kubernetes operator or use as a CLI to scan clusters, images, and IAC.

‍

It's a very advanced scanner that will use runtime information and scanning information and give you all of the results that are available in the cluster, CRDs, or the standard output of the files. DevOps uses it quite intensively in the CICDs and clusters to do ongoing monitoring, take the data out to Grafana, and combine the data into the CICDs to create policies. That's the main usage of Kubescape.

ARMO is an implementation of a back end that can take all the data that Kubescape generates, create a control plane across clusters and different functionalities, and create more insights. It’s things like smart remediation with the ability to cross-reference all of the things that go in your cluster via graph databases and combine them to create insights into the most important issues. Also, identify attack vectors in the organization. These are the things that you will only find in the ARMO platform. It’s a SaaS management portal on top of all of the different Kubescapes that are deployed in different places in your organization.

In terms of the genesis of those entities and the business itself, which came first?

Kubescape came first. We knew that our business model was going to be to create a SaaS platform on top of Kubescape but we first developed Kubescape. Three months later, we released the first version of the ARMO platform. Ever since then, both have been developed quite significantly.

In terms of the business itself, was that process all wrapped up in raising money and building a commercial business? What was the genesis of Kubescape?

We first created Kubescape. Once it was out, it was led by Tiger Global and Hyperwise. Hyperwise is an Israeli found that got into the company right after we launched the open source and they provided us great value. What we did was raise money to build monetization and platform capabilities on top of Kubescape.

The sequencing is very different but the overall model seems very similar to how OpenTelemetry works with regard to folks like Dynatrace, AppDynamics, Jaeger, and projects like that. Did you have a set of projects or businesses that had given you ideas about how a model could work?

We looked at all of the companies that you mentioned. Another company that we looked at as a reference is Redis. Another company that is in the same area is HashiCorp. We looked at all of the different models and we had a few to choose from. One was to open source everything and have an enterprise version. The Redis model I would call that, we decided not to go that way. Eventually, we decided to go the way of SaaS monetization. The paid version is a SaaS distribution on top of the open source.

Can you talk a little bit more about why that decision was made because you prefer that model over there? There were technical considerations around that.

It was mostly around speaking with users and thinking about what would be most valuable for them and how they would like to see it. It's a little bit of a tricky situation when we speak with users about how you are going to monetize because sometimes the interests are conflicting. We had very trusted users and people who understood where we wanted to go. They know it's a commercial company. We also consulted a lot. I spoke with the Redis CEO a lot because he is Israeli. I went to him and we spoke a lot about what he's learning. We talked a lot about different learnings and that's the way we do the conclusion.

In terms of inducting the Kubescape projects into the CNCF, how did that happen?

That's a big choice. We were engaged with the CNCF way before Kubescape was contributed and even before it was generated. We appreciate the CNCF quite a bit. We feel like it's an organization that genuinely tries to push and encourage open source as a platform for the future growth of organizations. We decided to contribute mainly to the practices that CNCF enforces and the recognition that CNCF brings to you as an organization.

If you think about the usage of open source, one of the main concerns users may have is whether this open source will stay alive. Does it have the right cadence? Is the governance okay? Do they maintain security standards? One of the things that CNCF forces you to do is to check all of those boxes and apply them to a certain level of standard. We want to commit ourselves to that standard. That was the main reason. We wanted to go into the CNCF. Also, to make sure that companies that would like to use us know that we adhere to those standards. It creates a lot of credibility.

It makes me think that it's quite different. Flagsmith went through SOC 2 compliance. SOC 2 is very much about the process that an organization has rather to a degree that you are scanning your software packages, applying patches, and things like that. It's very much about the organizational process. It's the stamp that people are after, whereas what you are talking about is leveraging CNCF, very much more around the open source project itself.

The level and the quality of the code, the quality of how you contribute, that we control the contributions, and the PRs are done correctly are big aspects of funding successful open source to run all of the governance. CNCF not only makes us adhere to a certain startup but also gives us a lot of resources to do it from knowledge to consulting, and people that we can work with. It's a community that we want and we feel like we should be active in.

In terms of the makeup of the governance board and the people who are involved in that CNCF aspect of the community, do you have to work towards and be conscious? You want to keep the commercial organization and the open source project. I'm not quite a Chinese wall but how have you gone about managing that and people going, “It's just this business thing,” which is what you'd want to avoid?

That was one of the biggest decisions we needed to make and feel good about before we contributed Kubescape to the CNCF. Kubescape as an entity is different and independent from the ARMO platform. There are no Chinese walls between the people. It can be the same people working on this and that. In terms of the technology, we made a lot of things in Kubescape that when we contributed it to the CNCF impacted even ARMO negatively. I will give you a very specific example.

Before we contributed Kubescape to the CNCF, we could promote our ARMO within Kubescape. You use Kubescape and we say, “There is this company ARMO. You should check them out.” It wasn't exactly like that but in general. We understand that you can do that once you are independent. We almost lost a chunk of ARMO lead and awareness generation capabilities but we knew we were going to do that.

For us, it was the right thing to do. Many companies decided not to contribute to CNCF for these reasons. It's a valid decision but it is something that you need to be very aware of as you go into the CNCF. You should come in with your eyes open about what Chinese walls need to be built between your open source project and your company.

In terms of the investment that you'd received, is it fair to say for these decisions and models? I'm thinking back several years ago, maybe if you have suggested something like that, your ARMO board might have been looking at you like, “Are you crazy?” Do you think those things have changed in terms of companies like HashiCorp and Redis who have trodden that path?

The perception of open source as a go-to-market or a development practice and company building practice is much more acceptable now than it was several years ago. The entire roadmap including contributing and giving Kubescape to the CNCF was on the table. The investors knew about it before they made the investment. In general, the open source world while getting much more acceptable is also going through very interesting times of how much open source you can be. We all know about HashiCorp and the changes in the licenses that they have done. The last years before that, an open source came in.

The perception of open source as a go-to-market or a development practice and company building practice is more acceptable now than it was several years ago.

The jury is still out about what model works for what company but there is much more understanding that commercial companies want to do open source. You need to let them do that and still have a commercial side. To think how it is going to be done, in many cases, companies find out that they need to change different things after the fact like what happened with HashiCorp. For us specifically, I believe we found a model that should work for us long-term.

I work quite closely with Dynatrace on the open feature project, which is also a CNCF project that Flagsmith has been very heavily involved in. I'm on the governance board for that. It's interesting because, for Flagsmith, it was a no-brainer. Having a standard set of SDKs and interfaces for feature flagging was great for us. There was no clear disadvantage or pain point to that. Elastic, everyone uses that as the poster child of how the ground can shift under your feet thing.

I do remember back before AWS had an Elastic-compatible service, one of the big differentiators between the commercial and the open source version of Elastic was this security layer, which AWS already had. I talked about OpenTelemetry already. It's interesting that the sequencing was a little bit different where there were businesses like Dynatrace, New Relic, and AppDynamics who are all doing this huge heavy lifting of building these incredibly complicated, technically advanced SDKs and then coming together and building OpenTelemetry.

It's interesting that the market is still large for that part of the industry and there are the ARMOs of OpenTelemetry coming and starting to compete with folk like Dynatrace. Everyone is happy with that because they all have their different niche. How conscious were you, two guys in a garage, to start a control plane on top of Kubescape or behind Kubescape?

It's weird that you mention it that way because, in my management team, the worry was always the other way around. No one said, “We should be concerned about two people in a garage building a SaaS of Kubescape.” Everybody was concerned with Amazon or Palo Alto doing their own. That was the concern. We believe that we are not doing something very simple about Kubescape.

We had proprietary technology and algorithms to it, that we believe have a lot of value and create a lot of value for the users. Is it a concern? It is a concern. If we are that popular, and maybe we are getting close to being that popular and that’s why Amazon is interested in copying us, I'm okay to deal with that but it's a risk that we are willing to take.

‍

You are singling out technical IP that is your moat around that business that makes you more comfortable with the decisions you have made. Thinking of Flagsmith, it's hard for a platform like a feature flagging tool to provide the opportunity for that. Was that a conscious design decision of ARMO and Kubescape in terms of that being your defensive moat or is that something that materialized over time?

It materialized. We started Kubescape and the ARMO platform. We started around visualization, management, and a very regular control plane. Users start to sign up and pay for it. You engage with them and you get feedback from them. Ideas come off, “Maybe we can do that.” Suddenly, the management was like, “Wait. This is smart.” It was a discussion.

“Should we develop it as part of the open source or only in the platform? Should we make that calculation in the cluster or make it a priority in a platform?” In many places, we felt that there is a technical IP that should be preserved at least in the beginning. Every technology eventually gets commoditized and then you can push it to the open source. We consciously decided to start only as part of the platform.

Are those decisions slowly changing over time? How closely you are holding those pull requests and stuff? Did you have a guiding set of principles, rules internally, and philosophy around whether something should be committed to Kubescape or ARMO? Were there arguments about it?

The ARMO platform is much more about consolidation, dashboarding, and workflows so the feedback that we got there was usually from ARMO platform users. When we got that type of feedback and new ideas that were groundbreaking and created IP advantage, we built it there. I don't think there is any instance to date where someone opened the PR on Kubescape and we decided to only develop it in the ARMO platform.

We could decide to do or not do a specific feature in Kubescape but we did not have an instance yet of a feature that we say, “He has a very good idea but let's only develop it in the ARMO platform.” I can imagine it happening. If I think about different feedback that could happen, it hasn't happened yet. Things that can happen is that someone asks for a feature for Kubescape that we do not feel is in the realm of what Kubescape was defined to do as a project.

I failed to think about something like that but I thought about integrating Kubescape into Slack, to get Slack notifications of different things. Kubescape is more of a scanner and that's the scope of where we keep it. It's not because we want to keep something to our services or something else. It's more about who is the user and what is the intention of that project that we are trying to keep the original thought of that.

It ought not to be in Kubescape because of the technical boundaries.

Yes.

In terms of security, you have got an interesting view on that. In terms of the industry, do you think that industry has a security problem still?

Yes. I keep watching for different attacks, penetrations, and things that are happening in the market. It still happens. One of the biggest things that are growing rapidly is that more and more human mistakes or configuration mistakes are being exploited. The complexity of systems is only becoming bigger and harder, and attackers love that.

The problem is only getting bigger. If you think about the number of resources that companies invest in security, it's quite significant. As an industry, we are getting better at it. We invest much more in it and that's why, in general, we are more secure. Security has always been and I'm afraid that it's going to continue to be a race where sometimes the industry is ahead or sometimes the attackers are ahead. It's a race that is always going to continue.

The open source industry is getting better at security. Nevertheless, it will continue to be a race where sometimes, either the industry or the attackers are ahead.

I always find it interesting as almost a layperson in this world. I have been building software for too long. Back in the day, none of the frameworks had any notion of sanitizing SQL injections and all sorts of things that if you start a Rails project or React Native application, you have got all this stuff built in. I find it interesting when you look at some of the big hacks and attacks. The level of technical proficiency in some of those is crazily high. Generally, they are not at the application level.

You can only get someone who doesn't understand what they are doing and writes a sequel injection vulnerability in their code but it tends not to happen. Also, the large outages that companies have, the mega ones, I'm thinking of the Facebook one a couple of years ago where they were down for almost the entire day. They are ultra-low-level configuration mistakes. They are at BGP. They are so far down the network stack. Those sorts of things have been around for so long. Is that an impossible problem to solve?

You hear about the big things and usually, the big things require a very proficient level on the attacker side. They investigate. It's a legitimate operation. A lot of money and proficiencies are involved. Those usually go into very deep levels but you can also think about many things like crypto miners and stuff like that happening all the time and you don't even know of. Those usually exploit very basic things. There's a question if someone wants to penetrate you. There are some very sophisticated governments that if they decide to penetrate the organization, I'm pretty sure, whatever you buy will not stop.

There are some sophisticated governments that if they decide to penetrate the organization, whatever you buy will not stop them.

They have got some zero days in their back pocket.

They are going to do zero days. They are going to use social hacking via your employees. Organizations are too big to be isolated and protected.

The rewards are getting ever larger as well. To turn into Kubernetes, do you think that Kubernetes is the last orchestration framework that's ever going to become popular?

I'm betting on it. I'm getting a company on it but at least for the next few years. Never say never. There is a very famous quote from Bill Gates that said, “64-mega should be enough for everyone.” I don't want to become that. If you ask someone on Docker, several years ago, is Docker going to be the last container standard? They say yes. That's not the case now. I'm sure something else will come eventually. I'm hoping that it's going to come in the next few years. It might be a development of Kubernetes.

I feel like maybe there's a subset of complexity like something that's maybe got a smaller surface area or something like that. It's interesting, isn't it? Back in the day, the first dot-com boom, there were dozens of different web servers. There were dozens of different ways that people could serve web pages. A lot of them yet, almost apart from Apache pretty much everything you had to pay for. Now, you don't even think about it. It's the NGINX in a container somewhere. There's a solved problem. You are right. It's commoditized.

I do think it's interesting that Kubernetes is one but the thing that it was trying to solve was for a very large organization. We see this sometimes with Flagsmith where we have people deploying Flagsmith onto Kubernetes and having a lot of trouble with it sometimes because they don't have a Kubernetes guy. Do you put any thought when you are working on Kubescape or ARMO in terms of like do you ever write code and try to solve a general case for something rather than a Kubernetes case for it?

It's not like there is guidance in the company. They make sure to write code in a generic way so that if Kubernetes dies, we still have value. The reality of security is that much of that is transferable and can be transferable but in general, we are better on Kubernetes. We think that Kubernetes is becoming big enough in companies that it will become almost equal to the cloud in many organizations.

That's why we are very much focused on solving an end-to-end solution for your entire security needs within Kubernetes because if you try to solve it, many people use end-to-end. There is the CNAPP. People are talking about CNAPP or Cloud-Native Application Protection Platform, which is like a Gartner term for everything in the cloud. That's impossible.

CNAPPs do not exist. That's what I think. They exist in marketing. I'm doing CNAPP. You are doing CNAPP but I don't think it exists from two sides of the story. One is on the provider side. No one created such a wide platform. Prisma Cloud is probably the closest to becoming CNAPP but it's not CNAPP. It's a portfolio of products that they bind together to call it CNAPP but you don't ever get the experience of one product. I don't think you need that experience.

There is no one buying CNAPP because no organization has one person or user that needs all of those CNAPP functionality. Eventually, you have different users in different parts of organizations using different parts of the CNAPP. That's why it doesn't exist. That's why I believe KNAP is a thing because Kubernetes is an isolated place in your organization that is even capabilities for and for that, you need a solution. That's my view at least on the market.

As a final question, what's next both for Kubescape and ARMO?

For ARMO, we are going big and wider. We are getting into more organizations. ARMO in my mind should be the platform of choice for any company that is better on Kubernetes as a significant part of infrastructure. ARMO should be their platform of choice. Kubescape is going to become larger and bigger in terms of what it can do in your cluster.

We released Kubescape 3.0 in the last KubeCon. We announced it. We added many more capabilities to it. We are continuously adding more capabilities to make Kubescape your open source security of choice, let's call it, for Kubernetes. We are not shy of even cannibalizing ARMO to make that happen. Many of the things like relevancy that are available in the ARMO platform are also available in Kubescape creating SBOs and KBOs. We are continuously growing that part as well.

‍

If people are interested in getting involved, contributing, and being part of the community, where are the best places for them to go?

GitHub Kubescape, that's how most people come to us. They could google Kubescape as well. If anyone is interested in contacting me, I am super reachable. Shauli Rozen on LinkedIn or Twitter. Reach out. I will always answer.

It was great to talk to you. Thanks for your contributions to the open source world. I wish you luck in the future.

Thank you so much for having me.

‍

Important Links

• ARMO
• Kubescape
• Kubescape 3.0
• Twitter – Shauli Rozen