Robin is the CEO of k6, a performance testing SaaS used by organizations developing systems and that need to understand their real capacity limits. Check us out at K6.io
More and more companies are choosing commercial open source business models when they launch. What is still pretty rare is closed source companies that choose to move to open source after they are up and running. For k6, that was the exact path that they took.
While this move took at lot of thought and effort, it led to them growing their customer base substantially and finding a way to monetize while building a community at the same time.
My favorite quote from the episode showed how humble Robin's view is on their value to customers: "Any engineer can mash together a load testing tool in an afternoon in their favorite language... we believe that the value we capture is different from just generating the traffic."
I see a lot companies out there that monetize on things that engineers can easily replicate which feel like great opportunities to drive more and more commercial open source companies over time.
Hope you enjoy the interview!
Robin, thanks for coming on. If you want to tell us a little bit about your background and the projects you're working on.
Thanks for having me on. I work for a company called k6, previously called Load Impact. I worked in this company for several years. My background is in engineering. I still very much identify as an engineer, but I'm on the business side now. I'm the CEO of the company. What we do at k6 is we're in the business of load testing. What that means is we help our customers identify problems both in terms of reliability and scalability with their software systems, whether that's websites, web apps, APIs, microservices, etc. Whatever needs to be tested, we can help our customers test. We have a solution that's open source based. We have an open source project called k6 and we have a SaaS service, which is how we monetize the open-source project. That’s how we make money.
What was the genesis of the company? Did it start as an open source project?
This can be a very long story or a short one, I suppose. The origin of the company is as a game development studio but for various reasons, that didn't pan out. The company pivoted to doing network related products instead. We were doing massive multiplayer online games twenty years ago. We took some of those learnings, experiences and technologies, and turned them into products for the network market. That’s the very early origins.
It’s like Slack?
I'm not too familiar with their story.
They had a massively multiplayer online game. There are a couple of others. You're standing in firm footsteps there.
That's common that a lot of people that get into tech have some origins in gaming, whether that's as a gamer or interest in games. We started doing that, and then we've done various projects around monitoring of networks. We did lots of consulting around performance testing, and roughly 10-11 years ago, we realized that there was an opportunity in creating a SaaS-based solution for doing load testing. That's how the origins of the product company came to fruition.
Was that scaling IP like tools that you'd built yourselves to help scale up a gaming platform?
No, not the load testing. That was too far after the gaming adventures. This was the stuff we were doing as consultants for larger organizations and governmental agencies here in Sweden. We built up a lot of experience with various tools of the day, both commercial tools and open-source tools, and felt that there was an opportunity to create a SaaS-based solution from this. This is when AWS was first launching. We took advantage of the elasticity of compute and data centers that Amazon offered. We were quite early adopters of the AWS platform. That's how it started.
At that time, the code was completely proprietary.
In the beginning, it was completely proprietary. I don't think we had considered open source as being a viable approach to structuring things. It was proprietary as a SaaS-gated product. As things happened over the past ten years in terms of how organizations changed, both the structure of organizations and how software has changed in how it's developed; smaller teams, automation pipelines, moving things faster from development to customers, etc. Things also changed in terms of how companies are approaching quality assurance and testing. We started looking at how can our solution fit well into automation pipelines, CI pipelines, etc. We thought this would be much better if we could offer our core technology as us open source. It would both help the larger industry in practicing load testing, but it would also help us grow our user base and eventually, a part of the user base would become customers.
“One thing about open source that you don't get in proprietary is that you get people who are to help fix bugs and report issues.”
That's completely unique to all the conversations I've had up until now. Tell me about how much IP was wrapped up at that point. There's a lot of questions around that. It's interesting as well. There seems to be this implicit assumption that CI/CD tools or things in that realm, that there’s a large proportion of them are open source. Did you feel you were the odd one out a little bit?
There was a combination of factors. We started hearing some customers say they were opting for open source solutions rather than purely proprietary, both for cost reasons, but also to avoid lock-ins of technology. We also had discussions internally and realize that, “This is a much better approach for us.” There's nothing special in being able to generate lots of requests. That's easy to do. We jokingly say that any developer could mash together a load-testing tool in an afternoon based in their favorite programming language. There's not much special in terms of IP in the load-testing tool itself.
We believe that the value we capture is different from generating the traffic. It’s the things we can offer in terms of managing infrastructure for people in the cloud service, storing and presenting the results, which can be quite a lot of data points in a load test. Visualizing that effectively is more of the value that we provide as a business. All in all, we thought there are win-wins here. So let's open source are our core technology, and see if people like it and people adopt it. It's been pretty successful so far.
You had like a huge ball of IP that was completely closed. You factored out the request engine and that's the open source project.
We didn’t open source that. We had a tool that's similar to k6, but it was proprietary written in C and scripted it in Lua. Test cases were written in the Lua programming language. We came to the conclusion that this was not a good project to open source for various reasons. From the beginning, it wasn't written to be public in the first place. It wasn't written to be published or have people scrutinize it too closely or have external people contribute to it, etc. It's also written in C, which is now perhaps not the most widely used languages, especially amongst people in the web development community. Lua is not also the most common scripting language when it comes to a common web developer either.
There is some strong gaming heritage in those two languages as well.
There are. We realized that the best thing for us if we want to open source this is pick a language that's going to be better suited for attracting contributors. We can foster a good community around the project and also change the scripting language to something that is more familiar to more people. That's why we chose Go for k6 and scripting language.
What time was this?
We started working on this in 2016 and then we open sourced it in early 2017.
Go was well established at that point. Now you might use Rust?
Rust was also one of the contenders when we discussed what language to write this program in. When we looked at other tools in what we call the DevOps tool chest, we saw that a lot of tools like the HashiCorp tools were written in Go.
Go has won that space.
We felt that Go was a pretty good choice considering the tool would be in this space. That was the winning argument that I would say why we chose Go.
Then you started with an empty folder. There's a Cargo init in it, or something and started completely from scratch.
It was an empty repo, nothing to inherit. In the beginning, we had urges to try to reuse stuff from the C program, but we quickly realized that Go is a different beast. You need to approach things in a different way and architect it in a different way. We also wanted to leave some of the baggage from that old C program behind. We also had to scrap the first iteration of it and start from scratch again. Partially also because we were at the same time learning Go. It was a new program for us. All in all, there were lots of moments of risk involved in this, but it panned out well.
Why did you start again? Was it because you realized it wasn't idiomatically Go?
Lua is not asynchronous by default.
At what point did you push everything to GitHub then? Was it straight away?
We worked on this product internally for good eight months or so. We then pushed it in February 2017. That was the first public push to GitHub.
Was that a fully functioning tool?
It was a very early version, but you could use it and you could execute tasks. We had some immediate early adopters as you tend to see when you publish these kinds of tools. We learned a lot in the early days from people giving us feedback and also running it ourselves a lot. It was a positive experience because it changed a lot of things for us also. We had done purely closed-source proprietary development prior to this. All of a sudden, we had this open source project, which was a new thing. It was unknown territory for us to learn how to have our previously core IP out in the open and having all of those uncertainties around it in terms of the business. We had a lot of discussion like, “Are we giving away the secret sauce now?” Over time, the winning arguments in terms of why we should do this have won out.
What are they?
The value that we deliver is not primarily in the generation of traffic. People don't come to us because necessarily, we can generate lots of requests. It’s because we can remove some of the burdens of managing infrastructure for them. Rather than having to manage hundreds of Amazon or whatever cloud servers, they can press a command in their terminal, and we spin up that infrastructure for them. We collect all of the many gigabytes of metrics and present it in graphs that are easily digestible. It's a lot of convenience. We primarily sell convenience as a SaaS service.
In terms of open sourcing it, what were your expectations in terms of the benefits before you did it? Were there any that you discovered afterwards that you hadn't considered before?
We had some ideas that we would get a lot of contributions initially, which hasn't panned out as we thought. We thought that, “We're going to publish this. We're going to get lots of contributors. Everything is going to be a nice and dandy.” We've gotten contributions for sure. We've gotten both small bug fixes, as well as some bigger plugins for outputting metrics primarily to various different destinations. We thought that it would be more a collaborative thing with the community and other companies than it has turned out to be. We still do the vast majority of the development and planning of the tool.
We thought in the beginning that it was going to be a little bit different. I think that it takes a good while before you get that commitment and contribution from other entities, before any shared governance of the project. That was for sure different in terms of the reverse. Things that we didn't expect but that happened. I don't think we expect that there would be as big of an adoption as there has been. It was slow in the beginning, then in the last year, it's exploded. That was something that's been positive, and seeing how people are willing to help us fix bugs, reporting issues, fixing bugs in the documentation. That's been super positive to see and something you don't get as a proprietary technology vendor.
It’s interesting because that’s a common point. We see this as well where you maybe get bugs raised and you might get minor pull requests here or there. Sometimes it’s for the free t-shirt that whoever is offering at the time. Core functionality seems to be almost unheard of, especially for some of the larger projects. The guys that worked on larger projects that I've talked to have said, “You need to do it as your full-time job anyway. Otherwise, the pull requests are going to be a total mess if you even get that far.” When you say it's exploded, how have you tracked that? How do you know that?
We have some telemetry in k6. If you execute a test in your terminal, there's a ping that gets sensed to one of our servers. We know that a test was run. We don't know anything in terms of installations. We don't have any way to track that, but we know when there's a test run. We can see through the cumulative sum of tests over time. We also have the public open source metrics like stars. We can see the stars and how they have grown on GitHub, for example. We can see how the members of the community have also grown, and questions in our forum in Slack and GitHub issue participation.
“Open sourcing is all about finding the balance between what’s valuable to large companies and what can bring more community value when free.”
Do you know why that started to happen a year ago (in 2020)? Do you think you hit an inflection point? I've heard this from other people as well. I remember someone was saying that once they hit 1,000 GitHub stars, things started to get like the little hockey stick a little bit. I'm not sure whether that's because of the stars, or the stars is a byproduct of something else that's happening.
We've talked a lot about this internally. I don't have a good answer. There's no one specific thing that we can point to and say, “This is the one thing.” If we look at the GitHub stars, our GitHub stars trajectory has not been an exponential journey. What's given us the most stars have been when we got the attention on Hacker News. We had a good weekend behind us where we got posted on Hacker News and ended up on the front page, which creates a pretty massive surge of traffic to the target hub repo in this case. Our star count increased by something like 30% or 35% over 2 or 3 months. For us, it's been like that. We had one of those events. It’s the same thing that’s happening in the first six months after we launched the k6 repo. That created the initial jolt of interest. We then had produced linear growth since then, but clearly upwards trending. We have this huge spike of interest now. It pushed us above the 10,000 stars, so we're happy and celebrating.
As you did your business, did you try and figure out what it could be? When things started to tick up, where you like, “Can we repeat this?” My opinion is that these things tend to be an accumulation of a bunch of little things. It’s not like if we do task X five times more, than this?
I don't believe in that as well. I don't think there's any one thing that do this 1,000 times and you'll become a unicorn or something like that. It’s the sum of all of the small things that we are doing and are fortunate to be exposed to that makes things like this happen. I don't think there's any one particular thing. It's lots of things that we've done over the years that's resulted in this.
Did the way that you marketed things change? How long had you been running before you open sourced your core engine?
The company as a product company doing load testing, we had been doing that since 2009. We had a good eight years before we open sourced k6. We had to change a lot of things. When we got started, we were targeting performance engineers and quality assurance engineers. Now with k6, it's more heavily targeted towards developers. We talk a lot about developer experience, which is one of the primary things we focus on. We’re making sure that there is a good, enjoyable, productive experience using k6, as well as documentation. We see the documentation as being a big part of the overall experience. We spend a lot of energy on that, which we didn't do as much previously because we had other things in terms of commercial support and services. Now, we focus more on making sure that the self-serving developer has a nice experience, which demands a different approach.
At some point, you swapped out the core engine that was driving your SaaS business and deprecated the C version, and brought in the new Go one.
We had to build a new SaaS service. After we had released the open source version, we had to reinvent the SaaS offering as well. We did that in parallel. While living off of the revenues of the old product, we built a new SaaS product.
It's a completely transformational change in all aspects of the business.
We had effectively between 2018 and roughly the beginning of 2021. We tend to see them as middle years like in between years when we have transitioned from the old Load Impact’s brand and products over to the k6 space brand and product.
Is it a pure SaaS offering that you're providing now, or do you do on-premises stuff for people as well?
Now it’s purely SaaS. SaaS is what we do. We offer some services, but it's a very small part of our revenue. We do offer help with scripting, execution and reporting if our customers are asking for that, but the vast majority is still SaaS.
How did you come up with the license for the core products? Was that something that was straightforward or not?
No, that's one of the most difficult topics when it comes to open source. There have been many things happening around licensing in the past several years since the advent of the big cloud vendors. There's this fear in the commercial open source space that companies doing businesses on top of open source, that you have to be very careful with how you license your product so that you don't get eaten or having your lunch taken by Amazon and the likes. That was a big conversation for us as well, coming from being a proprietary business, not having anything open source. All of a sudden, we were releasing our core jewels to the world.
We wanted to make sure that we had a license that didn't make it too easy for someone else to grab all of the value. We tried to look for license that would prevent others from more quickly commercializing k6 than we could do. As a small company, we were 10, 15 people at the time when we released this or even less. It's a very small company. There are lots of companies with larger resources than us. We picked AGPL as our license, which is the Affero new license. It’s a modification of the GPL Version 3 that adds a clause with some restrictions around how you could use the projects in a networked environment. There are some uncertainties how that clause would be interpreted in a court. It's not been tried. There are some uncertainties around exactly what it protects against or how it plays into the virality of the GPL license.
Are you talking about globally or just in the EU?
I would say globally. The reason why the AGPL license was originally created, as far as I know at least, was to try to prevent this perceived loophole that existed in GPL, where you could access GPL license components over a network. That would make effectively the larger work built on top of this GPL component, not having to be open source itself under the same license, so voiding the virality of the license. They added this clause in the AGPL version of GPL Version 3 that states that if you are accessing this component over the network, it will also be infected with that virality of the license.
They are one and the same as far as.
If you provide a service or a network service to your end users, you would also need to make the source available to your users. As far as I've been told by lawyers, this has never been tried in a court, and there are uncertainties in how this would be judged and ruled. There are examples as with MongoDB. MongoDB was originally AGPL licensed and Amazon released their at least MongoDB compatible API service. MongoDB changed it and called it SSPL, the Server Side Public License or something. They changed some of the clauses of the AGPL to explicitly prevent anyone else from launching cloud-based products using their IP or MongoDB source.
We came from those discussions as well. Being a little bit scared and fearful from what other vendors in the industry would be doing with our source code as we thought about it in those early days. I've personally come to believe over time that the tool that we are building is a little bit different from the databases, that MongoDB and others are producing. The value that we are trying to capture is not in the load generation itself. I don’t think there’s an inherent conflict there between open sourcing the load generation engine and the SaaS product that we're making money off of.
How much of that web application is proprietary? Is that all of it?
The SaaS product is close-sourced. We use the open source k6. We use the stock binary. We don't have a custom binary. We use the same binary as available from GitHub and Docker Hub, etc. All of the orchestration stuff, our pipeline for ingesting lots of metrics, and our storage solution, all of that stuff is proprietary and close-sourced.
Have you considered increasing the scope of that open source componentry? Do you feel like you’ve reached a sweet spot? It's quite interesting because you’ve got a very clean separation of concerns with your product, which some projects don’t have. That’s elegant. If you were to have to draw a line over what was open sourced, it would be immediately obvious to anyone looking at it for twenty seconds.
It's a good, clear cut separation between the open source and the commercial. We do want to move some of the things that we offer in the cloud-based product to the open source. For example, with the cloud-based product, you can do globally distributed load tests. You can run load tests across many geographic regions at the same time, and we collect results, etc. We do want to offer clustering or distribution support in the open source k6, which we don't at this time. We do want to open source something around that. I don't think exactly what we have in the cloud would be appropriate for k6. At least the functionality to run tests from either multiple machines on-premises or using the container-based services offered by the cloud vendors. Something that these two use multiple machines to run tests.
Talk to me about clustering because 98% of commercial open source businesses hide that behind their curtain. What's the driver behind that?
We don’t think the generating traffic is the major thing that we provide value. It goes on that same line of thoughts.
Do you mean you’re making a lot of these decisions based on that premise and philosophy?
As a business, we need to capture some value. Otherwise, we wouldn't be a business. We're trying to balance what we think is valuable to larger companies that have the capability of paying licensing fees compared to what we can get in terms of community value from the open source project. We try to see, where does this feature fit? If this is something that an open-source user would not care about, but an enterprise would care about, then it's something that we would put in the cloud service. That's usually how the discussion tends to go.
Are there heated arguments within the business around whether those are the right things to do? Does it always feel quite natural and obvious?
There are heated discussions, especially in the first two years after we launched k6, when we still had all of these uncertainties and doubts around things. We were still learning how these things worked. We also tried to talk to other companies and individuals in the open source space to try to learn how they had done things and try to absorb any wisdom and learnings from them. We had a lot of doubts. First of all, should we release this as an open source? When we started talking about this clustering and distribution, because it's been something we've talked about from the beginning, there have also been lots of doubt like, should we do this? Is it too much?
Ultimately, we've come to this conclusion that it's very beneficial for us as a business if we have a successful and useful open source project. We tend to argue that the people that are going to be forever open source users would anyway be forever open source users using some other project. They would not contribute in any way to our revenue. The companies that are larger tend to value having some business relationship when they use a tool or product that is in some way important to their business. They tend to value to have at least some support agreement or some way of contacting a human being and answering questions, making someone accountable. It plays out well, in general, if you have those understandings.
In terms of the community, what were your resources for learning that stuff? That's something that I've found weird. Any technical question, you can Google it and there are fifteen medium posts on it even if you find some completely obscure, esoteric area of technology. It’s like, “What open source license should I use? How do I track progress?” There are many things that we didn’t see when we were starting out because we were feeling around in the dark. Were you the same? Was there like maybe some amazing business book that explains this, so I’ll buy it and read it?
There's no specific book. We try to talk to people that we had in our network. We found out there was a law firm here in here in Sweden that had helped guys behind MySQL and some other tech-product based companies. We used the same firm and talked to them about licensing. They advised us on that and helped us in our internal discussions. That was very helpful. We've talked to people in the open source space, both here in Sweden and abroad as well, that have been in the open source community for a long time to get some guidance like, "How should we do this? How do we start this project? What should our expectations be? What is reasonable and what is not?” We've been fortunate in that way that people have been willing to help us and teach us things. We didn't know anything about open source or operating a project, or building a business on top of open source. We're still learning a ton. I talk to people every week and learn new things when it comes to what is a good way to do, whatever it might be. How do you track adoption of the open-source project, which is a topic you bump into when you talk to investors. There are things to learn.
We don't know how many people are using it. We shifted to a Century API key or something. We were getting Century reports from self-hosted products with weird domain names. We’re like, “What the hell was going on?” The penny drops and we realized that we didn't have any telemetry in our product. We're building that right now. I hadn't considered that before in terms of open source. I thought if you buy Oracle or whatever, it's got all of that stuff all over the place.
Everyone I've been speaking to who've been running larger projects, they all have them. It's completely normal and they're generally opt out. You get a small percentage of your community who burn everything and say that's not acceptable and disappear. For the most part, people are happy with that, and that's a contract. We'll be able to figure out how many people are using it. For you, it’s quite interesting to hear you got to run this balance between privacy, anonymity, and information. It's interesting to hear that you're tracking runs as opposed to something else. You’re trying to figure out, first of all, what's acceptable to track. Secondly, which one do you want? You've got quite a nice North Star data point.
We're tracking the aggregate number of tests right, rather than trying to track any particular installation and how that is being used. It’s very high level. It's very helpful. Our default stance when it comes to data is to collect as little as possible. It's a liability as a company to hold data. You don't want to hold it, but at the same time, it's very helpful to know where you're at because it's not always possible to do qualitative sessions with people to try to understand how big is the adoption of this thing. Are people using it or not? We get that quantitatively through the telemetry. We know this is increasing in usage. That's a good thing for us because we can make decisions on it.
I wish we would have known about that a two years ago. It would have been one of the first things we did when we pushed our open source projects. I've moaned about this to other people about the analytics in GitHub are weak. I've got a spreadsheet that I update every couple of weeks with how many stars we've gotten and how many clones we've gotten. It's like, “This is nuts. Why am I doing this?” There's nothing you can do. There's like Docker Hub pulls. It says 10,000 all the time. I'm like, “Is this going to change? When it changes, is it going to go to 100,000?”
We've been through one of those discussions, especially with investors, when they ask how many installations or users do you have of your open source project. We simply don't know. We don't have any count that says we have X number of installations or number of users. We can also look at the Docker Hub image pulls, but we have no idea what that represents.
It could represent someone with a script that's gone nuts.
Do they pull the image every time the CI job is running or do they cache it locally? Who knows? It's difficult to break it down into any meaningful number. It's just a big number. This test run telemetry tells us something. It's something that's valuable without being overly intrusive.
You mentioned investors come in, did that tick up when you open sourced the component?
There has been more interest. I don't know if it's because of the open sourcing, but it's helped that we have gotten more attention over the past 12, 18 months. It’s something I didn't know, but I realized that a lot of investors are looking at GitHub, tracking the stars progress on repos and looking if there's any company behind those efforts, or if there is a possibility of creating a company behind those efforts. That's something I learned, which is understandable and makes sense.
That’s my next business. I'm going to write a GitHub scraper and sell it into the Valley. I've heard that from other people saying that when they got onto the top ten fastest-growing projects or something, there are junior analysts that sat there, refreshing those pages. It's interesting as well. The stars are funny. They are a bit of a vanity metric, but they also are important. It's interesting as well. Before you open source anything, if I was to come to your website, I would have no idea if you were like one person doing it on the weekends or there were 500 people working in that.
Maybe I could do some research and find out, but now there's one link on your homepage to GitHub, and immediately can see you've got 67 contributors and so on. Especially for a commercial open source business, you get a good feel for the scale of the project and the business, which is good for customers and end-users as well. No one's going to say, “It's me and I'm doing it in my spare time. I have a day job at Google. I'm hoping to sell a big-ticket price license, then I'll be on my own.” I find it quite comforting in a way.
It de-risks things a little bit. It makes things feel a little bit less risky when you see there's a big star count and you see the project is active. There are commits in recent days. All of those are, at least from what I know, talking to people who work in this company who are our engineers and other engineers in other companies, it's metrics that we all look at. What is the freshness of the project? How many stars does it have? Does it look well taken care of? Is there documentation that explains how to use it? Is the installation experience nice? All of those things matter. The stars in itself, that's a vanity metric in a way, but It gives some credence to the largeness of the project. It's also an early indicator from what we have learned in terms of what can come down the line. For us, that means people trialing our SaaS service down the line.
“There's no one thing that you can do 1,000 times and then turn you into a unicorn. Growth is usually just an accumulation of small wins.”
When you get onto Hacker News, you get that huge star count bump, do you then get a bunch of signups two weeks later?
We get some immediate effects. In this case, since the link that we got on Hacker News was pointing towards the GitHub repos that resulted primarily in star counts. It tends to be some weeks and months after where we start seeing the effects of people coming and trialing. Both are using the tool because the star doesn't necessarily mean that they try the tool right away. It’s like, “This looks interesting," then a few weeks later, maybe some people start using the tool. After a few months more or a few weeks more, they trial the SaaS service. Hopefully, down the line, part of this huge chunk of people that star the repo will turn into customers for us.
Did you shift how you were marketing after you open source things?
Yes, we did. A big part of our marketing is now rather than being the traditional marketing tactics, it's more heavily leaning on developer relations. We focus a lot on making sure that documentation is good. We have all of the features well-covered, well-explained, good documentation of the APIs, writing content that's useful in terms of how you integrate k6 with other tools that you're using. It's been user-focused. Helping grow the user base rather than being value-focused and trying to sell it to the people that have the budgets. That's how a lot of the adoption and selling of our SaaS product happens. Someone adopts the open source tool, they bring it into the organization, and then the organization realizes that “Maybe we should see what is this commercial product? What does it give us?”
Before we finish up, is there anyone that you want to say thank you to or any funny stories about some crazy code happening?
I would like to thank the entire k6 team. It's a team effort so they should have all the kudos for everything that's happened to k6. In terms of funny stories, maybe one thing that people don't know is that we have a crocodile as a mascot. We call him Bert. The reason why we do that is that we had two real alligators in our office previously. We don't anymore.
You got the most stereotypical gaming company. I should have asked that question at the start and not at the end.
Robin, thanks so much for your time. In the past when I've wanted to do this, I've always downloaded WRK. It's great at generating traffic, but the data that comes out of it is rudimentary. I'll be another star in your star account and trying it out. I want to thank you again for your time.
Thank you very much for having me.