Ben Rometsch

Kyle Johnson

Welcome to The Craft of Open Source, hosted by Ben Rometsch. This bi-weekly podcast is focused on the ins-and-outs of the Open Source Software Community. Join Ben as he speaks with the brightest minds that have brought us some of the most adopted technologies on earth. Each episode is an interview with creators, maintainers, entrepreneurs and key contributors to the open source community.  

You can find the podcast by searching in your favourite podcast app, or with this RSS link. 


Thanks for joining us. We’ve got David Heinemeier Hansson on. Open source royalty. Thanks for your time. For those people who don't know who you are, maybe you want to introduce yourself and also your project and business. 

I’m David Heinemeier Hansson and I'm the creator of Ruby on Rails. I am Cofounder at Basecamp, now also the makers of HEY, an email service we launched at HEY.com. I’ve been doing open source for twenty years or so. I’ve been part of the Ruby on Rails Projects for more than eighteen years. It was in 2003 when I got started with that. I’ve written a couple of books on business, It Doesn’t Have to be Crazy at Work is the most recent one, and the most popular one is called Rework from 2010. We also wrote about remote work in the ancient times of 2013 with Remote: Office Not Required. 

I read through the bulk of it recently and it’s surprising how much of it is still spot on. Including the discussion about what would be the trigger. When is it that are we all going to move to this? One of the things was there's all sorts of natural disasters that can happen that force you to go remote. We didn’t have the pandemic in there at the top of the bingo card. The Remote Work discussion is intimately tied to open source because virtually all open source has been developed remotely. And it has been developed remote for many decades, which was one of the reasons and arguments we explicitly use in the book. This is not unproven. This is not an exotic imaginary way of thinking. This is the way most of the world’s most important software has been created for the past 30 to 40 plus years.

That’s a great point. I hadn’t considered that. You’re right, projects are complicated like a Linux Kernel. Back in the day, almost before version control. The original Linux Kernel, was it BitLocker they were using way back? 

That’s one of my questions for you. It was around the tooling for Rails. How did you discover Ruby, the language? 

Back in 2002 to 2003, I was reading IAAA Magazine and it was also ACM Magazine, some of these computer magazines where Martin Fowler and Dave Thomas were both writing about programing concepts, patterns, and the likes. They were both using Ruby to illustrate their examples because Ruby has this quality of reading like pseudo code. It’s like if you don’t know Ruby, you know Ruby. If you know any programming at all you know how it reads. You don’t know the syntax of Ruby to understand concepts explained in Ruby. I thought, “This doesn’t look like the stuff that I’m used to. It doesn’t look like PHP or ASP.NET, Java. It looks way better.”

Not only that, both Dave Thomas and Martin Fowler were enthusiastic about this programming language. It would usually preface like, “I’m going to use Ruby to explain this thing.” I don’t get to use Ruby commercially because of reason but if I could essentially, I would. When I had the opportunity to pick my tech stack, where it was a client mandating, “You have to write it in PHP. You have to do this and that.” I thought, “Why wouldn't I pick Ruby?” Here are some of these incredibly smart people who think highly of Ruby, I like the way it looks and now I have an opportunity to create something new, I should use Ruby. That thing happened to be Basecamp. Pretty much for the first project, I created a Wiki at the same time, it’s now lost in the ages. When I started in Ruby, it was Basecamp. Developing Basecamp and getting into Ruby through the development of Basecamp was also what led to Rails the framework. 

It all started with reading some articles, magazines and going, “That looks cool. I want to do that.” I started then giving myself essentially a budget. I was going to spend two weeks and see if I could make something work because I knew how to use PHP, for example. I knew that the thing I wanted to build and I could totally build it in PHP but I was still uncertain whether I could do it in Ruby. I set myself two weeks. Can I get the basics? Can I talk to a database? Can I put something on the screen? Can we render some HTML? 

It took less than a week and I was head over heels in love with Ruby. It was like this gift that keeps spreading when you see the galaxy brain at the end. Here’s the whole thing exploding and visions going off. This was how I felt. I've been programming for several years, but I didn't know programming could be like this and it was far more interesting, engaging and it vastly accelerated my sense of learning and capacity as a programmer. It essentially set me on a course where I went from being, if not outright, a reluctant programmer, but certainly a doubting programmer. Was this the thing I was going to do for the rest of my career? This is the thing I'm going to do for the rest of my career.

The tooling at that point must have been pretty rudimentary and the libraries and stuff. Were there database drivers back then? How rudimentary was it?

It was rudimentary. There were raw database drivers in the sense that you could send SQL strings to MySQL as I was doing, but not much more than that. There were some things that weren't quite to my liking, so I essentially had to build everything myself, which was how Rails ended up being a full stack framework. We had to build the database integrations, the template languages, the controller code, and all the frameworks that were required to build web applications. 

I essentially went like, “I'm going to build that myself.” That came after basically reading all there was because there wasn't that much. I read through the entire standard library, which happens to be in Ruby, so you can read it. You can open it up and read through it. I learned a bunch through doing that. I read a bunch of other code that was released but there wasn't that much, so it was simply a necessity that I had to write everything from scratch. At the same time, it was also a pleasure. It was a pleasure to show up in a community that was that fresh. When I showed up at the first International Ruby Conference in the US, this was 2004, there were 42 people at the conference. 

When I asked how many in the audience were using Ruby commercially, how many people get paid to use Ruby, I raised my hand because I had a contract to build Basecamp and one other person raised their hand. We were two people at that third edition of the International Ruby Conference who got paid to do Ruby. The rest of them were there because they love it. They were there because they were excited about Ruby. That was a quality that was new to some extent. 

PHP had been commercial from the get-go. It wasn't a bunch of people doing PHP because they love PHP the language. Much the same, I'd say in Java. This idea that you show up in a community that has the quality of attracting people simply because they love it was new, invigorating, and inspiring. There were other people who felt the same way I felt; that they've been programming and they didn't even know that they had been searching for what Ruby was the answer to. 

I was working professionally back then and it's interesting how Python as well, to a degree were languages. It feels like back then it was much more around you were doing it out of necessity rather than out of love, and people didn't tend to love the languages and the tools that they were using, whereas now it feels it's different to me in a way.

Part of it was one of the other things that Ruby introduced was a complete paradigm shift in thinking about what are your tools supposed to do for you? Up until I would say, Ruby, at least for me and my discovery of Ruby, programming tools and ecosystems were thought of as that, tools. These were things like, “Can you do the job? Can you bang into the nail with this hammer? Who cares what the shaft is like, what the materials are like? Can you get the job done?” This is still a popular strain of thinking about programming language and tools but Ruby came in and already from its inception in ’95. 

Ruby's first mission is to be the programmer’s best friend. It’s to put the programmer first. We're going to put the machine 2nd, 3rd, or whatever. The first person I'm writing Ruby for is the programmer and I will do things to the machine that it doesn't want to do, because this makes a language that is more of a joy, more of a pleasure to use and it's going to be complicated to write the compiler. Ruby's compiler is famously intricate, complicated, convoluted, and hard to optimize in a bunch of ways because it wasn't a primary concern for Mats. 

The primary concern for Mats was the compiler in the human brain. When you optimize for that compiler, the human brain, you end up with a completely different language and you end up with a completely different approach to language design in a whole way that the thing feels and is constructed. Even that word feel is a thing that now gets evaluated on at least at par if not put higher than memory efficient, runtime performance, and all these other engineering terms you can use to describe whether a programming language is good or is not good. These were factors, but they were secondary factors to how does it feel to be a Ruby programmer? What do Ruby programs end up looking like? 

This idea of aesthetics is also a somewhat novel turn at least again, it was for me from the backgrounds I came to. I know that there are other programming environments like Smalltalk, and perhaps to some extent, Lisp or otherwise that has the focus on this. For me, the focus Ruby put on it was quite extreme, that there would be multiple ways of writing the same thing simply to provide the peak aesthetic for a given problem space. Do you put the if before or after the condition? Is there an unless? That's an If Not. Why do you have an entirely new keyword for a thing that could be expressed as an If Not? Most language design up until that point was much in the vein of doublespeak. Can we reduce the language to fewer terms and it will become simpler through that way? 

That's also one of the key contrasts with Python, for example. Python has it stated that there's one and preferably only one way to do something. Ruby went the extreme other way. There are preferably as many different ways of doing something as you could imagine in natural languages, more or less, not quite, but far more so than Python has. Ruby embraces this level of divergence that you could write the same piece of code 100 different ways and a lot of them only had aesthetic differences but to me, that was the appeal. 

That was what made me fall in love with Ruby, as not a programming language, but as a language, as a way of expressing not only creativity but what you want to do. I look to this day, as programming is writing. Programming to me is not engineering. I don't think of myself as a software engineer. I do software engineering a minority of the time, where I do structured tests and all these other things but the majority of the thing I do is writing. It’s where I try to optimize something to make it clearer to the reader and to myself to make it beautiful and make it flow in much the same way that when I focus on writing in prose. It's not about, “Can I convey an idea?” No. I want it to be lyrical to some extent. I want my programs to have a dash of poetry and Ruby allows that like no other language I've ever known or seen.

Ruby’s first mission is to be the programmer’s best friend. It puts the programmer first.

You don't think Rails could have been expressed at that time in a different language? That wasn't by design, that was a fortuitous accident.

I know it can't be because many people have tried in different languages and I've looked at most of those implementations and they're fine. They're good for what they are. They're not Rails. This is one of the things when Rails first came to the scene and got popular. There were a bunch of other ecosystems that tried to adopt it and at the time the trademark was a little wavering in the wind but it was one of the things that made me trademark Ruby on Rails and Rails itself. 

I didn't want someone calling it, Pearl on Rails or Python on Rails because Rails is so intricately linked with Ruby. It's a celebration of Ruby and the things that make Rails, Rails, you look at a piece of modeling code that uses an active record and how it reads like pseudo-code, in a way with absolute minimal simple noise is difficult to recreate in other languages. You have to have way more simple noise, in most cases in other languages. 

I felt like, “Rails is the celebration of Ruby and it should be that.” Let's call other things other things. In fact, this whole idea of re-implementing Rails in a different language is a bad way to go. You should take what's special and unique about the language you're working in and use it to its best abilities. You don't get that by trying to re-implement Rails in another language. You get a poorer version of Rails. That's not what anyone wants. 

I remember I was writing Java at the time, and people were tying themselves in knots, smashing around Java, Bytecode, and stuff to try and approximate the magic of a scaffold tag or something. I remember thinking, “This isn't what the language is designed for.” You mentioned you had a contract to write Basecamp. Can you talk us through from an open source context, how did that work? When did you originally have the idea that this is a framework that should be abstracted out and published as open? 

The good thing about this contract I had to write Basecamp, contract is a fancy word that implies that there was something written down that both of us had signed, which was not the case at all. This was a verbal contract for me to do programming for hire to create Basecamp. I was the sole technical person on that. I was working together with 37 Signals, as the company was called at the time, which was three people. Jason Fried the sole owner of the company at the time, and two employees. He didn’t care. 

The reason he had hired me was that he wasn't interested in learning programming and he needed someone to help out with it. I essentially had all the options. I could do whatever I wanted, as long as what we ended up with was good and we could ship this application that we were building. I took that as essentially, “I'm going to do whatever I wanted.” I did and part of doing whatever I wanted was, let's start with this obscure Ruby programming language that you would never have gotten permission to use if you had asked someone on the merits alone. 

Do you want to use a programming language that there are 42 people in the US who will show up for a conference at none of them were commercially in the language? There are no libraries. I have to write everything from scratch. There's no support system and nothing. Do you want to use that for our new application? You were going to get a no which was exactly what people were getting at the time. This was why there were no commercial Ruby developers at the main Ruby Conferences because when you make the case to, let's use the stereotype here, the beancounters, the beans didn't add up. There were not enough beans here.

I never had to make that case, because it was solely up to me. Once I had decided to do it and fell head over heels in love with Ruby, everything else was almost on autopilot. As in, there wasn't that much considered thought to it. First, I was here. I was doing this because of open source. If I'd had to buy a database license, a web server license, and an operating system license, we would never have gotten off the ground. When we got started with this, I was getting paid $15 an hour. 

There wasn't money for this stuff. We couldn't have bought big sun servers, application servers, and all this other stuff. Our entire opportunity was created by open source. All of my learnings about programming were thanks to open source. Ruby itself was open source. Now what? I'm going to build something on top of all these things and I'm going to charge money for it. What? Even if that hadn't offended my moral sensibilities, it also wasn't practical. At the time, there were basically two entities that could charge for development software were IBM and Sun. Anyone else trying to charge for something like Rails wasn't going to happen. It was also the only realistic path to getting this out there and getting it adopted. 

As I said, I was head over heels in love with Ruby and one of my driving motivations for putting all this personal time into Rails was I want more people to experience Ruby. Why would I throw up barriers to that by putting behind some pay scheme? Secondly, I didn't want to be anyone's tool vendor. This was one of the things I stressed repeatedly, infamously in a slide that simply said, “Fuck you,” in one of the earliest conferences around Rails. It was in response to people treating me as though I was IBM, and they had a service contract and they could phone me up and say, “I want this thing implemented. Whatever you're doing is wrong. You should change that.” I was like, "I don't work for you.” 

The reason I do open source is I'm publishing this as part, sharing my tools, my poems, and because I want to. None of it is because I want to be someone's vendor, none of it is because I want to fix your bug fixes if they directly further the good of the framework overall, not because I want to do your features. No, this is open source, you fix your bug fixes. I'll fix mine and we'll trade. I'll implement the features I need, you implement the features you need, and we'll trade. That trading will end us up in a better place and that'll be good but I don't work for you. I’m not interested in working for you, and interested in getting paid. 

This was perhaps one of the other things, as you mentioned in the intro here is that there's a lot of different ways of making open source sustainable. I'd say in the past few years, the misconception that the only way to make open source sustainable is that the people who work on it have to get paid directly from that open source, has gained force in a way that's simply ahistorical and wrong as a sort of categorical declaration of it. There's never been more open source than exists now. There have never been more successful open source packages that exist now. There are plenty of packages, including Ruby on Rails, where the people who work on it are not interested in being vendor programmers. They're not interested in doing the Rails part and the Ruby part because someone pays them as a client. 

There's a lot of different ways of going about this. I went through it because I felt the moral obligation to do it, because it was fun, and it was a way to improve my craft. It almost felt like what publishing would do in academia. You have these ideas, you think they're good ideas, you don't know, let's put them out there, and let's get some peer review, which is one of the great benefits you get from open source software. It's almost like programmers are incapable of not sharing their opinions once something is like, “That code is wrong.” 

The best version of those opinions are expressed as pull requests. When it comes to specific implementation, there are other areas that people can contribute that are also valuable but when it comes to discussing the code, we can discuss it in terms of trading pull requests. We didn't even have pull requests back in the day. You were talking about version control systems. Rails started on CVS and I took pull requests on email and on Wikis. We had a Wiki at the time where someone would post pull requests, and I was the sole gatekeeper for the first year at least until someone else got the commit bit and maybe we moved to subversion shortly after we have more contributors. All the way back to CVS was how it started.

One of the things that's interesting is that a lot of the people that I've spoken to on this show, one of the common themes is that you write a database, you open source it, and 95% of the lines of code on that codebase are being written by people that work for the company. There's almost a fallacy of this idea of sharing. What you said is quite counterfactual to that. What do you think the reasons are for that? Is it, again, down to Ruby, the language? What was it right off the bat with dozens of people submitting code, ideas, and sharing as opposed to it’s basically a company working in public?

One of the key differences here is, as I talked about, I explicitly said from the get-go, “I'm not interested in being a vendor.” I'm not interested in dominating this space to the point that anyone who needs to use Ruby on Rails essentially has to purchase a service contract for me to be able to do that, at least commercially and grand scale, which is how a lot of commercial open source is done. They have a specific interest in dominating that because this is how you end up with those kinds of contracts, whether they're service contracts, teaching contracts, or consulting contracts of various kinds. 

When that is your business, you end up dominating that. Maybe that works out sometimes but that's not what I wanted to do. It's not the programming that gets me excited. I get excited about solving my own problems. I'm egotistical in that sense. Solving my own problems meant sharing those and that was an invitation that is far more inviting than I'm a company. I want to solve all of it so I can sell these commercial contracts on it. I said, “I am explicitly showing up with half a solution. Not half a solution, I'm showing up with 7% of the solution. These are the 7% that applied to me if someone else wants to show up with the other 93%, let's have an open source party.” 

That invitation works better and I've seen that in other open source projects we've done at Basecamp. Not all of them had the adoption that Rails did and some of it was because what we published as the first version was too good. It was too polished. It felt too complete. It's almost counterintuitive, that if you want to build an open source community around something, you almost have to release something that's slightly shitty. It has to be a little broken. It has to be incomplete because those are invitations for others to step in and join the party in a way that does not feel intimidating. 

When things are incomplete and broken, someone can very well sit down and go, “I could fix that. That's not hard,” versus you push out this perfect diamond that’s polished and all the ways. People are like, “These people seem to know much more about this than I do. I can't get involved with that.” That’s part of it. The other part of it is Ruby. The fact that Rails, when it was first published, was 1,000 lines of code. That included an active record, an action pack, and all the stuff you need for templating, talk to a database, and all the stuff you needed to build a web application. It was 1,000 lines of code.

That meant someone could sit down and read the entire framework in a day, maybe two days, and they could understand the whole thing. Read it and understand the whole thing. We're a little further away from that now after years of development and not 1,000 lines of code anymore but the readability of much of Rails is still approachable in a way that's not true in a lot of other programming languages. This whole idea, as we talked about, the reason I got introduced to Ruby was that Martin Fowler and Dave Thomas were using it to explain programming concepts. 

Someone can open up Rails when they hit a bug or something that doesn't work quite right, look at the code and probably understand it, possibly even create a pull request on it. To this day, the last version of Rails was released, Rails 6.1, had 700 or 800 different contributors to that version. Over the years, I believe we've had 6,000 or 7,000 people contribute code to it. The amount of code that's left in Rails that I've personally written, nowhere close to 7%, I don’t know, 1% or 2%. 

We ended up with different outcomes, in part by the design of the language, by the design of the community, and by my egotistical approach to open source and the fact that this wasn't my full-time endeavor. I'm not a full-time open source developer. I do open source when I need to and I don't do open source for months at a time. That leaves space for other people to come in and contribute and participate in a way it wouldn't be if I have dedicated 40 hours a week and I knew everything and built everything so forth.

It's one of the things I wanted to talk about or ask you was how you and Jason figured out the balance between writing Basecamp and writing stuff that you knew is going to be open. I've never considered that. If you leave gaps of time for people, they can come in and work on it. Whereas if there are five engineers working on an open source project for a company, there's never a breath.

That's a key part of it and part of it was that I've had mixed feelings about consulting-ware, which is essentially software built by consultants to be sold as consulting. The idea that we're going to keep going and there's always more work, I don't subscribe to that at least not when it comes to the feature side of things. Sometimes you end up with too much. It’s ironic, perhaps that I should say that, given the fact that Rails does so much and often is critiqued for perhaps doing too much. The amount that I contributed is the amount that I need. I don't build speculative software. I don't build software because I imagined that other people might need it. I build software because I have needed it and already built it. I extract it and I put it into Rails when there's been a proven need for it. 

This is the bar that we often go through with new requests too. How did you need it? Tell us a story about how you needed this? I'm not interested in speculative software, not interested in possibly maybe. I'm interested in definitely, and here's how. This is how it was worse if we didn't have this. The A/B test. Show me the code you had before then show me the code after whatever the thing you want to put into the framework is there. Is it better? Oftentimes code fails that test, even when we are earnestly trying to do so from real code. The amount of times that I've thought I had a new idea for something in the framework, built it out, and I do the A/B and it fails the A/B. I was like, “This is not sufficiently better to make it into the framework" is common. 

This is also what allows me to step away because I'm not a vendor. I can take six months off from doing Ruby on Rails if this was the commercial thing where you can't take six months off. You’ve got to show constant progress and so on. In terms of the split between what's open source and what's not, for me, it's clear. Anything that is plumbing, anything that's generic code that I could imagine using in a different application, than Basecamp, it goes to open source and Rails. Anything that's specific to Basecamp, doesn't go anywhere. This is closed software. I don't have a free software ideology attachment. I don't even have an attachment to open source, particularly. 

There's a lot of code where I feel, “Why is this even open source?” As we talked about this consulting-ware, a fair amount of it, I go like, “Does this even need to be open source? If there's one company dominating in any way, what is the point?” I understand that there are some benefits and you could theoretically introspect things, and you theoretically have additional freedoms and different options, but practically speaking, is that true? 

Databases, to some extent, have that to me. I've used MySQL for more than twenty years. MySQL is open source. I can never in a million years imagine me opening up the MySQL source code and doing something to it. That's just me. Having that practical approach to it is also why I love the MIT license. The MIT license is essentially me saying, “This is stuff I'm doing anyway. I'm putting it out there, if you want to use it, that's great. If you find something that breaks or breaks your application, that's your fucking problem.” You can report it and sometimes there are kinds of people who want to work on other people's bugs and that's great but mostly, you report it because you're going to fix it, not because I'm going to fix it for you.

I'm terrified to imagine how you were managing this community, source code, and pull requests before they were even, conceptually a thing. How did you avoid getting burned out and walking away from it all? Especially before, not necessarily GitHub, but when definitely before distributed version control? I'm getting scared thinking about that, especially I grew up with CVS and subversion. They weren't designed for that type of development work, right?

They weren't. Also, it's one of those things where you can have a present bias and you go like, “Look at all these wonderful things we have,” and you think, “How did people even cope?” I'm old enough that I've used computers with one megahertz of CPU performance. They were computers. They were great. They did a bunch of things. They entertained me and taught me things. You can imagine, and it's sometimes difficult for so called modern people to imagine ancient things. This isn't even that ancient, we're talking about things that are 30 years old. It was fine. It wasn't a big deal. Yes, the tools have made it easier and tools have made it easier to scale but in the beginning, we also didn't have that scale.

I was not processing patches from 700 people. There were, in the first year, 50 people who had patches into Rails. There was probably a high mark for it, so it wasn't that big of a deal. In fact, to some extent, it provided something else. It provided me the opportunity to set a path to pave that path. This is what Rails is going to be. Before it got flooded with contributions that were trying to pull it in 500 different directions, this was one of the things that were a major part of the work in the early days. 

It was not about reviewing code but around battling philosophies and values. What should Rails do? How should it feel, what should go into it, and what should not go into it? This was what a lot of the discussions were about. Because it was still so new there were a bunch of people who showed up and are like, “I like this rough thing. I want to take it this direction. I want to take it over here.” I had to, sometimes with less manners than perhaps I've refined now, say, “No, that's not where we're going to go.” This is my vision for what a framework is going to be and if you want to sign up for that mission, or if you want to complement that vision in ways that I can stand behind, let's do it. If you want to do something else, go off and do something else.

Were there ever any forks that made it past the first month?

Yes, one. Out of probably hundreds of either you call it proclamations or threats, but there was one, it was called Merb and it happened around 2008 maybe. We merged it back into Rails in 2009. That was a fascinating story in and of itself and it taught me a lot about what gives rise to forks like this and it's much the same ways as what gives rise to rebellions of all kinds. Do people feel heard? Do they feel represented? Do they feel the issues that they care about are respected? Through error, miscommunication, at times a bit of malice, that ended up not being true for a bunch of people who wanted it to be true. That was the thing that paved the road for the Merb reintegration. When we finally sat down and talked about the issues, we realized that there was no difference of opinion on a broad scale. 

There were a bunch of people who cared about different but complementary parts of a framework, people who cared about making the framework extensible, configurable, and certain parts of the performance that I didn't care about. I expressly didn't care about it but I want it to happen. If someone wanted to show up and do the work, I'd want it to be with open arms. I haven't done a good enough job of communicating that and making that clear that it was okay to show up and part of that was, as I said, my crude at times manners with contributors who wanted to pull things in different directions. 

What I didn't realize enough at the time was that anytime you have an interaction in public, you have an interaction between you and the person you're talking to and there are possibly 10, 100, 10,000 spectators, and they will all take certain things away from that public interaction and form their own opinions about whether it's worth their time in the future to engage in likewise. I didn't appreciate the extent to which that was true and the extent to which people I had never even talked about these issues with thought they knew exactly what I thought about them because they had observed other interactions and drawn their conclusions and their deductions from it. 

We finally sat down contentiously so in the beginning and hashed things out. We ended up thinking, “All these things that Merb cares about can totally fit into rails. Why are we re-implementing all the parts of Rails that we are already in agreement upon should be like they are so you can add a bunch of things that could be in Rails as it is?” That was the one fork that made it far. It turned into a whole framework. It was a whole community at the time and the reintegration was difficult but totally worth it and Ruby on Rails as a community is so much stronger. Not because we are in this weird setup, in part that Rails is the only web framework in Ruby of large-scale community success or whatever. I don't even like the word success. Let's just say mass because there are other frameworks in Ruby. 

Sinatra is a micro framework that's been quite successful, and it can be complementary to Rails and there are some other things, other frameworks in Ruby that are more of direct competitors to Rails as it were. In terms of the broad scale, mainstream adoption, Ruby and Rails have an equation between them that's not true in most other ecosystems. If you look at Python, for example. There are multiple choices, which are not clear in the same way that is clear in Ruby. That has its pros, and it has its cons but we've done well and we've done well for Ruby because of this. 

I'm thinking back to the screencast that you released in 2004 or so, where everything went crazy for you for a while. How did you come up with a design for managing it and delegating decisions, ownership, and commit rights? Did you put a lot of work and thought behind that or did it happen organically?

It happened pretty organically and part of it happened because I walked into the situation with pretty strong opinions about certain things about what I cared deeply about, and I wasn't willing to delegate. Most of the API design fell into that category. I have, to this day, specific opinions about how an API should feel. I have less specific opinions about how most APIs should be implemented. In many cases, I barely care. You look at the router or even large parts of the active record now, I couldn't tell you how they implemented it. I could probably sit down and figure it out but I couldn't tell you how to implement it but all of the APIs that sit on top of that I've cared immensely about them. 

I've finessed them and polished them over years and years of collaboration with people who did care much about the implementation. Having that clear delineation of what do I care a lot about? What do I not care a lot about? How can we work together in that way? It made it easier to get people on board, because I said, “There are thousands of things you could care about. I care about these 30.” Anyone else could show up and claim any of the other spots and you will be welcomed. You will be welcomed by simply showing up and doing the work and doing the work is showing up usually with a pull request but this was even before the days of pull requests, but showing up with a patch, the Wiki page, the mailing list or whatever.

You show up with a patch, you argue for why it should be and I can review that code and be like, “That's good. Let's put that in.” The other thing with commit bits was essentially encouraging people who showed a little bit of interest to go further. Someone shows up with a good patch. It's small, but I see that this person gets it. They have the same aesthetic or at least they can be guided towards the aesthetic and they're pleasant to work with. These other things. Let me try to rope them in because the more I can rope more people into it, the easier it is for us to make progress faster because I will make progress on the few things that I care about and as we talked about and that will be the 7%. 

If we want to get to, if not 100% or 80% of what someone needs, we need more people to do it. I took that quite seriously from the get-go. It's also more fun. It's more fun to make software together. As we said, this sense of peer review is important and peer review works more pleasantly, I found, when you do you review with people you trust. It's easier to trust people who you've worked with repeatedly over a longer period of time than strangers who show up in a pull request and start complaining. 

It happened quite organically, I would say but it still wasn't random. It wasn't un-thought through but at the same time, we didn't set it up. Rails, to this day, still does not have a foundation in terms of legal organization that stands behind things. In terms of a bureaucracy to process these things, there wasn‘t that. Our license as I said, was essentially the barest bones license you could get away with, which is MIT, which at the time when Rails picked, it was not the default in some sense that is it now. There were much more stringent licenses. 

GPL had a huge influence at the time. I looked at that part of open source and I was like, “That’s not the part of open source that I’m interested in. I’m not interested in extracting contributions from people who would rather not give them to me.” I’m doing this for the fun of it. If I want to twist someone’s arm and be like, “You better give me all your extensions to your framework or else.” No. In fact, what I found is if people showed up with this code dump like, “Here are a bunch of things we made. You take it.” I’m like, “That’s a bag of trash. I’m sorry.” 

After 18 years, the readability of much of Rails is still approachable in a way that's not true for a lot of other programming languages.

It could be turned into something but right now, it’s a bag of trash and I don’t want the framework. I want the framework to be pristine for as long as possible. That was the other part of it. It’s like, “Yes, I want contributions from people who are like-minded, who want to be on the same path, want to share the same vision but I don’t want contributions from people who are not on the train.” It’s not that you have enough contribution and that makes sense and we should carry it. There are plenty of contributions and half of the contributions over the years have gotten rejected in the sense that this is not where we want to go. More code is by no means always better code, more features, more configuration flags, and everything is not the way to get a better framework. You need to have some crystallization of a vision. Know where you’re going, drive it from there, and invite people into the tent.

The tent is defined that the people inside are there because they want to be there and they enjoy the same kind of show. When people don’t want that they can go to another tent, there are so many tents. One of the things I learned over the years. In the beginning with Rails, people would ask, “When do I not use Rails?” That was one of the earlier questions, “What are the limitations?” Objectively for me, the answer to when to use Rails is always. 

The reason for that is, I have designed Rails for my brain to the fullest extent. The class of problems I work on are all solvable by Rails. You’re not going to find this objective definition of when Rails fits and when does not fit because, for everything it was designed to do, minus is 5%, it is an answer. It is a great answer but it doesn’t mean it’s the answer to everyone because there is are plenty of people who don’t have a brain that works like me and don’t think of themselves as software writers, who very much think about themselves as software engineers. 

For example, think of type safety as a baseline requirement of a language and anything else is a scripting language, a toy language, or whatever. You have these dividing lines where Rails is the correct answer to none of their problems not because of the problems but because of them. That is something I’ve come to appreciate more, but not something to appreciate that people are different but as in, “The world is way more interesting when people are different and when we don’t all write Ruby.” It’s so fascinating that we have an amazing software platform called the Web that allows people to bring whatever programming language they want and they did. I’m pretty sure that there is a web page somewhere on the internet that is powered by every single programming or language that has ever been invented. I don’t know what the count of that is but it’s high. Isn’t that awesome? Isn’t that wonderful? 

Part of what makes the world so interesting and also what makes it possible in a language like Rails and Ruby on Rails is to be strict in the vision we pursue because we can go into that and know, “This doesn’t have to work for everyone.” Not all learning types and brains have to be compatible with this because there are options they can do. We can take this vision that we have and say, “It appeals to maybe 10%, 20%, or 30% of people out there and we can make it the best it possibly is.” It doesn't have to appeal to 100%. In fact, if we try to make it appeal to 100%, we will end up with this mush that doesn't taste of anything and needs enterprise salesforce to push it because no one would want to pick it by volition.

Are there any languages, projects, or frameworks that you see? I'm feeling like there seems to be a vibe of the Ruby community that you talked about that seems to be reflected a little bit in Rust, to a degree in terms of people doing it out of enjoyment rather than professional necessity. Are there any things that you see Rails in?

Rust is a good example. There are several members, or people who've worked on Rust, who used to be in the Ruby and Rails community and some still are partly in both camps. Rust solves a problem that I don't have. I don't do any system-level programming, so it's not something that I've particularly engaged in but I love seeing it. I love this idea that even after all this time, you can still make a new programming language, and you still do something awesome, new, novel, and take a different spin on things. It means that there's still progress because one of the things that I thought when I first started Ruby on Rails is technology doesn't last as long. It’s probably a five-year thing. In five years, we'll be done. We'll be on to something else. Ruby won't be an answer anymore because if you looked historically speaking, it happened a lot. 

I was thinking about how frameworks have a half-life of two years.

They still do. One of the communities that have the biggest framework churn is in JavaScript. The vast majority of things that get announced today are forgotten next week. I thought that it was most likely that Ruby and Rails would go through the same thing. We'd have a 2, 4, or 5-year run and that'd be good and here we are 18 years later, and not only has that not happened, but we’re also stronger than we've ever been. We're moving faster than we've ever had but that's a rarity and exception. 

Even if that's true, I still appreciate this idea that there is fresh blood and ideas that are new concepts. They're radically different and Rust is quite different in quite a lot of ways from the system-level languages that it's competing with are trying to replace. I love seeing that. Another example of this is Elixir. I absolutely love the influx of that. You take something old, Erlang, which at the time was niche in much the same way that Ruby used to be niche. Perhaps it was less niche, well known, or more well-known than Ruby was because it had been used to implement big things with WhatsApp and so on so forth but it was still not something a lot of programmers were using. 

Some of it was because of the ergonomics of the language and here you have Jose Valim come in, “I'm going to take all the things I learned in Ruby land and I'm going to combine it with five different influences I have. I'm going to turn Erlang into this other thing that has better ergonomics and is more appealing." I love seeing that because it also felt that was what Ruby was and is. There's a postmodern language in the sense that it isn't a single paradigm. 

Ruby goes like, “Functional programming. That sounds great. Let's have some blocks. Let's have some of this, that, and the other thing. Let's import that in. Object-oriented program. That's great. Let's have that too. Procedural program. That's great, too. Let's put it all in.” It's a patchwork language that takes influences from many different sources versus there are other languages that are pure. Functional programming is one of the things that in the time I've been a programmer went from totally niche to quite mainstream. Some of those functional programming languages are single paradigm. It’s no everything, no side effects, no this, no that, and the other thing. 

I love to see these things where you go, “There are some influences of that.” That means that we're not done because wouldn't that be boring? It wouldn’t be boring where we’re like, “The perfect programming has been invented, we might as well all go home. We don't need language designers anymore. We don't need any new ideas.” In some ways, I'm disappointed that it doesn't happen more often and with more impact. In some ways, I remain head over heels in love with Ruby. I am surprised. How is it that it’s Ruby? Ruby is from ‘95? How are we still writing Ruby? It so happens that those evolutions happen inside of languages too. Ruby itself has gone through quite a lot of evolution over time and it had a base that was flexible in a different way. I remember when I first implemented a new method on string. 

It's a mad language, in some respects, right?

It's completely mad. In the dominant respect as to how people used to look at languages. If you go back and read some of the stuff that James Gosling wrote about Java and about how you needed to protect programmers from themselves, so on so forth. If you go in, and you nod your head at everything he's saying, and you look at Ruby, you go like, “This is a madhouse. This would never work. People are going to override string five different ways from Sunday. It's all going to trample over each other. You can never build large systems.” 

That's what's so great about this. We got the answer to the skepticism, which was wrong. You can absolutely build huge systems like Shopify. By one account, it had a million lines of Ruby in their application. There are a million lines of Ruby running an enterprise worth $140 billion or something. I guess you can scale pretty hard to several thousands of programmers working on that. They probably have their issues, as does any organization that tries to push any program or language, or environment to those scales. 

It's been fun to see those academic disputes play out in fact and in reality, which we talked a little about but the remote thing has been the same thing with remote. Right up until 2020, people were legitimately saying remote doesn't work. It doesn't scale. Unless you have a water cooler and whiteboard, you can't get creativity, you can't get serendipitous interaction. It doesn't work. Reality forced a verdict on that bullshit and the verdict was, “It's bullshit. It could totally work,” as we talked about open source had proved for 40 years, but now the proof was definitive. 

Ruby brought a bunch of definitive proofs to programming that were novel. You can focus on the programmer first to the sacrifice of the computing cycle and still create a language that's not only fast enough but the speed that you produce is the speed of the programmer’s ability to implement code. That speed is more valuable, particularly directionally speaking for the industry at large, which was totally not a thing that was the case in the ‘80s, or whatever. 

They were trying to make Lisp machines and Smalltalk machines. You couldn't do it and so forth. With Ruby, it bet on the human. They’re like, “We're going to make our bet one way or the other. Are we going to bet mostly on the computer or mostly on the human?” They bet on the human. It’s an overwhelmingly successful bet. Computers have never been faster, cheaper, and programmers have never been more expensive. That fundamental economic calculus turned out to be exactly right for the vast majority of web applications, which is the domain that I ply my trade in.

I'm conscious of closing things out. I know, you've had interesting few months having a spat with the second largest company in the world and all that stuff. Give us a brief overview of what you're working on, and how your new product is going?

Over the summer of 2020, we launched a new email service called HEY.com that we've been working on for several years. We've invested millions of dollars into it. It's created an absolute faucet of open source releases. On the front end, we've released something called Hotwire, which is novel in its presentation and its breath of an alternative to the single page application approach where you use React or whatever. We’ve spun off a bunch of things off of it but it almost didn't come to be. The day we launched HEY, we had an app in the App Store and we submitted an update to that app and Apple rejected it because they wanted 30% of our revenues. 

We went into an existential fight with the biggest most valuable technology company in the world and that lasted an utterly grueling two weeks or so where we battled Apple on all sorts of fronts, mainly in the medium, on Twitter, and elsewhere. We finally arrived at a reprieve that we won the right to exist without handing 30% of our revenues to Apple, in trade of some other minor yet still annoying concessions. That whole experience further emboldened my view that we're in a bit of a rough spot right now with technology. It's all due to monopolists that have power. I remember the Microsoft monopoly power. I remember the Windows monopoly power and it almost seems quaint in comparison. 

It seemed impossible to get out of at the time. I remember thinking no one's ever going to defeat this. 

More code is by no means always better code.

A thousand years of darkness and it did get defeated. One of the reasons it partly got defeated was because the American Department of Justice launched a lawsuit against Microsoft. It reset what Microsoft thought they could get away with or what they could do, whether they could just cut off the air supply to new, emergent platforms. It gave the breathing room for companies Google and Facebook to come up and exist. We're in need of another grand reset like that. For several years, I thought that it's not coming. We're in another thousand years of darkness. 

All of a sudden, just in the past year, you can sense the light. You can sense that the sun is below the horizon but there are already rays coming up. Perhaps it doesn't have to be a thousand years of darkness. Perhaps we can push back against these monopolies and bust this death grip they have over the distribution of software when it comes to mobile devices. This is what made it an existential fight for us. We've been a web company since the beginning and to some extent, taken for granted what an utterly historical anomaly the web is as a software distribution platform. 

The freedom it allows creators to reach consumers directly without mediation, without gatekeepers and we lived through the glory years of that from let's say, the early 2000s through the early 2010s when the mobile devices and platforms became ever more important to the point where they're now essential. If you want to launch a mainstream major service and you are not present on mobile devices, you're not a serious consideration for the vast majority of the mainstream consumers. This was exactly what we faced. We'd built this wonderful email platform. 

We had built it with six different clients but the two clients that mattered the most, in fact the client that mattered the most for our demographic of trying to sell commercial email services for money, not for privacy, not for attention, not these other things. It just so happened to be that the majority of the customers for that service were on Apple. About 90% of the customers that we have for HEY, use at least one Apple device. Something like 70% of the paying customers use Apple devices predominantly.

It’s bananas. It's a completely dominant market force to the point when people say, “You don't have to do an iPhone app.” In the same way that if we don't have to have 70% to 90% of our customers. That's not a choice. It’s the same false choice that people were presented back when there were other monopolies. In the US, the railroads were one of the main monopolies that the state power took on and the argument then went, “You don't have to use the railroads. You have some cotton or barley you need to ship from one area to another, donkeys still exist. Carrier pigeons can probably carry a few grains. You could do something else. You could walk it there. You're going to have to use the railroads.” Yes, you do. 

You can't be competitive if you don't use the railroads. Today. Apple and Google control the entire railroad system of software distribution for the two only mobile platforms that matter. They have 99.9% of the market share between the two of them in the US, at least. I know there are other things going on in Asia, China, and so forth. It's funny because Jason said at the time and I felt the same pain. This was the thing. This monopoly power was the thing that if I'm going to retire, if this is what it's going to be, if I can't make software anymore without having to growl in front of Apple or Google and give them 30% of our revenue, maybe I’ll pick up a spade and start farming. I’ll do something else.

It's interesting how much we took the web for granted back in the late ‘90s and early 2000s. It was a glorious thing. Pretty quickly, that was the safe assumption. As time goes on, there's definitely been a divergence there. The problem that there is now is those antitrust questions and those questions about what you do are much more difficult to solve, than if you're trying to break up Standard Oil or something like that. You don’t think?

Maybe on the App Store, not so much but definitely around the question of, what would you do with Google, for example, or what would you do with Facebook, WhatsApp, and Instagram?

I don't think the questions are as difficult. It behooves the tech giants to make it seem the remedies are difficult. You take the app stores, for example. Barring the app stores from forcing applications to use a certain payment processing would solve the 30% question overnight. If we were simply given the choice, “You can use the Apple system, which by the way, is a pretty great system. The credit card is already on file. It's a single touch to pay. It's extremely low friction.” For lots of applications, that's the right way to go but it becomes abusive the moment it's not a choice. When you say, “You have to use it,” so you could ban that. You could simply say that the app stores do not have the right to mandate that this has to go through that way. That solves the app stores. 

Google, for example, one of the ways that has been thrown around to promote the growth of alternatives is A) prevent them from becoming an ever larger conglomerate. This applies to all the big tech companies. Don't allow them to use their existing monopolies in certain areas to gain new monopolies in new areas. Bar them from mergers and acquisitions. Don't let them buy new, nascent competitors. Unwind the acquisitions done so far. Facebook should never have been allowed to buy Instagram or WhatsApp, whether they were smart acquisitions at the time or not is irrelevant. Break it up. Force those breakups. 

This is the Standard Oil playbook. You break up companies that have too much integrated power and suddenly you restore an open marketplace where there is room for competition in a way that isn't right now. This is the thing that gets me. Once you start reading about antitrust history, particularly in the US, you go like, “Everything old is new again.” It's the same thing. It's the same fundamental clashes between power and democracy, particularly in the US, where you have a political system that's so dominated by corporations, funding, and so forth. 

It is a struggle that is more than technology that once you have companies that are this large and this powerful, they also suddenly are vested with the jurisdiction to police all expression on the internet. Is that great? Do we want three private companies dictating who could say what and when and how? No, we probably won't. A lot of that ties back again, to the monopoly question, which ties into whether it is possible now to build something to compete with them. 

To a large extent, it isn't unless we have something that upsets the cart and that's what hopefully we're getting now through the legislative action through the enforcement action that people are going, “What we have now is not a good place. We don't want more of that.” The irony is that in the past year we've gotten way more of that. All the big tech companies have gotten dramatically more powerful and rich in this time, during this pandemic, and we've managed to wipe out ever more small companies that should have been the ones who are going to grow up and give them a fight. 

That’s one of the things that I care about a lot now and it feels like it's one of those things where if you were an open source advocate back in ‘95 and you were looking at Microsoft, you'd be like, “This is impossible. The Office monopoly is too strong. How can Linux ever have a chance? How could anything else ever have a chance?” All of a sudden, there's a fork in history, and the DOJ launches their case and Microsoft gets thrown off their balance. The web comes up and not a thousand years of darkness. Hopefully, that's what's happening now, but also possible that doesn't happen. You look at all these forks in history and you go, “In many ways, we turned out quite lucky but the odds are it could have gone the other way.” 

I don't want to make the comparison in terms of monopoly but look at 1945 and ‘39, the world could look quite different if things had gone slightly different. There’s this idea that there's a historic inevitability that things are going to go right and in the end, “The good side is going to win." It's absolutely not guaranteed. We have to fight tooth and nail every day to make that happen, both in terms of open source and the ability to use software largely. We go with open source. I mentioned earlier that I'm not that much of a free software activist or purist. Has there ever been a greater affirmation of all the stuff that the free software movement talked about back in the ‘80s, and ‘90s than what we’re living through?

That's pretty amazing how prophetic that was because, at the time, everyone was like, “This guy is nuts.”

Totally. Perhaps he still is in other ways but sometimes you do need to be a little nuts to be able to clearly see what others aren't seeing because the paradigm and the normalization, the well-adjustedness is adjusted to a certain paradigm, a certain society. If you're outside of that, you can diagnose it in a way that looks prophetic. Absolutely, some of those diagnoses were prophetic and it's one of those things where you can't listen until it's too late. It reminds me a little bit about the pandemic response. Anything you do too early looks alarmist. Anything you do too late looks negligent. We're the negligent phase of tech monopolies. 

David, that's been super interesting whirlwind. Thank you so much for your time. Thank you, again for Rails. I remember seeing the percentage of Y Combinator companies that were built on top of Rails, and it was 60% or something. Thank you from me. Good luck with HEY. Hopefully, I can pay you without paying Apple at some point in the future.

Hopefully, that's part of the equation here, too. It’s to get people to realize that they can vote for the future they'd like to see with their wallet and it's not all of it. It won't bring total relief, but it will bring some relief and we're trying to offer at least some options in that regard.

Thanks again for your time and stay safe.

Creator of Ruby on Rails, cofounder & CTO at Basecamp, best-selling author, Le Mans class-winning racing driver, frequent podcast guest, and family man.

Founder and CTO

On this episode of The Craft of Open Source, I was able to interview one of the most influential characters of the open source community, David Heinemeier Hansson. David's work has been well chronicled over the last few years and his recent spats with Apple have been nothing short of Twitter Gold. I personally learned about David through the massively successful Ruby on Rails project which he created and launched back in 2004. 

What stuck out to me in this interview was his level of passion and the underlying reasons that drive him to do what he does. 

This was an extremely interesting conversation where we touched on the beginnings of Ruby on Rails, his thoughts on Open Source and what he has been up to in his commercial endeavors. 

I hope you enjoy this one as much as I did.

It’s great to have you with us, Quinn. Do you want to give us a bit of background about who you are and about your business?

My name is Quinn Slack. I'm the CEO and Cofounder of Sourcegraph. We make universal code search for companies that have code and anyone who works with code. I'm a coder myself. I love coding and have felt what it's like to work on a big open source code basis. I've seen what it's like to manage software projects inside JPMorgan, Bank of America and tech companies. My Cofounder and I ended up coming back with one thing, which is we love code search, and yet most devs, most companies never even heard of it, never had it, never knew why it's awesome. We started Sourcegraph to get code search to every developer and every company.

Did you start the segment, do you think almost?

There are code search tools that existed before Sourcegraph and others have come up in the last ten years. We started Sourcegraph in 2013, but we have OpenGrok from Sun Microsystems, now Oracle OpenGrok. We have Etsy created Hound, and you have Livegrep. There are other code search tools, but there wasn't a code search tool that was like the one that Google has internally. Google has this amazing internal code search tool, all of their devs love it. It's got code intelligence and code insights. It lets you make campaigns its large-scale fixes and it's called code search, no other name. If you're a Google dev and you quit Google and you go to another company, a lot of times the thing that you miss most about Google is not the micro kitchens or the fresh salmon every day at lunch, it's this code search tool. We wanted to bring a code search tool like that super high quality, that tool that people love. We wanted to bring that caliber code search tool to every other company out there so that it's not just Google's secret weapon internally.

We got into a private ex-Facebook group because they have a similar story with their feature flagging tool. There were people leaving Facebook and going like, “How do I do this now? I don't have this amazing tool.” Did you have a stint at Google where you were exposed to that?

I personally was not at Google. My cofounder was at Google and we have other team members from Google. Facebook is another company. It's Google and Facebook that have internal code search tools. Google's has been around for a little bit longer. It's a little bit more advanced, but Facebook has Big Grep also known as TBGS, The Big Grep Search and they have a pretty awesome developer platform internally. Ex-Facebookers also have the same sentiment.

It’s interesting that they don't open source those things as well. That's a big signal to me.

I don't think that they're refusing to open source it because they want to keep it internal. I think it's because the codebases of Google and Facebook are special. They do things in their own way and it wouldn't be that helpful if they were to open source it. They have open sourced some nice components. Google has open source libraries for doing engrams that make code search indexing easier. Google's got a single massive monorepo. They've got so much tooling, same with Facebook.

It would be very hard for them to take what they use internally and throw it over the wall. It would be a ton of work. One thing we found is that code search for a company is a different product than code search across all public code. We didn't realize that when we were getting started. You think, “It's search.” The scope of what you're searching across changes the implementation, what the search UI looks like. As a company, we decided, “Let's make code search for a company amazing because then we'll know that we've got a business model that we can then grow, and then we can then make good public code search.”

I cloned your repository and had a look at commit number one, which is always interesting. Do you know what commit number one was?

It would be from hello it's Sourcegraph and making it open source in public.

Published Sourcegraph as open source rocket emoji. You had a fairly established codebase at that point by the looks of things, is that right?

It has come a long way since we made that commit. We had a basic code search then.

What was it? I did the same thing for Ansible and the first Ansible commit is amazing because it's like 1 file that's 300 lines long and it goes from there. You guys already had an existing product that you then open sourced. What was the history behind pre that commit?

In Sourcegraph, we did start out first by making public code search. You could go to Sourcegraph.com and search across open source code. You could do find references and go to definition. You could find use examples for this open source library you're using, and that was great. One of the cool things about that is sometimes you'll be working with the library. You'd be looking for use examples and you'd realize, “There's this other library that's an even higher level that does exactly what I want. I can use that instead.” That was good, but we are a business and we want to invest a lot in making amazing code search. You can't make money directly on public code search, and we want Sourcegraph to be useful for companies.

We started by having a cloud service for Sourcegraph where you can go on Sourcegraph.com, you could sign in via GitHub and click that big scary green button that would OAuth all of your code up the Sourcegraph and then we would index it, and make it searchable. What we found out is that no one trusted us with all of their code. Why would they? We were a tiny little company. I think our team is and was trustworthy, but why would anyone believe us at that time? Certainly, the kind of companies with a lot of code that needed code search the most, they are not going to be trusting us. We decided, instead of trying to offer a cloud service, it didn't work. We decided to go self-hosted and self-serve so you can take Sourcegraph, do a Docker run command and run it on your own network.

You get code search on your laptop or on your own EC2 instance. You don't need to send any code off your network. That's when our business began to take off. We decided to also go open source so that developer who's thinking, “Should I or should I not try Sourcegraph?” It gives them a little bit more trust in us. We're a little bit more transparent. We've also done some other things in our business. We've got a public employee handbook. That's the thing that documents all of our processes, and that's all public. We want to be transparent to the win that developers’ trust and to communicate to every single new person that joins our team that we want to do the good thing. We want to find strategies that force us to do the good thing.

That makes us always think about, “How are developers going to perceive this?” With that said, we are not 100% open source. We adopted a model that's similar to Android or Google Chrome or VSCode or GitLab where we've got some open source code and non-open-source code. For us, those both live in the same repository. The non-open-source code is in the directories’ main enterprise. It's clearly delineated in the licensed file when customers or anyone runs their Docker image that's built from the enterprise sources contains a combination of open source and enterprise code, and that can access our online extension marketplace. You can get all these editor language extensions for code intelligence. That's like how the underlying base of Android is open source, but if you want the Android Play Store, then you've got to work with Google. 

With VSCode if you want to get the extension marketplace, you've got to use the official Microsoft build. With GitLab and with Chrome, similar situations. That's the model that we ended up on. Some people ask like, “This Sourcegraph repo on GitHub, is that what you all work in?” Yes. You join Sourcegraph as an employee. On day one, all of your PRs, all of your issues, everything is going to be in that. We're very transparent and public there.

At what point did you start talking about, “Maybe we should publish everything on GitHub?”

It was several years ago, and it felt like such a big decision. We were maybe 10 or 15 people at that time, and we had a few customers and it felt scary. It's like, “Is this an irreversible step?” We think it makes sense for a lot of reasons, but maybe some massive company is going to come along, take all the code, repackage it and offer it. It felt scary but in hindsight, it was such a non-event, it made it easier for us to commit code and share links.

Open sourcing something, people don't care unless they care about that product. We were very early as a company. Not that many people cared about Sourcegraph as a product back then. It was such a non-event. Even if it is a big event, even if it's some big famous company that chooses to open source, that data they open source might get to the top of the Hacker News, but that's a one-time thing. Open sourcing is not a big splash anymore. There are some people at Sourcegraph that thought that it was going to be a much bigger deal. In the end, it was business as usual the day after, and it doesn't come up a lot.

We're waiting for the massive spike and the hoards to come and that didn't happen. We worked on that. We knew that we would get there through an amazing product that a ton of developers used and loved, not through the legal framework under which we released our code.

That then enabled developers to do Docker run or get put in and run something locally on their machine to kick the tires?

We did publish a Docker image before we had an open source release, but making the code public and open source showed developers a little bit more trust on the margin. Most people are still using our prebuilt Docker images. There's a self-hosted, and that is a very important step for us as a company, which let us get bigger customers early. There's open source. Self-hosted, that was a much more momentous decision for us. Open source, on the margin, makes that developer trust us a little bit more when they're thinking, “Do I or do I not use Sourcegraph?”

It’s interesting because most of your back end looks like all of it is written and Go right. We’re in Python. If we were to distribute Docker images whilst having closed products, that would be weird because it would be trivial to steal all our stuff.

Open source is not really a big splash anymore. People don’t care about it unless they care about the product.

Obfuscating code feels weird. GitHub enterprise does that that I'm not sure why.

You must have spent a good few hours making sure there weren't any API keys hidden in there before you published it.

You should be doing that anyway if you have a private repo and you can use Sourcegraph to help you do that.

How much time did you spend thinking about licensing and what license to choose?

That was a fairly easy decision. It was, “Do we go copyleft or not?” For our customers, every company with developers, we'd like them to be our customer in the future. We'd like to have a broad customer base. We'd like to have something that is unobjectionable. We chose Apache2. We didn't feel like copyleft was the right choice for us. We also didn't feel that we needed copyleft as a thing that might scare companies into needing to get a commercial license with us, because we had other elements. What we chose to keep on the enterprise side and not open source, that was enough of differentiation between the paid, free, and open source product. Apache 2 and in our rebuilt, it's not half Apache 2 code and then half non-open-source code.

That code is still in the same repo. It's still public and visible. The licenser doesn't matter. It's like, if you go on your blog and you publish some posts, that post is copyright, the default of creative expression, published is copywriting. Technically, if a GitHub repo is out there and it doesn't have a license, that's not open source, that's copywriting. The other 50% is not open source, but that's under an enterprise license that clearly states that. There's no particular magic in that license.

In terms of the marketing of a company, did that change after you went open source? Were there unexpected benefits around that from a marketing perspective?

I wish. It had a surprisingly small impact. We need to market why code search is awesome, why you need to be using it, how it's going to make you a better developer, what are all the other companies that use it. That's what we need to be marketing and our code being open source or not, that doesn't matter to people. There might be something here where we are an end-user product. We are Google for your code, code search. We're not some library. We have an API, but we're not some API that people integrate into their own code. End-user applications, people do treat them a little bit differently. You don't go to GitHub to discover end-user applications.

You might go to GitHub to discover libraries, open source companies that offer something where the key entry point is the library. For example, a feature flagging company, that's a library that you pull into your code. GitHub is probably a lot more effective for discovery there, and having an open source repo is a lot more effective for discovery, but not for us as an end-user application. Or maybe we're being stupid about it and we could be getting a lot more out of it.

I'm still learning. We find that they have a benefit to each other. You can try and get traffic to your website and that can then take a look at your GitHub repository and then vice versa. It's interesting because I haven't spoken to many of the people for this series who've had that big close to open splash. For a lot of the people I've been speaking to, it's always been how they've operated. There are also soft things around validators. You go into your website, see the GitHub logo, click on it, see that the whole thing is open especially as time goes on, it seems to becoming more common and has more weight within the hive mind.

I strongly prefer using code that I can view and it’s open source. I get that.

In terms of contributors, I see you've got 149 contributors. Was that like as soon as you opened it up, you were getting pull requests? What were your expectations around that?

We had no expectations whatsoever. Any contributions that we would get from the community would be a pure bonus. One benefit is we got a lot of customers, they report issues and we love it, and they love it. They go and file a GitHub issue where sometimes they'll email us or in Slack channels, they'll mention it. It was nice and we can always share a GitHub issue link with customers and show, “Here is the problem that’s been reported. We're working on it.” That's a nice way to keep us on the same page and to establish trust so that they know we're not some black box, that we're transparent with issues that they've reported. We weren't expecting any contributions. I think this is another difference where we are an end-user application and those things get fewer contributions.

You could imagine a company who is thinking of, “Should we open source our code? The secret sauce.” Someone who said, “It's going to be such a big splash. We're going to get many contributions. People are going to be building our product for us.” That's not going to happen. We were very conservative by estimating no external contributions. In the future, we will get some more, especially as we become not just code search for the company, but public code search. We've got around 80 people at Sourcegraph, multiply that by two, and that's the total number of contributors we have. We're going to be the key driving force behind contributions.

It’s rare for someone to send a PR that's a significant feature implementation or something like that.

That's been a common thread from everyone that I've spoken to. Maybe a little bit of a shame. I find that there's a huge amount of value in someone raising an issue. Like if there's a bug in your code that you weren't aware of, you wouldn't have been aware about it quite possibly unless they have gone on to GitHub and done that. There's still a tremendous amount of value in that. The vast majority of your code basis has been written by members of the company. 

One of the things that we've had a lot of trouble with, especially from a self-hosting point of view, how you strike that balance between, mainly from an infrastructure perspective on making the platform simple to self-host and balancing that with wanting to use some core piece of infrastructure that you wouldn't necessarily want to have anything to do with if you were self-hosting. Is that something that you've come across frequently?

Our principle is we want to deliver Sourcegraph in the way that it would be delivered if it were an open source tool. That means trimming down the number of primitives for persistence and deployment that we might rely on. We have Sourcegraph rely on Postgres with a fairly wide version range, Redis, and block storage or object storage. We don't need all the performance assumptions there. That's easy to deploy. We've got a Docker image that will bundle up Postgres and Redis in the same Docker image. We have a lot of customers that will use Amazon, RDS or Google Cloud or SQL to Postgres in their respective cloud offerings of Redis. They don't even have to have a Sourcegraph cluster.

You can deploy Sourcegraph on Docker and Docker deploys in Kubernetes. It means that we can't use the hot new AWS API or service for some complex thing, but that's okay. We keep it simple. That's how some random person working on a random open source self-hosted single-tenant tool would need to build their tool. That feels natural. People massively overthink how to deliver a self-hosted software when their mindset is like, “This is our company's product.” They think about wrapping it in all of these super complex deployment mechanisms, but no, keep it simple. Ship ML files, ship a Docker image and rely on a small number of primitives.

Was that a philosophy you had pre-opening the code up or did you have to start trimming off all these crazy whizzbang AWS services that most people wouldn't know?

It was a philosophy that always appealed to me. We had one external dependency on a user authentication that we had to remove, but otherwise, you can go a long way with Postgres and Redis.

I know some of your larger customers or larger code bases, you must have some fairly stringent performance and underlying technology requirements.

We have customers with tons and tons of repositories. 500,000 repositories or instances with more than 1 million repositories. Another thing that we realized that a lot of people don't think of with self- hosted is each company is going to be running their own instance. You might have 10,000 customers, but you don't have to have a single instance scale up to 10,000 customer’s data. You have to make it work for the largest customer that you have.

If you've got a multi-tenant cloud service, you have to do so much scaling. There's so much complexity. There's also application complexity, deployment complexity. It has to be available 24/7 online. There's a lot of other security considerations that are involved in a multi-tenant cloud service that is not involved in a single-tenant service that customers run. Self-hosted is massively simpler and you can get away with an order or two magnitude less scale with self-hosted. We've got massive customers, but multi-tenants would be 10 to 100 times bigger than any of our customers. It's helped us move quickly and still offer a good performance to our customers.

That's a good technical way to explain it, shard your customers.

We're in the same boat. Although we don't have as many large customers as you. They have different ways of approaching performance, tooling, and stuff, but we're targeting the same thing. Are a lot of your larger on-premise installations running on top of Kubernetes and OpenShift?

We see a lot of running on Kubernetes on all the different cloud providers. We get to come up with the common denominator, and Docker Compose is fairly popular. We've got some customers that have their own container systems. One awesome thing about being in this business is we see how all kinds of companies deploy. The reason why a company might have their own container system is because they had a need for it before Kubernetes was out. They always bring an interesting angle and they're not in Kubernetes model culture.

There are a lot of things to learn and I'm glad that there's not a single model culture. We see it all and we try to make it so if you've got some custom container deployment scheduling, whatever system, that you can look at our Kubernetes or Docker Compose file and break apart the services and translate that to your own deployment. We don't have many Docker containers that need to be run. There's not that much inter container communication that needs to occur so that you can replicate it in your own system. Keep it simple.

Code search is something you won’t be able to live without once you have it.

In terms of deciding what to make proprietary and what to make public open, do those decisions tend to come naturally or are they big arguments for weeks on end?

It’s important to lay out a principle. For us, the principle is if a feature is way more important to bigger companies and is not as much of an individual developer feature, then that's a good candidate for it being in enterprise. We'll have plenty of discussions and you can see them in our issue tracker and our handbook, about particular issues and how do we classify them. Overall, people are understanding that this is a difficult decision for companies, and you're always going to see some people hating a company for not making everything open source and free. The silent majority understands that there's a difficult choice and you need a business around a product to be able to invest in making it better.

Did you put a lot of thought into that upfront or did that come out naturally? We've had exactly the same thing where we were trying to come up with a set of moral guidelines around some rules around this rather than what felt right.

We did have a draft of this principle and we put it in our CONTRIBUTING.md file. It's a lot less contentious than we might've expected it to be, in the same way that going open source was much less of a splash than some people might expect it to be. Even for a company like GitLab where they had a lot of community interaction on this thing, the vast majority of the GitLab community understands and trusts GitLab on these kinds of decisions. Have you seen any kind of commentary or backlash and decisions you've made here?

We're still starting out in fairly small. We are starting to build more proprietary code. We've taken a different path and it's a separate branch effectively, but it's in a private repository. We spent quite a lot of time trying to decide whether it was better to keep it in a public repository or not. We were a little bit worried about if someone forks our public repository, they've then fork some of our copyrighted stuff and what's their responsibility for that.

One of the very first forks we had was the Financial Times, and it’s like, “Let's find whoever it was that did that in the Financial Times.” We talk to them and they were using it internally. I'm still not sure what the right choice is. It's maybe different for different people. We felt like, if someone's forking a repository and they don't own everything in that fork, then it felt like it added complexity, but then at the same time, we don't get any oversight on top of the code that's private. 

For people in your situation, you have the problem of like, “How do I trust all that stuff?” We're not asking for access to technology companies, ninety-eight percent of their IP upfront is the first thing you do when you installed it. For you, I can totally understand. That's a huge win for being open. If some of your code was then still private, it wouldn't make sense. There are many different ways of solving that problem.

The overhead of when you make a change to both public and private code at the same time to make it to two repos, isn't that annoying?

It's generally. All of the work happens on the open one and then we merge that into the private one. The features that are private tends to be fairly isolated. For example, we'd run on top of Oracle, when you're doing those mergers, it's not like your Git clients suddenly start. The red warning light starts going off and you've got 58 collisions or anything like that. That might come into a problem down the line, but it hasn't yet. 

Do you guys do any telemetry tracking of the people that are running the open source products?

If you're running the open source version, no. If you're running the enterprise version, it's going to ping our servers for an update check, and you can check the handbook. It will send back also some information that it documents such as the quantity of users using it in a given day and so on. These scalar values help us understand high-level usage, but never sending anything like a file name or any code or anything like that, not even all the users and their email addresses. We wanted to pick something that was understandable and unobjectionable to 99.9% of the user base and something we can document.

You've not been curious to get any telemetry on the open self-hosted side of things. You don't know how many people are running it, right?

We assume it's a fairly low number because the Docker run using the images or getting the enterprise build. You can use it for free for under ten users. It's like how many people use Chrome versus how many people build Chromium from source? The vast majority of people use Chrome. I do use Chromium, I don't hold it to myself.

In terms of how do you dissuade or limit people from using those enterprise features if they haven't paid for them or do you not?

You need a license key. You could make your own enterprise build that comments out the lines that would be violating the enterprise license. Because we're an end-user product, companies are not going to be foolish to circumvent illegally a products licensing, and then have all of their developers use it. That would be a risky thing and we haven't seen that happen.

Do you think that's happened? Someone must've done it. It's common in one line of code out, or maybe they haven't had any?

I have no idea. I don't care. We'd find out if it was being done at big companies because let's say like ABC Corp was doing it. We'd get random bug reports or tweets from people at ABC Corp saying like, “I love Sourcegraph, but is this available.” I didn't realize ABC Corp is using it. We don't get many of those. There's one very large company where we suspect something that might be going on. It's weird. It's like talking to random reporters and someone’s like, “My dad works at this company. He uses Sourcegraph and he loves it.” All these weird data points, we're not hearing anything from the developers, but we're hearing things second hand. It’s more like a curiosity. For our business model, if 99% of companies pay and 1% don't, then that's a cost of doing business. Not that it's right, not that it's ever right to do that, in my opinion, but that's not a fatal flaw in the business model.

Every company has a breakage. I've not heard of a situation like that, but you’re never quite sure what goes on under the covers. What's next for your project? Have you got major landmark plans or incremental gains? 

It would be sad if I said, “The incremental gains.”

Those are important. We’ve still got a ton of work to do, to get code search out to more devs, more companies. Our business is like we've got a ton of awesome customers where all their devs are using Sourcegraph for code search. What that means is Sourcegraph is the only place in those companies where all their code is accessible and all the devs use. From there, we get all kinds of awesome requests such as, “I want to be able to fix code, not to spine code. I want to be able to programmatically roll out this fix and make a thousand PRs and monitor the progress of that.” We're working on that. We get requests like, “When I am looking at code, I want to see not just find references, go to definition, who wrote this, but also what does Datadog say or for this block of code, what feature flags are enabled or disabled and what proportion of users are getting that?”

That's an interesting integration with feature flagging tools like yours, bringing all kinds of other requests like that. Also, people integrating internal tools around code ownership and code compliance like, “If this file changes, who do I need to get approval from?” There's that. We're coming out with public code search. We have Sourcegraph.com. You can search across all kinds of code, but there are a lot of product changes that we're making to make that an even better experience. We want to make so if you want to to find awesome, automatically curated use examples for any open source project, and we're going to show you, “Here's the slow dash function.”

I feel like half of the GitHub searches must be around that use case and they're always like the results. It's like AltaVista for code.

Podcast

The Craft

of Open Source