Sentry

Check out our open-source Feature Flagging system - Flagsmith on Github! I'd appreciate your feedback ❤️

Episode Overview

One of the first interviews that we wanted to have when launching the podcast was with David Cramer of Sentry. We've used their product over the years and his story of taking Sentry from an open source side-project and turning it into a successful business has been an inspiration for us at Flagsmith.

We got to talk about the origins of Sentry, David's thoughts on open source software projects, and dove deep into some of the key product and business decisions they've made over the years.

One thing that struck me was how successful Sentry was before they even raised money. At the time, investors were a bit skeptical of open source business models. To put them at ease, they had already collected 2,000 paying customers, were profitable, and had 3 people working full-time on the project...not bad for something that he had started 3 years earlier as a side project.

My favorite quote from the episode came when David was talking about his confidence in his team's ability to ship a product that people would pay for early on:

"I remember this very vividly because it's still one of my favorite things I've said to somebody. When we were raising our seed the first round and people are like, “Why would they pay for it?” I'm like, “It doesn't matter. If they don't pay for Sentry, they're not going to pay for anything else because we will make it so good that nobody will be able to compete with it. We'll find a way to monetize it one way or the other. Again, it won't matter.”"

I'm sure you will enjoy this episode, I loved connecting with David Cramer.

Enjoy!

-Ben

Episode Transcript

Thanks again for coming on. When we came up with this idea, you guys were one of the first people in my head to talk to. For the few people who don't know about Sentry, do you want to give us an overview?

The two words I would use to describe it with would be “Crash Reporting”. TL;DR - we instrument your applications where the code level collects a lot of telemetry, mostly focused on errors and crashes. We also do some performance stuff now. We do that universally, but our bread-and-butter is front-end applications. Browser, JavaScript, mobile, desktop, we also do it for servers or APIs. We don't do infrastructure. It’s very developer-focused.

{{divider}}

Tell us a little bit about the origins. Was it an idea that we need to build this open source from day one or was it closed for a while? How did that work?

Almost all of my professional career, I’m a big open source contributor. I actually don't care that much about open source. I'm not a big advocate and I won't yell at you on the internet or anything, but everything I build is open source. There will be little snippets of code. Sentry was a 70 lines snippet of code a long time ago, back in the IRC era where somebody is like, “How would I get errors undo a dashboard?” This was specifically with the Django web framework. I'm like, “That seems interesting. I'll do it. I don't know why you'd want that,” and I did.

I think it was interesting, so I kept working at it, poking at it, and seeing what else I could make it do that’s interesting, and made it more fun to debug your software. It was heavily inspired by Django. The capabilities early on. It's got to be many years ago. That was the story of it for half of its life. It existed because I would say. I joined a company and they were using it. At that point, it was called Sentry. It doesn't matter, but I was using it at a company and it did not work well. I will say that I never thought about performance or scale but I did a little bit, but TLDR, the first week on the job, I took down the application. The application was big and the downtime was worse because of Sentry.

It cascaded in, Sentry made it harder to recover it. Within that first month, we rebuilt Sentry. I mostly did it but a few folks on the team helped out and contributed here and there. That's also where I brought on my cofounder for the business side of things, Chris, who's a designer because products need to look good as well as work well. We spent eight years since then building what is the current generation of Sentry, which arguably looks the same as it did many years ago. It’s a lot more powerful and there is support for different programming languages, things like that.

{{divider}}

When you took it to this company, was it like a little side project at that point?

This company I joined was called Disqus. If you've been around the internet for many years, there used to be Comet widgets everywhere. There still are but if you'd go to CNN, there's a comment widget. Disqus will power that, so it was this prolific thing. I joined and I was there for about three years, mostly working on infrastructure type things and using Sentry while I was there. Because we were using it, I found it more interesting to build it and to add features to it. You’re like, “I'm getting value out of this. I'll go poke at it in the evenings,” because that’s very much my hobby. It is lesser these days, but it was then.

It was nice. I wouldn't think of it as improving my skills, but I can develop this thing, we're all going to get value out of it, it's fun, and there's a community that's open source. Because it was free open source, it was BSD licensed at that time. I did that for about three years and at the end of that, we decided to spin up a business out of it. That's when I brought on my cofounder Chris who worked with me at Disqus. We had been convinced to build this as a Heroku Add-on. It turns out to build a Heroku Add-on, you have to build a SaaS offering. So, we built a SaaS offering. We did that in two weeks, which is a credit to services like Stripe. It made it very easy to get the nuts and bolts we needed.

{{divider}}

I'm trying to think back because years ago, things were harder than they are now. I started doing web development many years ago and whenever I look at modern Python code now, it blows my mind how much you can do in a 75 or 70-line script.

It was harder but it was easier in a way too. My biggest pet peeve with the industry is, it's not literally this but this is the best example, but npm modules. This infinite dependency tree of code obstruction that you can't reason about anything, and it takes you three days to start a project. I'm self-taught, contrast that to when I got my start, you upload a PHP script to the server. It might even be editing it on the server. You see the results right then and there. Some of the changes in the industry are necessary to deal with the scale of everything. We have made some things easier to build, but I often feel it's one step forward, two steps backwards with technology, and we're sort of praying that we leap progress ourselves.

Early on, it was easy because of Django. It was the first boon for me. It opened my eyes that there's something here that's interesting. I had tried working with Rails briefly and I didn't grok it. I would say it wasn't too bad, but it obviously changed a lot since then as well, which I would say is the need for Sentry. If anything, Sentry has become more popular because software has become more complex and harder to understand. Sentry makes that a little bit easier for folks.

{{divider}}

I wonder if you invented FTP and PHP hosts now, they're probably regarded as a sorcery. You'd be on the top of hacker news. They're horrifically insecure. I know where you're coming from, but I can see how all this additional complexity and there being fifteen things we'll need to talk to each other will help you guys immeasurably. The thing that I immediately thought of when you told me about that 70 lines script is it sounds similar to how Segment got started. Did people make that comparison to before?

People have made every open-source comparison. It's not dissimilar. I've met the folks at Segment. I'm not deeply familiar with the origin story. I think almost anything that comes from open source is born that way. What we've done with the direction of open source is an edge case or uncommon at the very least. Segment is a closed source product. They happen to have this open-source SDK, but every company in the world has open-source source SDKs...at least, any company you would use these days. Their SDK helped them a lot and it inspired them to do things, but that's where the comparison stops for me.

{{divider}}

It was like this analytics old wrapper. It sounds very similar. Disqus must have had some crazy traffic volumes as well. It must have helped in terms of a baptism of fire getting your infrastructure to a scalable position.

It definitely helps. The Internet was a lot smaller. I could say some numbers these days and people would laugh at how small scale that is, but nobody remembers that. It was very different back in the day to make those scales work. If nothing else, CPUs were much slower. You couldn't buy boxes with a terabyte of RAM on them. Before Disqus, I

worked for a gaming portal called Curse. That was my first time being thrown into the fire. It was a big site for World of Warcraft and we'd data-mine World of Warcraft. We did all these things that build content. What would happen is every patch day for World of Warcraft, which is a huge game, everybody would flood the site. Some of that was because we built content that they wanted, but you would go from 0 to 1 million instantly in terms of volume of data.

I learned a lot of tactics and shortcuts to build things back then, and then Disqus was similar. When I was building Curse, we were twenty-year-old kids and Disqus was my first real job. It was interesting because it was a lot of data. When something went wrong, it was pretty impactful. At the same time, Disqus primarily became a CDN cache where you're reading comments most of the time. You’re not posting since we dropped them.

The interesting thing for me at Disqus, given it was my first real company exposure, it was the first time I had to deal with what I would describe as the pain in the ass ops mentality, where it's like, “We don't want to give you access to a server.” It’s like, “I need access to logs to be able to diagnose what's going on, but you refuse to give me access because you're not collaborative.” There's not a real good reason behind it. We've probably all experienced this. These days, it's not so much that people don't want to give you access. It's way too complex and insecure to give access to all these wide- scale systems. That helped me build Sentry more than any other problem because I needed to make sure Sentry had enough information that if there was a problem, I didn't have to refer to other systems. That was critical.

{{divider}}

Years ago, that would have meant getting an SSH onto the box.

It was direct SSH onto a box. It's obviously insecure but the attack and threats, not everybody knew how to exploit these things years ago. They do now. We also have new defenses now.

{{divider}}

When you set the company up, how much velocity did the open source projects have at that point? Was it steamrolling along and gaining momentum naturally?

I will say that we set it up not just because of the circle name, but because people are like, “It would be great if you hosted this. We'd pay you.” I’m like, “Maybe it'll make a little bit of money.” There are no ambitions but we had a lot of adoption, but it was still Django focused mostly at the time. Django is a rising star in web frameworks. In Disqus, we were using Django. Instagram was Django. Eventbrite, and Mozilla were all using Sentry already. That was pretty cool. They were all running the open-source thing. I remember, when we set it up, the first day we had a paying customer, and then it grew organically from there. We had a lot of help along the way.

For example, there was a company, they got acquired at some point in the past. They used to run our servers like managed servers. We’ve got something like $10,000 from them as a credit. We didn't have a server bill for the first year, which was awesome. It allowed us to grow. I never had to put money into the company other than some registration fees or something. We got some of these sponsorships, some help along the way, and the open source community helping. We've always been very ethical about how we do it. All of our decisions, while they might look business decisions, they're very much grounded in emotions and ethics. I'm always like, “We're going to keep it free for people to run because it's only fair they're contributing, and we're going to sell this service where we run it.”

Even when we've changed licenses, which we can talk about, that mentality stayed there. The majority of open source for better or worse at Sentry looks like it does at any closed source company where most of our contributions have come in the form of a new programming language like an SDK, which is no different than if you had an API or a third-party integration like Jira or something like that. Which is great. We don't want to build all this, but if you look at the blame and the stats of the core repository, it's rounded off 100% employee contributions.

“I can develop this thing, we're all going to get value out of it, it's fun, and there's a community that's open source. ..I did that for about three years and at the end of that, we decided to spin up a business out of it.”

{{divider}}

They round the edges rather than right in the guts of all.

It's usually a minor bug fix. Every so often, somebody would build a small feature, which was awesome. But, the challenge with any feature is somebody has to maintain it. Is it stable? This person's probably not going to come back and fix any issues with it. It’s no different than the SDKs or the integrations, they'll be there. That version of open source, that community contribution aspect became smaller and smaller over the years because Sentry is a massive project at this point. Sentry is insane. You have to learn so much and have so much domain knowledge.

{{divider}}

When you gave up your day job and were working on this full-time, did you have a plan in terms of raising money? Did you have ideas in your head about, “We could do this close source feature and sell enterprise licenses?” Was there perfect clarity in terms of what you saw the future being?

We had the advantage on that. We spent three years working on this as a side project. We’re paid and we were running it as a business, but it was a side project like nights and weekends thing. The trigger for me, especially to go full-time on it was first that I could match my salary at Dropbox, which is where I was working at that time. Which meant I had zero risks and downside other than I gave up seven figures-plus a few points of equity, which is fine. I don't care at this point. The other side was Dropbox was a full-time job. It was growing very quickly, which meant a lot of stress, lots of new people, politics and stuff in bigger company. Sentry was also becoming very demanding.

Instead of me building cool features on Sentry in my free time, it was answering customer tickets. It was trying to patch things to keep it online. It was no longer fun. It was pure stress. Before we went full-time, we weren't paying ourselves anything. I would sponsor conferences, travel and speak at them, and we'd cover costs for things like that, but it wasn't like there was a huge financial reward or anything. It made it easy to go full-time because of the salary match. Once we did that and we convinced a friend of mine, Arman, to join as well, it was like, “Now, what do we do?” We spent the first three months like, “Do we raise money? It seems like that's the thing to do. We don't know why, but let's go raise money.”

Going through that, I formed some opinions. This is my personal story, not the company's story at that time. When we went through that first fund raise because that was primarily my job, that's when I decided we were going to grow this into the biggest thing that it can become. We didn't know what that was. Our ambitions grew over the years, but you definitely don't raise money without ambitions. Anybody or any good investors are not going to give you the money, but you shouldn't because it's a waste of time. We said, “We're going to raise this money and we're going to grow it into something significant, but we're going to do it our way.”

We didn't talk about, “Is going to be closed source or anything.” We did but not in the sense of we said we're going to do it. Investors would be like, “Are people going to pay for it?” That was the first conversation in our seed series, which was frustrating because we had 2,000 paying customers at that point, which in our industry is farther than almost anybody ever gets for tooling and services. Two thousand customers, profitable, three full-time great employees, and they're asking like, “Is anybody going to pay for it?” We raised money from Accel, which is a great firm. That worked out well. The question that came up in Series A was like, “Why won't Amazon run this? If it's a real business, why won't they just host it?”

I'm like, “It doesn't matter. If they do, we'll deal with the problem.” I hate this idea of distractions. I have some form of ADD but at the same time, I try to hyper-focus on what my goal is and ignore everything else. It doesn't mean I don't do multiple things at once, but unless Amazon is showing they're going to host it, we don't think they're going to host it. Why would they? It doesn't make sense from a technical lens if you think about these things. Post that conversation, nobody's ever had any concerns about it being open source. We're no longer technically “open source.” We have a different license but it has nothing to do with Amazon, people paying for it or anything like that. It's an unrelated decision. People developed conviction that our open source model was totally fine and better than fine over those years, which is good. One of our investors is very much like a hacker news or something when we changed his license. He's like, “I told them not to do it. I wanted to keep it free open source.” That's meaningful because that was not an idea when we started. When we first fundraised, everybody was like, “Woah. This is scary. It's not open core.”

{{divider}}

If you were doing that raise now, do you think those conversations would be different?

I think they'd be different because if nothing else, they could point to Sentry. That's a big deal because we didn't compromise at all. We still have none of the threats. Even if we didn't change our license, we would still have none of those threats that we're worried about. To be fair that people pay for it, it was a mix of open source and it's a developer tool. People pay for developer tools. That's clearly changed. In the open source thing, there's enough evidence in the industry that people want open source especially when it comes to infrastructure and tooling. The models behind open source are going to continue to adapt. It's not a risk. People would point to companies like Elastic as an example of the risk, but Elastic is hugely successful even if Amazon also monetizes it. Nobody should pity them. Everybody should be happy if they managed to get into that situation.

I don't think it's as much of a concern these days, but you still do have to find forward-thinking people. Like Silicon Valley, we've got them everywhere, but if you go talk to a traditional banker-investor from the East Coast...You get the point. They might not have great opinions. It's like the one stubborn head of infrastructure that still refuses to use cloud services. There is still some of that in the world.

{{divider}}

People mentioned Elastic but there's the example of one pretty much. You're right, it's not that they got extinguished out of existence by any stretch. I don't quite know why so much focus is put on that. Maybe because it's Amazon and they're this huge gorilla. What was the impetus for changing the license in that case, if you weren't worried about AWS?

I mentioned I'm this big focus person. Nobody wants to be frustrated all the time. You want to be able to spend your energy in a way that's productive and useful. One of the ways I became frustrated overtime was we would see almost always startups. They come in and copy-paste pieces of Sentry and things like that. I'd be like, “It's legal because the license says,” even our documentation. They would be like, “We copied pasted this entire page, but it's open source. It's legal.” We would have people take our SDKs, which are free open-source still now. They would fork the thing and not attribute us in the license.

I have to go and be like, “Do you realize this is illegal? I can take down your entire project,” and then they would go fix it. I'm tired of having conversations about that, especially with our server code. At some point, there was a larger than us company that was trying to distribute Sentry. I'm like, “Why would we help you do these things? You don't contribute back. You're not helping us build a business which funds everything. Funds not only the development of Sentry but it keeps everybody paying for everything that they have concerns in life, that works for us that developed Sentry.” Those things were annoying. They weren't a threat to the business, but they were annoying in a sense from an ethical point. In our moral high ground, we were like, “We don't like this and we don't want to spend energy on this.”

We changed the BSL license, and it effectively squashed all those concerns. On day one of us converting to this license, somebody is like, “I'm going to fork Sentry and it's going to still be free. I'm going to build a new Sentry.” It's like, “No, you're not but good luck to you.” That was a good move because at least we don't have to think about it anymore. If somebody does this, we're going to straight-up and go submit a cease and desist. These are not individuals that are nice people making mistakes. We wouldn't go after them, but it's a company that has $50 million from venture capitalists that are not playing fairly.

{{divider}}

They weren't taking the whole thing and putting a new badge on it. They would take components of it and then encapsulating their stuff around that.

It's always hard to say what has been used and what hasn't because we can only see what's publicized in open source. Most people do not have an open source product. I cannot tell you who has and who hasn't used our server code, signed if they do or whatever. Open source is for learning as well as everything else. When it was things like using our SDKs, which is fine but they're not contributing back, they're just forking it or using our documentation, which is straight up, “What the hell? It’s annoying. Why would you do that thing?” It was this frustrating thing that we'd see it. It's like, “Why are you doing this?” To where I'd have to have conversations with their lawyers. I'm like, “I'm not a lawyer but I'm going to explain what's going to happen if you don't address the situation.”

{{divider}}

Did that change overnight? Was it all of a sudden all those issues went away?

I would say that we've not had any of those issues since then. I'm sure they'll come back in some shape or form. I'm sure people are still going to continue to fork our SDKs, which is fine. It's their prerogative. The thing that we were not okay with it, which is taking our server code, which is how we operate the business and using it to build an entirely separate business or something like that when you've never contributed, that is gone. At the very least, if it comes back, we have legal protections.

{{divider}}

What’s your stand on telemetry beacons if I fire up a Docker container with your project's code in it? Does Sentry do that? Do you have data or self-hosted?

Anybody building something like Sentry, whether it's an infrastructure service or product, I would highly encourage it. We've had it for quite a long time now. It's anonymized. We ask folks for contact email address, it's optional. We say it's for security updates. We've used it once in life. Otherwise, it's never been used and then we get some anonymous stats. The anonymous stats helped a lot because we get things like the version of Sentry being used, which is useful. We get things like the volume of data they have in the systems, which helps us understand the landscape of like, “Are there a lot of people using it in self-hosted? Are those bigger customers potentially?”

{{divider}}

When you introduced that, did that cause any issues within the community?

One person complained maybe. We're like, “You can not use Sentry.” That's what people forget now. There are a lot of vocal people these days. People forget less about that problem, but it's your project, you can do whatever you want and it's their choice to use it. It's no different than what products you pay for. Honestly, you can turn it off for sure. It's on by default, which was the contentious thing. People are like, “I don't want this reporting.” I'm like, “Don't use Sentry. Or go turn it off. This is helpful for us to build the product better.” I'm not anti-privacy. I assume everything is on the internet by default, but we should act in good faith.

Things like GDPR, parts of it at least, because of that, but also at the same time, I recognize that people need some of these data points to build things that make you happier. Facebook, I’m sure is a contentious topic for everyone, but the idea that Facebook is monitoring you and doing all this stuff can show you more relevant information to you. It's hard to not want that thing. That's how we thought about the data. This helps us understand like, “Do we need to think about these big customer cases? Do we need to patch these old versions?” Things like that. It was super valuable.

{{divider}}

Do you think it directly informed your product design and path?

It helped us understand even the rough scale of people self-hosting as well because there are things like, “Are people self-hosting?” Those are small instances because that's one common case. They're saying like some of them are big banks because of compliance and security and things like that. That was helpful because it helped us understand that like, “Sentry was applicable to all these customers.” That particularly didn't necessarily change the product we built. The version information was extremely helpful to be able to understand what versions were still heavily in a while, were people upgrading quickly, or how long we're going to have to support things. We've found over the years that there were other things that were valuable. At some point, we added this notion of like, “Was it running with docker or not so we could understand, can we ship docker support and ignore run docker?” The things in there is very much based on what the product or project is, but the very least the versions were immensely helpful.

{{divider}}

In terms of how much thought and design you put into the self-hosted story, this is one of the things that we've been slowly realizing that every time the product gets more complex and the infrastructure required that run it gets more complex. As soon as you include a time series database, then that's adding another ratchet up of complexity if you're self-hosting it, but more outside of docker name. How much did you have that in your mind when you were designing the platform?

I would say that's changed a lot over time. Early on, Sentry’s goal was to adapt to infrastructure. We use Django. We’ve got support for different databases out of the box, SQLite, Postgres, MySQL. MySQL is going all this other stuff at some point too. We had that model early on. We need characteristics of services. We needed a SQL database. Djangos got that for us. We need a key-value store. We use this technology called React or you could use SQL for it, or you could use something like S3. We use big table now. We had a key-value store. It had a set of interfaces.

Our whole goal, and this is still the goal of the company, we want everybody to be able to use Sentry. That's why the open source thing is also critical for us. Adapting infrastructure is one way to do that. Now, we changed our mind on that because what that turned into is this mess of, we have to support all these weird edge cases, and nobody is informed about how to do things. We're like, “That's all dead. We only support Postgres.” Key value is still a little bit flexible. It's hard to mess up key values. We're like, “You can write an adapter. We only support Redis for this thing. We use a technology called Cortana. We only support ClickHouse for this thing.” We're over this era of, “Can we adapt it because it looks similar?” and it made our lives so much better.

It's not like people were coming in and fixing all the issues with say the Cassandra adapter or something like that. We'd have to support it. We didn't even use Cassandra. When we did that, it’s a lot less maintenance burden which made it easier to support that open source thing. For clarity, Sentry's open source is the same as our closed source. We have closed source infrastructure, our server configuration, our business analytics, and all these other things, but the product is what's on GitHub that's available for everyone. That's what we shipped.

We made that decision. As you said, as the product got more complex, we had to add more services. Sentry used to be able to run with a SQL database and Redis, and then our application services. It's straightforward. It's a traditional web stack. Anybody could deploy that. It'll have performance characteristics, but it was easy to deploy. Now, the docker composes thirteen different services and I'm sure we run twenty in production because the docker composes, “What's the lightest weight version we can get.”

We introduced the Kafka, ClickHouse, other layers of Redis and all these other things. It's almost an inevitability. On one hand, we talked about this internally, it sucks that there's not a simple version of Sentry anymore, but there's not a lot of value in having a simple version of Sentry so somebody can run it themselves and be an idealist. Who cares? We're trying to build something that solves an actual problem, not build something that fits some ideals.

That has certainly been tricky for folks, which is why we decided that docker was our own solution. We standardized on docker compose. Most people can spin it up successfully, but I will say the challenge with it just like anything that has these things is nobody knows how to run any of these services. We invest in making self-hosting stable. We have an open source team. It's two folks and they're responsible for the open source release and the quality of it. There's still an immense number of problems with it because nobody knows how to use ClickHouse. It’s an extreme technology but it's certainly an uncommon and newer technology.

{{divider}}

Most of the people I've talked to for this series have mentioned it, which is weird because I heard of it before. It seems to be winning in the space that it's in. You're the fifth person who's talked about it.

When we picked it up, I only knew of two other companies that were even using it outside the Yandex themselves. It's definitely one of the things that has its flaws, but it's well-designed that people are like, “I don't have to roll my own solution to making MySQL scale at some crazy thing or dealing with the caveats of Cassandra.” We explored a bunch of them, and ClickHouse was the only one that fit the requirements. It’s a thing that people don't know how to run at the very least, and running it is operationally complex. It's something that requires real operation expertise. It's not as simple as a Postgres database. I would argue Postgres also requires an immense amount of operational expertise as well.

We ended up in this fickle where we have a lot of these things now that have to exist, and you have to deal with it. We've tried to make it so they run, they're stable, and people can still use it. We've also said there are two cases of self-hosting Sentry. One, you're a small use case and you want to stick it to the man. We're like, “More power to you. It's cheaper to pay us or to use our free plan, but who cares? You can do your thing.” That's a cool way to POC Sentry as well. There's that benefit from the business side. There's also like, “I am an agency that has government contracts and I can't ship data to Sentry because you don't meet these checkboxes.” The self-hosted thing, we shipped the docker-compose that doesn't work for them, but we try to document things. We're not great at this. There are enough details in that docker-compose.

{{divider}}

Why didn’t it work for them?

It doesn't scale out. They still run ClickHouse in production. You need to federate it. You might be able to get away with one reticent since but for high availability, you almost need infrastructure of our compose. They have to figure it out from there. Now, if they are one of those companies, they've probably got an operations team that has expertise in a lot of this stuff. They will figure it out from there, and that's totally fine for us. Our hope is that over time, we'll either meet the checkboxes or they'll go, “We'd rather pay Sentry to solve this problem for us,” which I will say this is the fire and forget model, but it has worked well for us.

We've had a lot of folks say like, “We've been running Sentry for a couple of years. I don't want to take on the burden anymore. We'd love to use SaaS.” The great thing for them is they're a four-year-old version or some ancient thing that was marked stable at the time. It's night and day product difference, which is also why we focus so heavily on SaaS because nobody wants to upgrade software every month, which is what our release cycle is now. You get to the latest and greatest every single day.

{{divider}}

Do people ask you for the full being, docker-compose with high availability everywhere and everything scaling out forever? Is that closed?

Our infrastructure is a mix of things and that's all closed source. It's very much our infrastructure. I would argue that infrastructure is almost always specialized in the sense of it's almost always catering to your exact scenario. It's your customer profile. It’s the machines you have available to cloud provider you use. We don't open source any of that. As an example, we also don't open source say a Kubernetes set of configurations. Other people do and that's fine, but we don't support it for a couple of reasons. One, we don't use it. We're not going to support anything we don't use. Two, that's where we draw the line on what is our business and what is not.

We don't monetize Sentry by you running it yourself. We don't fund development of you running it yourself, other than we want to make sure we're sticking true to our roots, our origin where you can run it. We're not going to give you our secret sauce at the end of the day. That's always a delicate line. We've talked internally about we needed to find that more clearly for customers. We do need to make sure we're at least being ethically good where we don't have huge gaps in architectural documentation or things like that.

“Twitter and Facebook and post IPO for Twitter, how they're going to make money. These big-cash burn funnels for supporting these consumer networks. They all make tons of money. It doesn't matter. If you have the network, you can find a way to monetize it. We always viewed that in the same with the open source thing. If you have the following, you'll find a way to build something sustainable around it.”

{{divider}}

Do you have that written down anywhere or is it a vibe within the organization that everyone gets? It's a huge gray area.

We have a lot of it written down. It's still a little bit fuzzy. It's one of those things that you need to continue to refine and clarify. We're 140 people now. After you get past 50, this word-of-mouth knowledge transfer drops significantly. We're talking about this because we brought on a new person in open source team and it's like, “Why don't we do this? Why don't we do that?” I'm like, “Let me regurgitate all these reasons and stuff. We should very much clarify this, but let's also clarify it for our customers. Write all this down, ship it to our production documentation, call it a day. We don't have to have any of these conversations anymore. Everybody knows where we stand.”

We've done a lot of effort to help folks to internally understand how to do open source, especially because we use VSL on the server. It's a little bit tricky. We have this whole vent or this decision tree of like, “Is the component for the server VSL? Is the component in SDK has to be BSD or something? Is the component anything else? Use Apache unless X, Y, Z condition, then use MIT because one day, who knows?” This kind of scenario. That's made it a little tricky. It's at least somewhat easy for folks to reason out because we don't have a policy of you have to ask to contribute to open source or anything. We're open source by default, as long as you follow these rules.

{{divider}}

To people who start making noises, it's getting too hard to self-host, you're throwing all these new things, we don't care about all. Was that a point of friction?

That wasn't a point of friction and we're like, “That's life.” We try to not let it get to us because it's the reality. That’s what has to happen to consumer scale. People will be like, “I don't need these features that you're adding.” I'm like, “That might be true, but you're also a two-person shop. We're not building Sentry for you at the end of the day.” You can certainly use Sentry. You can go use the free account and we’d be more than happy and nobody will care. Here's the analogy. In Sentry, we support every programming language.

We have an immense amount of infrastructure and service work that is, “How do I transform a C++ compiled crashed dump into something that looks a Python stack trace.” That's roughly the analogy, which is a lot of work. We built the technology around that. You can imagine that for a variety of different runtimes. iOS has its own thing, Android and Java have their own thing. There are other formats of these exact same things that we also support. There's a lot of that stuff that goes into Sentry. If you're using Python, it's like, “I don't need any of that.” That's fine. Most of the stuff you don't have to run if you're not using it, so it’s okay, but that's unrelated to using Kafka or ClickHouse. They might think they don't need that, but what they do is they allow much higher uptime and availability. It’s obviously much more complex to run but they allow that. They also allow much more power.

When we adopted ClickHouse, we were able to remove a ton of operational overhead from Postgres. We were able to kill tables, kill indexes, and now we materialize all of it in ClickHouse. Not only does it allow the product to answer more interesting questions, which if you have any volume of data is helpful, 100 errors or whatever, who cares? You could use a log file and you'll be good enough. You have 10,000, which is even a hobbyist project we'll have. All of a sudden, being able to pivot on that data is super valuable. People don't like work is what I would say. When somebody comes in and complains like, “This old docker only required these three things,” or it wasn't even docker. They're going from pre- docker. Now, I need the East fifteen services and I have no idea what ClickHouse is. I have to figure out how to run ClickHouse, and also ClickHouse is erring out or something like that. I get it because I would be annoyed by it as well. It's one of those things. It's the reality of what's happening. You're getting this for free anyways so take it for the cost that it is.

{{divider}}

That must be having an effect on the number or the profile of the people that are self-hosts to get because you're not going to run like this stuff. We run the rust version Bitwarden, for example, because it's great. It's one container and works well. There must be millions of people doing that in the same way. Have those telemetry stats born out that fewer people?

It hasn't changed much. What's always been interesting about Sentry is open-source growth has declined. It stayed at a steady rate, which generally means the month-over-month growth declines. We've always been like, “It's fine.” It's not unhealthy but it's not growing fast. We're not trying to get it to grow fast. There's not a lot of reason for us to force that thing. What we've found over the years is people very much value open source for a variety of reasons, but it doesn't mean they want to host it themselves. A lot of those people that would do open source by default are like, “We're more than happy to pay for the service.” One, because it funds the open source which we believe in. Two, we don't have to deal with any of this complexity. We have better things to do.

What I will say is often the vocal people are people that, earlier in their career, have different scope of context. They don't understand that they have better things to do. If you go talk to a bank, they fully understand they have better things to do. They're like, “We'll happily pay for everything. What's the price?” You have to jump through other hoops at that point, but it's one of those things. It's almost a maturity thing. We've recognized that and we're like, “It sucks that you have to deal with all this.” For the most part, the customer profile has always been similar. We still get some big customer self-hosting. We get a lot of small customer self-hosting. A lot of people are spinning up the self-hosting to try it even if they're a big customer, but the majority by a long shot go to our SaaS service. Our SaaS is at least five times larger than the open source insulation base. It's not a typical open source thing. Our SaaS for years has been larger than open source. That by total organization count is how we think about it. We comply with installations.

{{divider}}

Things like databases or web server is a classic example. No one pays for a web server nowadays. People never did. They paid for Netscape way back in the day. I guess you get a few organizations that want to pay for Oracle, but open sources want in that segment completely. For services like Sentry, do you think the puck is going to land at that same point? Do you think close companies like New Radical or whatever? Are they still going to be there in twenty years' time or is everyone going to be running open source everything?

You'll still have both forever because I'm sure there's still a bunch of commercial web servers. I don't know what they are.

{{divider}}

I remember back in the day, there was a Java app server called Blue Martini or something. It was $50,000 per CPU or something.

I know Load Balancers, which are not that different because you can run an app-level server. There are still commercial enterprises selling big Load Balancers. It will continue to exist. The way I reason about this is two things. If you are running it in your infrastructure, you're going to prefer to be open source for a variety of reasons. As a bigger company, you don't want to be locked into perpetual fees for something that's core, is a good example. You might buy services because you want to set this up or upgrade or do all these things with Postgres, but you don't want to be locked in pain for Postgres forever and build your business on top of it.

This is why Postgres has slowly overtaken things like Oracle as it was a common use in some of these bigger orgs now. The other more relevant version is who is the primary consumer. Almost the buyer and not the person who's paying for it, but the person who's bringing it in. If that's a developer, by default, they're going to go with open source. That's the change we see. One, that's a truth. The other truth is that developers are now given decisions on software. The IT guys are not just handing out software anymore. Companies like New Relic will continue to exist. What I will tell you is they will struggle to compete with an equivalent open source solution. We're number one in the error space. We're still early in performance but we have more than 2X the number of customers as New Relic. They make a lot more money than us, but we'll get there. I don't think that means that these commercial closed source things go away in that space. I do think they have a serious disadvantage long-term if somebody like us exists. If nobody like us exists, they're going to continue to dominate.

One of the companies I help is Posthog. I'm like, “This would be great. If you believe in open source, do an open source because people will value that, and it will not harm you whatsoever. If anything, it will help you over time.” They talk about, especially before IPO and stuff for Twitter and Facebook and post IPO for Twitter, how they're going to make money. These big-cash burn funnels for supporting these consumer networks. They all make tons of money so it doesn't matter. If you have the network, you can find a way to monetize it. We always viewed that in the same with the open source thing. If you have the following, you'll find a way to build something sustainable around it.

I remember this very vividly because it's still one of my favorite things I've said to somebody. When we were raising our seed the first round and people are like, “Why would they pay for it?” I'm like, “It doesn't matter. If they don't pay for Sentry, they're not going to pay for anything else because we will make it so good that nobody will be able to compete with it. We'll find a way to monetize it one way or the other. Again, it won't matter.” That's roughly been true. We've had to compete with ourselves by making it free and self-hosted. We've completely demolished anybody else in the space from a market share, from all members. That's one because we built the better product straight up, and that was our goal.

We built it with all these things that developers wanted. For the people that care about open source, it was free and it was great. That was what resonated with me and how I saw things changing. It is very much the developers who hold the power in a lot of these decisions. A lot of people still don't recognize that. Going back to that cloud provider example, I was at this enterprise dinner one time, which these things are not me. I’ll go to a wedding, wear a suit, have a lot of fun, but chilling with a bunch of IT execs, not fun.

I sat next to this head of IT operations for bank that hopefully nobody's ever heard of. I don't even remember what it's called. I'm like, “You should buy Sentry. That's why I'm at this dinner so I can sell you stuff.” I'm not a salesperson either. He's like, “Is it a cloud service?” I'm like, “Of course, it's a cloud service. Why would it not be a cloud services? This is 2018.” He’s like, “We don't use cloud services at our bank. As long as I'm still employed here, we will never use a cloud provider.” I'm like, “You're not going to be employed in a few years,” that I guess is the answer. That persona is how I think about this change in IT and tooling in general for developers. Your competitive advantage as a company is catering to what your employees want because there are too many jobs and options out there.

{{divider}}

One of the other things that's been interesting is we've seen this where engineers can't get budget for anything. They don't have a credit card. They don't ever get given a credit card, but what they do have is some AWS keys that allow them to spin up a 50,000 pound a day box. No one is stopping them. We had a customer and one of the guys as joke he spun up there. I don’t know how many GPUs it had. He left it on by mistake over the weekend and stuff. That was tens of thousands of pounds. If he was to ask for a credit card for the $20 a month thing, you need to send in all your TPS reports and then five months later you might get approval. That plays to the self- hosting side because if it costs $1,000 a day to run, you're a hard hardcore stack on AWS, they can do that. It's weird. I don't know if that's going to change or not. I guess it will if the bills get big enough but it's a common use case.

Especially for larger organizations because it's that goal of, “How can I have less tools?” I want to standardize on one. That's not because they think that one is going to be better for them. It's because they don't want to have to deal with vendor management. They don't want to have to deal with contract negotiation for all these different vendors especially at a larger organization. If you've got every single person wants a new $20 vendor, that's a huge cost and liability for the company. I will say as a growing company, that's something we definitely recognize and see a challenge behind. You're always going to have that problem. Throughout software history, it's going to continue to exist. I have this mega vendor that can do all this stuff for me, and I've got a contract with.

That could be hardware or it could be New Relic or something like that. You'll happily spend money with them and give people access to spend that money. You've got something your developers or users want to use that. You're like, “Is this worth the effort?” That's a tricky thing because when somebody comes to me and they asked for new software at our company, I'm like, “Do you need this? Why do I have to waste my time having this conversation right now?” Most of the time the answer is, “No, you don't need this.” You've haphazardly. They're like, “Can I have this thing?” Our approach to that has been like, “Let's at least try to be great at the one thing we do, or prior to the new thing we do and let's do it for every application they have.”

At the very least, they know they only have to buy one vendor for that problem. They'd love for us to solve every problem at the same time, but we're never going to do that. It's the reality. The other side that's interesting, especially if you're an infrastructure company and in the open source, if you can monetize by selling your stuff on an AWS or something, you should because they already have arrangements with those cloud providers like companies do. It's unified under one bill. They don't have to go for budget approval, all these other things. It's valuable especially if you're selling to enterprises.

{{divider}}

Is that something that you guys are working on?

I would say not really. Our goal is to never build anything that looks like on-prem software, at least from a monetization point of view. We're still figuring that out because we're still not quite in the era where everybody will use cloud providers. Even though most companies do, they don't necessarily use it for everything. For example, you may have a big financial institution that uses AWS, but AWS is limited to these constraints and stuff. It doesn't touch their core services or something like that. We're trying to figure that out, but part of this is also you ride it out and it will change in time. You’ve got to decide how important is it to you now and how long might that change take?

{{divider}}

What is next for Sentry?

We launched our first, we're calling it the Prolog, V Zero something, our performance monitoring product, which we've wanted to do more than error monitoring for a long time because we thought we could do something better. As engineers, we always want to build something. We finally got something out the door, it's only for a couple of languages. It's like JavaScript, Python, and we’ve got PHP now. We've got a lot of work to do to not only make that good even for those languages but to build all the other support for other languages.

That's it for quite a long time. We're very focused on we have to be great at this error monitoring thing that we built and now we want to be great at this performance monitoring thing. What I think is important about the industry is all the problems are the same now as they did years ago. Software hasn't changed. The complexity has changed. We've always focused on the concern that has changed has been software is more complex and be able to iterate quickly is the only goal you have.

This is what is interesting. If you look at over the years, take Windows 95 or 98, multi-year release cycles because of QA and all this other stuff. There was no way they could do it faster because they didn't have the ability to update in real time. They didn't have the ability to automate QA and all these other things. If you fast forward now, the whole thing is like, “We balance this ability to iterate quickly on our business, which is software these days with how do we make it stable and fast?” How do we make it stable and fast is what Sentry is honed in on, and what most companies are focused on. CI is one tool that makes it so we can do better. That whole, “I can edit PHP scripts over FTP,” you could literally do that as long as they're automatically tested, and they automatically have vulnerability scanning and all that. All we're doing is building that glue so we can go back to PHP era thing.

{{divider}}

We've been using Vercel a lot and it uploads your scripts by FTP but it's done in the proper way where the change is logged and tested, and it's a blue green deployment. It feels like it's converging on that same experience.

It has to because the only way you can succeed is by being able to make change. The faster you can make change, the more successful you can be or the better you can compete with anybody else. We focus on that. When we try to reverse engineer our mission statement, it was very much like, “Sentry must be about enabling velocity. Who cares about meantime to resolution and all this other stuff? We're not about saying it's acceptable to make mistakes. We're about making it so it's easy to recover from those mistakes that are inevitably going to happen.”

{{divider}}

Thanks for your time. That's been fascinating. I look forward to what comes next. I'm going to get off this call and I'm going to fire up docker and see what happens with the compose. Thanks again. Take care.