CapRover Interview 1

Check out our open-source Feature Flagging system - Flagsmith on Github! I'd appreciate your feedback ❤️

Episode Transcript:

I’m happy to introduce Kasra Bigdeli, who is the Founder of CapRover. This is a personal interest of mine. I’ve been fiddling around in a hobbyist capacity with Heroku-like platforms that don’t have Heroku-like price tags attached to them. I’m interested to talk to him. Kasra, welcome. Thank you for your time. Do you want to introduce yourself and a little bit about your background? Also, you’ve got quite an interesting background in terms of where you grew up as well.

I’m Kasra. I am originally from Iran, moved to Canada many years ago. I actually worked in several different industries. My academic background is mechanical engineering and structural engineering. I transitioned into the software industry and since then, I’m mostly working in software.

{{divider}}‍

What made you interested in moving to software? I have a lot of friends who studied mechanical or aeronautical engineering. They’re all working in software now. Was that the same with you? Is it a bigger market?

It’s a multifaceted problem. Of course, one aspect of that is the fact that the market is nervous in software industry but there is a second part, which is very important. When I was looking back through my experience, my projects in the past, I realized that I didn’t study mechanical engineering or I didn’t study civil engineering. I studied applications of software engineering in mechanical engineering, applications of software engineering in civil engineering, particularly I worked on a lot of free optimizations, basically the techniques that are being used in machine learning. Instead of optimizing to recognize an image, I optimized where to place vibration dampers in buildings to absorb the most energy during an earthquake. Effectively, I was always studying computer science but the applications were in different fields.

{{divider}}‍

Did your career in software engineering in terms of what you were doing professionally lead up to CapRover or is that something that’s a little bit unusual in terms of what you’ve been doing professionally?

Yeah, it was mostly on the side. I started transitioning into software when I was working for a start-up that was focused on robotics. I picked up some software engineering tasks here and there, and slowly transitioned into mobile engineering, building apps. You know how start-ups work. You essentially do everything from firmware to the website, to the back-end service, to the mobile app. Most of my professional life was on building scalable mobile applications. Having said that, I had always had a side hustle. It’s just the way I operate it. I worked on a few start-ups on the side, a few side projects, and every time I wanted to deploy any of these side projects, I needed a backend and Heroku was the way to go.

The latest one that I put together, it just scrapes the vaccine appointment, so I can get a vaccine. We got CapRover actually have to host these things on Heroku. This is an expensive platform for something that you don’t want to spend that much money on or it doesn’t consume that much ramp. Their business model is actually pretty smart. They bring you in as a free user and as soon as you start consuming resources, it exponentially goes up and up. It was mostly my side projects that pushed me towards CapRover.

{{divider}}‍

Were there any other open-source projects that were inspirations to you? Was Docker around before? That’s older than the CapRover projects.

Docker was around and I did have a look at that back then. It was surely one of the projects that started many years ago. It was pretty much when Docker came out after that. The way Docker advertise itself, I think originally was Heroku in hundred lines of bash or something like that. That was their tagline. I didn’t want that. I found that if you want to go and read Docker’s documentation, I might as well go and read it. I’ll build it myself. I wanted a point-and-shoot application where I install it and everything is in front of me.

If you go to a documentation to find out about a feature or how to do something, I consider that a failure of a software. You shouldn’t need to go to documentation. When you look at the UI, there are information icons next to important features, which lead to documentation. You don’t have to dig through the documentation. What is this environment variable? How do I set this up? You can click on it and it redirects you to the appropriate page. Having descriptive error messages, these are the things that I wanted from a software for someone who don’t want to spend so much time to deal with these foundational concepts.

{{divider}}‍

To me, that’s obviously like an itch that you want to scratch. It feels like quite a daunting project to take on from a scope point of view. I know with Docker, you started at a hundred lines of bash. Definitely not that anymore. I used to use Docker quite a lot. We still do in some of the companies that I’m involved in. For my projects, it’s interesting what you say about that, because my experience with CapRover has been exactly that. Docker is great. I love it. It has provided a tremendous amount of value for companies I’ve worked in, but to have some sharp edges and the user experience, you need to do things in the right order. Otherwise, things don’t work quite right.

The thing that I was blown away by when I discovered and installed CapRover for the first time was the level of polish around the whole user experience, not the interface, but it was almost impossible to break. I found that impressive. I got this feeling that it must’ve been like a large amount of work to get to that point. Had you been drinking when you started the project because it seems like this huge task?

It has been a huge task. It has been a huge project. Let me go back to my side hustles. I was always thinking, “I’m going to make a side hustle and it can turn into a business.” I tried many things and what I found out is that although I might be good at building product, I might be good of an engineer, product designer, product manager, I’m not good at sales. I’m getting paid by big tech, now let’s just have this out for my passion and see if it was something that I wanted to gauge myself. Am I doing the right thing? Is my understanding correct about user behavior, about their expectations? This has helped me a lot in my career. When you understand user’s behavior, it helps you to build a product that is manageable.

You mentioned that this is a huge project. Yes, it is, but I had to say no to a lot of feature requests. I had to add explicitly in our scope that, “We’re not going to be mirroring every single flag in Docker CLI.” CapRover is meant for simple standard deploys. It just works right off the bat without any customized configuration. If you want something fancy and say, “I want these two apps to work on these two nodes and these three apps on these three nodes,” probably I’m not going to be porting that to CapRover UI. You can do that using Docker CLI and CapRover completely respects it. Whatever you change using Docker CLI, as long as it’s not controlled by CapRover, it’ll stay the same. That part was an important part that helped me in my professional career as well, because now, when I designed software for a large company, I understand how a larger scope could come back and make a huge damage to the project. By cutting the scope, you’re actually helping the project to survive.

{{divider}}‍

What was the version 0.1 release? Did you think like, “If I can do this very first most minimal case thing?” Is that how it started or was it just you by yourself at the very beginning?

Yeah, pretty much. Now, I’m the main contributor. There are people who are contributing here and there, but they’re almost a one-off contributor even for larger features. Somebody introduced dark theme, which is amazing, but most people introduce one feature, go and move on with their life, which is fine. What started this project? I was reading how Heroku works in general, and then I realized that there this technology called Docker. I started learning it. Once I felt like I have Docker concepts mastered, then I designed the interface on the paper. Within 2.5 months, I had a working version of CapRover, which was very simple back then. It didn’t have support for persistent data. It didn’t have any volume mapping, etc. Very simple. I released it, blew up on Hacker News, and I got a lot of attention. It was like, “This thing has some merit to it. Let’s spend some more time.” Since then, every few weekends, I spend a few hours polishing it.

{{divider}}‍

In that first Hacker News post, how many people were complaining that it wasn’t doing replicated Postgres and multi-node clusters? Did you get any of that or was it the hive mind kind in this case?

It did have clustering in the first version but it didn’t have the persistent data. Of course, a lot of people complained that, “This is only 100 lines of bash. I already have this in my workflow.” Have you seen the infamous Dropbox posts on Hacker News?

{{divider}}‍

Yes.

It was pretty much like that. Why would I need this? Of course, if you have a working workflow that works for you, you don’t need this. It evolves and for feature requests, I tend to say no unless I see that this is a repeating theme and a lot of people have this. I then consolidate them and find a generic solution rather than a one-off solution that works for one particular use case.

{{divider}}‍

One of the things that we found at Flagsmith is two things. One, it’s important to say no, but also two, it’s hard especially if people contribute code you’re then like, “Thanks but you’re going to have to put that in a fork.” Did you have experience of dealing with this prior to CapRover or in terms of it? Had you been involved in a successful open-source project before? Were you feeling your way as you went?

I wasn’t experienced in terms of open-source projects, but I led multiple development teams before. I’m used to saying no. Usually, younger engineers come in and they want to rewrite everything from scratch, that part was fine. What was not fine was when I found out that people go and do things and they expect the code to be merged, etc. One thing that I did that helped a lot is to document this. Having that as part of clear parts, “This is our scope. This is what we don’t want. This is not going to go in and have it as a pool request template.” When people even try to open a pool request and say, “This is not going to get merged.” I have a line in the pool request template saying that, “If you’re making it change that is changing 50 lines or more, it does not get merged unless you discuss it beforehand in our Slack channel.”

{{divider}}‍

How often does that happen? How often do people not read that line?

Since I added that, it did not happen. People see that and they’re like, “It has a very harsh language saying that it does not get merged,” but that’s fine. Otherwise, people move this software to many directions and you came up with this Frankenstein software that is not working in any way that you intended to. That’s essentially the goal.

{{divider}}‍

If you do get a consistent use case as requested, you can have your mind changed on that potentially.

There have been more than a few use cases where it pushed back and it came back to me. I was like, “This actually does make sense.” One of the ones that I can think of the top of my mind is we now default to Docker file if there is no captain-definition file, which is the CapRover version of defining how to deploy your app. You don’t need to have a captain-definition file. If you have a Docker file, it would fall back to that. I pushed back because I want it to be more explicit, but after 10, 12 requests here and there, people were confused and I was like, “This is a real use case. Let’s add it.”

Another thing to be added was pre-deploy scripts, which is a script that runs before you deploy a new version. I did not originally add that but people asked for a number of different customization steps. That’s the parts where waiting helps because you wait for some time. After some time, you see these different ideas and you can come up with one solution that addresses all of those problems rather than introducing 300 different drop-downs and controls here and there, “If you want to make this customization, this is your hook, go and build it.” That way you can evolve and you can move faster because the solution that you developed is not going to only address one problem, it’s going to address multiple different problems.

{{divider}}‍

When you get issues like that come in, do you close them and then have them in your mind or are you explicit about like, “Let’s wait and see if this coalesces into a more common use case?”

I use a lot of GitHub tags on issues and I tag them using out of a scope because I’m the only one who close them. Usually, I have them at the back of my mind that, “This has come up in the past,” but every now and then I also search for out of scope and see if there is any usual trend, if there is anything that is being asked over and over again, and I’ll address those.

{{divider}}‍

It’s mainly written in TypeScript. Is that correct?

Correct.

{{divider}}‍

Can you talk a little bit about that? What made you choose that language?

CapRover is a full stack application. It has back-end components and it has front-end components. One of it was monetary pool. Having one language throughout your stack helps you to move faster because you can reuse code here and there, you don’t have a mindset shift when you’re working on the back-end and front-end. Initially, it was JavaScript all over. One of the changes that I made halfway through the project was to change JavaScript to TypeScript, which was perhaps one of the best engineering decisions with regards to helping the project to move faster. We used to have like random JavaScript version of NullPointerExceptions. One of the comments that I received on Hacker News, every now and then people rediscover CapRover and post it there, they’re like, “This is TypeScript. It must be slow.”

The point of CapRover is that it is completely out of the loop when you deploy your application. It’s not the stack that you run your application on. You run your application on Docker. If you pause CapRover as soon as you deploy your application, things work normally. TypeScript doesn’t have any negative points for this project. If it’s a multi-tenant project with hundreds of thousands of users with huge payloads, then TypeScript becomes a bottleneck. At this point, it is definitely not a bottleneck.

{{divider}}‍

Have you had any condescending remarks about why isn’t it written in Go or has that been fairly easygoing?

People made comments like that on the Hacker News, but they assume that somehow, the stack is being involved when they’re running their application. I usually go and respond to them, “By the way, this is just an interface when you deploy your app. Other than that, it’s purely NGINX and Docker.”

{{divider}}‍

Are they the only two things that you’re driving?

Yeah.

{{divider}}‍

Has the Docker API surface changed? Have you been having to keep up with that platform? I’m naive to this. I’m assuming it’s that mature and it doesn’t change that often. Back several years ago, was it a constant pain with Docker and NGINX upgrades breaking the platform?

Yeah. Every now and then we have the issue where a certain version will get released and there are some problems with it, but Docker aPI is properly versioned. Now, we are using 1.4. If you’re using 1.4, it doesn’t matter what Docker version you are, as long as you are in minimum Docker version of X, then everything will work fine. Every now and then, I upgrade the Docker API to the latest version and that breaks some functionality. That’s where testing comes in and manual testing. When I do that, I make it very clear in the remarks that, “We are upgrading Docker API. Make sure you do not update your production applications until you upgrade your staging and give it some time. “

{{divider}}‍

Talking about production, have you had issues with people using the product for relatively higher value projects and then stressing out that something’s broken? I don’t want to be rude and say like, “It’s for hobbyists because it’s a serious platform.” It’s an App Store like huge behemoth projects like OpenShift. It’s got a lot of similarities to OpenShift in a way. I always feel nervous for you because there might be these people coming at you and saying like, “I didn’t upgrade. Everything’s on fire. You need to help me.”

If you are writing an open-source project, you are not obligated to help them unless you provide some support. We do have commercial supports. I used to help a few startups but I don’t have time anymore, so I’m asking contributors to add their name in our commercial support page. A few of them have added their names and people would contact them if they need commercial supports. Other than that, I was surprised to find out that there are medium-sized companies that are using CapRover. I advise a few start-ups that are using CapRover, which is fine but medium-sized companies with 50, 100 to 200 employees using CapRover as one of their main platforms, I was blown away.

It is essentially Docker Swarm and NGINX, so it does work if you use it properly but you can break it. If you go and modify an NGINX configuration manually and of course, you can break it. Moving back to people asking for help, oftentimes, I do offer my help if I see an issue being opened by somebody who is running a serious application, I’ll try to help them. Nothing crazy has come up where people lost hundreds of thousands of dollars because of their platform being down.

{{divider}}‍

In terms of the commercialization, is that something that generally doesn’t interest you or is it more that you’re just doing it as a chance to hack on code that you don’t get to do during your day job? What are your feelings about that? Did you have any preconceptions when you started the, “I don’t want to be HashiCorp,” or something like that.

When I started it, I was considering it to become commercial at some point but throughout the years, my career has developed in a way that it is not a good opportunity for me to leave my day job and work on a side business even if it has a funding opportunity, which it does. Some of the bigger names reached out to me throughout these years and they asked if I’m interested in commercializing this platform. It’s one of those problems, which I’m not sure if I should say fortunately or unfortunately. I don’t want to do that because of my current situation.

{{divider}}‍

Are there any design decisions or are there any major tentpole features that it needs X and then it’s finished?

Are you talking about the features that I hope one day I’ll add it to CapRover?

{{divider}}‍

Yeah.

One thing that I want to add to CapRover is a better backup system. CapRover works great if you have a non-persistent stack. If you host your persistent stack, there is no reason that even a larger company cannot use CapRover. The problem is persistent data and losing the machine is just normal because it happens, disk issues, whatever. Now, there is a backup process but it’s fairly manual. I have scripts that I run manually on my machine to back up my CapRover instances, but I don’t want to release them to the public because you need to understand how they work and it doesn’t jive with the rest of CapRover. That’s definitely top of my mind for the next version of CapRover at some point.

I have smaller features that I’m planning to release in the short-term, but having a more reliable backup solution is on top of my mind. The other thing is creating a potentially non-open-source CapRover management platform where you can hook up your DigitalOcean, Amazon, ETTU, API keys, and click on a button and it automatically creates a CapRover instance for you. That’ll work great when I have the backup instance. That means that this is literally a true Heroku-like platform. Just click a button, it provisions the machine for you.

{{divider}}‍

In terms of the clustering and the things like blue-green deployments and stuff, can you talk a little bit about that? It must be like a vast majority of the users of the platform. I think it’s Hexnode. It’s got Bitwarden and a couple of side projects I’ve written, and a bunch of self-hosted stuff. It’s a single instance. I’ve got a script that backs up the persistent volumes to somewhere. If I was to think about using it in a clustered fault-tolerant environment, again, that sounds like a daunting problem, how did you go about designing that?

Creating a cluster for the sake of creating the cluster is super easy, Docker Swarm, and that’s what CapRover uses. Attaching a node, if you just run out of resources in your current machine and you want to attach a second machine, that’s fine. You can do that and it’s much easier than creating CapRover from scratch because all you need to do is to have the machine that has Docker on it. You just copy and paste its IP address, SSH Key and click connect, and everything will be done under the hood for you. The engine that decides where to deploy your application is Docker Swarm. When you deploy a new application, Docker Swarm looks at your resources and it says, “I can deploy this new app on the second machine.”

What CapRover does is behind the scenes, it configures Docker Swarm to do that. It configures an overlay network between different nodes, so your request will go to the main node and then it reroutes to the other nodes. It automatically anchors your persistent application to particular nodes. If you set up a cluster, when you create a database, you cannot move that database to another node because there’s data on that particular node. These are the little intricacies that CapRover take care for you. You don’t see that. It anchors that down, so when the system restarts, that particular application stays on that particular node.

{{divider}}‍

Are there any other open-source projects that do solve the problem around shared disk that you’ve been thinking that would solve that big thorny problem of distributed persistence?

Yeah, there is another project. These are commercial-grade projects, Gluster and Ceph, also simple solution could be a network file system, NFS system. Having said that, I haven’t invested too much time on them because I don’t think if you’re at that level, when you need that, when you need resilience, persistent disks across multiple nodes, you shouldn’t be using CapRover. It surprises me how people think they need to over-engineer their simple applications. You can run on a single instance, a web application that can serve hundreds of thousands of users easily. There are machines that have 120 gigabytes of RAM. Easily, you can purchase them on DigitalOcean and that should be able to serve all your traffic. If you are in that state where you need that, where you cannot handle a few minutes downtime where you’re migrating your cluster to another cluster, then you shouldn’t be using CapRover. You shouldn’t be looking for an open-source solution. You should be looking at a commercial solution.

{{divider}}‍

I do remember back in the early days of Flagsmith when it was called Bullet Train. I didn’t know that the CapRover existed, but we were just running on a big 64 GIG physical box that the agency that the projects came out of. If we had small clients that didn’t want to pay for ECS, that’s the super reliable solution as well up to a certain point. It’s way more reliable than dealing with failover. You’re right. People forget especially now. I’ve been doing this for many years and a $20 a month box now can serve an insane amount of traffic if you’ve got it tuned. We often forget that. That’s a great point. In terms of the license, did you spend a lot of time thinking about what license to choose for the projects?

No. I probably spent a few hours just to read through the licenses and see which one I pick. I did consider moving to a license where it disallows the commercial usage of competitive product. They started it because Amazon has started creating a commercial solution for them. I did consider that, but then I considered the fact that I’m not making any money off of it, so let’s have it for now to be a regular open-source project.

{{divider}}‍

What tools have you found most useful in terms of managing the projects in the community? You mentioned you’ve got a Slack. What other tools are you using to manage and communicate with people?

Slack is the live one and then for updates, I use MailChimp. In terms of the code managements, I make heavy use ofGitHub actions. They’re super useful. They can check the formatting for you, etc. These are super useful when people open a new pool request.

{{divider}}‍

Is it mainly you running the community and the project or have you got some lieutenants that work with you?

There are definitely people who are super active, perhaps even more active than me in the community. In terms of maintaining the code base, it’s mostly me.

{{divider}}‍

Have you got any goals or are you just curious and interested to see where it goes?

No particular goals at this point. I just want to keep this alive and it is serving me at this point. If I find another software that’s serving better than CapRover, I might pause CapRover but I haven’t found that yet.

{{divider}}‍

It’s super interesting you’ve got these organizations like Docker or HashiCorp, as I understand it, they seem to be in this huge trajectory and these projects that you guys and Docker seem comfortable with scratching the itch that you have.

In theory, it is possible to make this a commercial application.

{{divider}}‍

You must be getting emails from VCs every week.

Not every week but about every month or so. As I said, this is a weird situation because it doesn’t make sense for me at this point to leave my career and work on this. If it wasn’t the same situation, like a few years ago, I would have definitely jumped on to this ship.

{{divider}}‍

It’s interesting as well. I have this feeling in a way that the project’s better for it. I feel like it’s one of those projects that would be pulled potentially in a different direction. It’d be a different thing. If someone gave you $10 million and a shiny office, in a short amount of time, it would be a very different thing, right?

Yeah. This is one of the blessing of truly open-source, non-commercial projects because I don’t have any short-term plans to make this a commercial project. At this point, this is self-sustaining itself and it’s moving in a very steady manner and to a good direction where it’s becoming more mature and mature. It’s solving a lot of problems that I have to deal with day-to-day.

{{divider}}‍

The App Store side of things, talk a little bit about how that came about, because it’s an amazing experience from an end user’s point of view to have a fairly large complicated open-source projects that are difficult to set up by hand, like a one-click deploy. Was that always a design goal?

It wasn’t the design goal until I introduced the persistent directories. I realized that this is possible and the first version of One-Click Apps, I had it set up in a way where there was a JavaScript script that would install these apps. It creates a container. It would set up the environment, variable, etc. This is how it evolves and it’s a very nice case to study. After a while, I realized that there’s a lot of duplication pattern between a Postgres script that generates Postgres and MySQL for example. I thought about it for a little while and then I realized that I should be able to just define some schema and have a script that reads from a schema and deploy this. That became the second version.

After a while, I realized that, “Docker Compose does this. Why am I duplicating the same schema?” The third version, I moved to Docker Compose schema and I have a parser that reads the Docker Compose schema and deploys the app, which is super good because now they’re like a zillion different Docker Compose files in the wild and Docker Compose is that One-Click App magic. You set up your configuration file and say Docker Compose up, and it automatically sets up that app for you. It’s Docker Compose behind the scene.

{{divider}}‍

How much of the Docker Compose schema do you support? Are they actions that you can or just don’t support?

There is a limited setup fields that we do support. There’s environmental variable, volumes, and a few other things. There is host and other Docker Compose-specific things that we do not support. Unfortunately, Docker Compose doesn’t have an API like Docker does. Docker Compose has additional plugin. Otherwise, it would have been much easier to pass the Docker Compose schema and build it. There is an internal parser that reads the ammo file and it goes and deploys. It basically mimics what Docker Compose does.

{{divider}}‍

I didn’t know that. Do you think it does that intentionally on Docker’s side?

I think so. It is a different binary. You can install Docker without Docker Compose and CapRover uses Docker HTTP API to talk to Docker. If CapRover was not a Docker container and it could talk to the CLI interface, I’m assuming that would have been possible to use Docker Compose. Now, there is over 100 apps on the One-Click App Store that are manageable using the fields that we do support. There is a field called CapRover extra, which is in addition to what Docker Compose has, and it has things like container HTTP ports, because it needs to map it to the right NGINX configuration. For example, I want to add more things to this one. This one is more important to me than other nonsense Docker Compose fields. I want to add, “This app only works on HTTPS. This app only works if you have web sockets enabled.”

{{divider}}‍

Have you been surprised of the take-up of the One-Click apps? Once a month or so, I log into my mine to see if it needs an upgrade or whatever, but I always have a little browse of the One-Click Apps because there are always projects. It’s a great way to discover projects. The scroll ball gets smaller and smaller every time I look at that list. Does that surprise you?

It has and when you look back and you look at the success of all these giant platform companies like Google and Apple with the success of App Store and Google Store, this is essentially it. You just give people some platform and they go and build amazing integrations with your platform. It’s like what you said, there were a number of different, interesting projects that I discovered by seeing the pool requests coming to One-Click Apps.

{{divider}}‍

You’re still manually approving them. I need you to go and do something at your computer for my app to appear on that list.

If you wanted to appear on the public list, yes. I added the functionality so you can have your depository that you can deploy.

{{divider}}‍

Are people doing that?

They could be doing that. I don’t know. That was another thing that a lot of people asked for, so I’m assuming that a lot of people are using it.

{{divider}}‍

When I look at that list because I’m like, “This must be a drag dealing with so many pool requests for this catalog.” It’s great to see it.

The ones that you see are from the public list and I have to manually approve them and it has been a tough job because that repository is the most active repository among all the repositories that we have. One of the ideas that I’m tinkering with is to extend the third-party repository and have people to have a list of third-party repositories and they can manage it on their own. I’m going to make it explicit in the UI that, “These are the third-party contributors, install it on your own risk.”

{{divider}}‍

It’s interesting because that’s almost exactly how OpenShift works. It comes with like the Red Hat repository, but then you can point it at Here Be Dragons repositories. Almost word for word, it’s almost the same design. It’s interesting. I really appreciate your time. Have you got anyone or any projects you want to thank or say hello to before we close up?

I’m not a front-end engineer, so I was amazed by the level of abstraction and design framework for front-end design. It was amazing. It helped me a lot to make things move faster, so shout out to them.

{{divider}}‍

Kasra, thanks so much for your time and also, legitimately, thank you for the project. There are applications that I run on my server that I use every day that are being powered by it. I appreciate it.

Thank you, Ben. It was really great to talk to you. Take care.