Jenkins

Check out our open-source Feature Flagging system - Flagsmith on Github! I'd appreciate your feedback ❤️

Episode Transcript:

I’m excited to introduce Kohsuke Kawaguchi. For those of you who don’t know, he revolutionized the world from my point of view 15 years ago. I will let you introduce yourself and your background, Kohsuke.

I grew up in Tokyo. I took up Programming. I was originally involved in XML and web services, and I thought that was cool. Somehow, I stumbled up from building the software that eventually became Jenkins. It’s the CI/CD automation systems. That’s probably what I’m best known for.

{{divider}}‍

You’ve got an interesting project that you are working on at the moment.

I found developer productivity and I’ve liked it a lot. I’m stuck with it and that’s my passion. After 15 years of doing Jenkins, I’m working on slightly different parts of developer productivity. That’s about a long testing problem. Hopefully, we get to talk more about this. In Jenkins, toward the end of the week, if I wanted to make an online change, I have to wait for an hour for the whole test cycle to complete. I knew that was wasteful. In talking to different teams, I knew I wasn’t unique. In fact, some people say, “One-hour testing, that’s wonderful.” I’m happy to trade my place with you.

{{divider}}‍

I’m going to dive in and ask a question about that. That has been something that I have been reading about. Other than it feeling wasteful, do you think the other tangible issues with having a long CI/CD build pipeline loop?

When the cycle gets longer and the entire idea of CI/CD is to use for the cycle when I was working for Sun Microsystems, initially the build will not only be happening nightly. The QA was only getting the new bits the next day. Everything is in the strips over time and slow. There were a lot of parallel comparing programs that make tracking everything so painful. We tried to reduce that. We have always been trying to reduce that cycle as an industry. Where is the longest pole? By far, in talking to different teams and different things, it’s the process of building confidence, the change, otherwise known as testing. The testing is a strange beast. I never came across anybody that felt like, “My testing is perfect. The change passing the test, I feel great.”

Everybody is feeling naked, yet they are also struggling. It’s a long journey on their time. It’s like a body of density. It’s like finding a needle in the haystack. We can only do it in a very incredibly low, efficient manner. We try to make up the efficiency by having a large volume like a thinned down orange juice and try to get enough vitamin C by drinking multiple gallons of all these kinds of vitamins. Drinking isn’t exactly known as a great beer or holy beer either. We had our own fair number of people like this in our environment. I remember at one point, I had to do it a few times. That was embarrassing but that’s the norm. You don’t hear these stories when you show up at conferences. When I go to talk to people about this and ask about their talent needs that go from the things everyone has. Part of the reason on this test they sign is you have a big system. Any successful project grows. In the ten years making this, it’s usually incremental and small.

If I’m standing here, why am I running all the tests that are over there? We have these things but there’s no way to effectively communicate those things to a computer. We’re stuck around the whole desk scoot. If you have enough budget or if you are on the Cloud, you can run the test in parallel. Even that is difficult for some people, embedded software. They just can’t clone their hardware. Intuitively, we should be able to select the subset to the tests that are “meaningful.” I saw the paper that came out from Facebook. It has been a while. They had this program back in the city order of magnitude more than what normal the best of us have. They have more of a decoded base. This is not microservice or anything like that besides what they offer, testing changes tech in every single minute.

They have to come up with something like this. In the paper, that’s very simple what they did but they were able to cut down the test execution cost by half or something. They must be saving millions of dollars on just the compute cost alone. I saw that. I thought this is simple. Some people did it. The thing about Facebook and Google and the likes, the software development environment is so abnormal for the rest of us. It is built on top of their own build like a version control system on top of their build system on top of everything else. There’s no hope that what they produce is ever useful for the rest of us. That station was very similar to when I originally built Jenkins. Back then, I regularly run into people who built their own CI system for their team. What everybody did was incredibly fine-tuned to their particular way of doing things. It’s not generally useful. I suppose I can produce those more generally.

{{divider}}‍

We used to write Java software. I remember very clearly the day that I downloaded Hudson as it was then. I gave it a twenty-minute test. I had done this with a bunch of other tools prior and it all failed dismally. I remember very clearly Hudson being like, “Oh my god. I did it.” It was fairly straightforward. What I was trying to remember was what we used to do before. It’s a long time ago. I seem to remember there were a lot of bash scripts, SCP and things like that but it didn’t have anything. Not as old as I am. It’s probably a bit like how people existed before the internet or something, the amount of variability in terms of people building more followers on their development machines, and then pushing them to production stuff. First of all, thank you because it did change my life. I remember we were running. You were working at SUM. What was the impetus for you? Were you the one person on the planet who was doing that bash script and then thought this needs to change?

You are giving me way too much credit than I actually deserve. There was a golden standard back then that was called CruiseControl. They had a GUI. The people were supposed to author the configuration file in XML and align it to the service. That was state of the art. Most people were doing a cron job and like a shell script. I have done a fair amount for this. Two things happened around that time. One was a volume of things that are going up. If you are only dealing with 1 or 2 products, then a cron job is fine. CruiseControl had to be one server or one build. If you are having two services and one library, you have to set up two CruiseControl services.

{{divider}}‍

I was running my agency at the time. We had dozens of clients. I was trying to remember what it was that didn’t work for us, but I guess that would have been it.

If you are only dealing with one system and one build, then there’s no need to drive lots of computers. All these systems could only deal with one computer. A CI system can only use that one computer. I don’t think about it still. Anything like that is probably the key innovation. It seems like I build this at the right time. People’s workload was exploding. One system that can deal with lots of different software deal that’s useful. You can handle all like distributed deals. Not knowing in a way that was expected now, so that had people. They will go to the cost. The other thing is the planning. The product managers and engineers that we have, the more they are small, they tend to think everyone else must be doing the way they are doing things.

It’s easy to produce something useful for you but it’s difficult to produce. It’s very useful. I was able to pick up to the target. That combination is open source. I was working in the Java EE and had a team that’s building a platform over each section. Someone was saying, “You are building a Java application and web application, build on top of this.” I looked around and I realized I never built that on top of the platform that we are producing. It’s normally my entire team. People are not eating dog food. It seems like there’s something wrong with their cycle. I needed to write some stuff here that uses my stuff. That’s how it came to be new.

{{divider}}‍

What was your inspiration? Was CruiseControl one of the inspirations? You were like, “I need this but I need it to be slightly different. It needs to be maybe distributed in multi-project.” Was it just as simple as that or was there anything else?

I used to be a guy who breaks field all the time. The people occasionally called me, “You touched this file an hour ago and it doesn’t compile for me, what’s going on?” I’m sharing that file. I was like, “This is embarrassing. I needed to fix it. We need to have a program to get it before my colleagues.” I discovered that I wasn’t the only one who’s causing these problems. It’s a more universal program but that was also another motivation. There was also another software called Damage Control that was written in Ruby, Python or something I forgot. That’s had a much better UI of this compared to CruiseControl. It also was one server that dealt with multiple projects. I liked it. I was using it for a while. It wasn’t very stable. Maybe it ate too much memory or load. I looked at the source code and no wonder, “I can do this better.” I originally started as a lead implementation job in Java. It quickly grew out of there.

{{divider}}‍

Was the initial version designed purely around building Java platform stuff? Did you have an aim of it being a part of your solution?

One of these common traits that make this platform people and the engineers should be following is that we tend to overgeneralize. My immediate use case is indeed to build the Java project out. My team was doing at the company. At least in my mind, it shouldn’t be Java-centric. To fix the abstraction entirely existed from the very beginning that this was not meant to be Java only thing. The same thing for the volume control system. You are using CBS back then. Maybe your audience might not even know what this thing was. That was what I wrote in the beginning. I also knew that that was the only version of the system in the world. The strength abstract of what creates a commonality between CBS and the limitations.

{{divider}}‍

Was this a project that was immediately or initially conceptualized as being an open-source project within? You stand an open-source then.

This is one of my hundreds of open-source projects I created back then. It evolved from the very beginning open source, but that didn’t mean that it had any traction. For a year or two, I was only working on this myself. My people around me are being a user. They kept giving me feedback and that kept me going. Somebody else noticed this. He’s a guy from the Nepton team. They are more two-minded. He started packing on this a little bit. This was also around the time the people are leaving from some left and right. It’s a slow decline post the dot-com bubble. I suspect that some of these people took that knowledge elsewhere. They probably introduced. They evangelized the software wherever they went. In the summer of 2007, I noticed a sudden influx of new people. It started in 2004. There was a long three or so years in which we had that. I had the time to build up some massive features before and glad it helped.

{{divider}}‍

How did you manage that guardianship? Did you have other colleagues that were working on it with you? Was it just you and the community outside of your organization?

Someone did it easygoing in that regard. It helps that back then the CEO is saying, “We are going to open source everything, Java. Don’t worry about the money. We hope we get out alive.” That could be the tailwind of that. It was also useful because the software wasn’t. Our department also has consulting stuff that records the whole Java EE GlassFish. I had no push back or anybody’s raising any stinks about fundable open source projects I was seeing. Lots of my colleagues were doing the same. I studied in that way. I never heard anybody making any complaints there.

{{divider}}‍

What tools were you using to work with the community then? This is all pre-GitHub.

One of the struggles in the open-source projects, or I suppose the expected implicit model back then was we have this hideous team set of people called commuters. If you are trying to join any open source, you have to work your way up to it by sending parties, email and stuff like that. That process for me is somewhat painful. I hated that. I was contributing to some other open-source projects back to Cruise Control this process of proving yourself, as far as I’m concerned, I didn’t need to prove myself anything. The bar gets set up. If you are trying to work in a single Sunbelt or if you are trying to build a house together, there are some level of collaboration that needs to happen. Distribution is interrupting the open-source project where you don’t even know these people. It tends to be painful.

Architecturally, I thought we could solve that problem. The beauty of the plugin is everybody gets their own sum bulk. I don’t need to pre-watch your crazy idea before at least you do it. You can produce that and then the end-user can combine them. There’s something in there where you get traction. If not, then it’s going to die away. It’s more reflective of what happened in nature perhaps. I did that in technology rising the introduction of plugging system. What that meant is I didn’t have to set this commuter bar high at all. If you are building your own plugin, I could go for it. I created this voice. You need to ask for the commuter seat and then it will be granted. I need to keep my eye on the foundational core path. I deserved the right to be part of everybody’s hands maybe by these chains and I said, “I only needed to do it once or twice.” Lots of people exploded a different perspective. Different parts of the software industry were doing things differently. They needed different features. That worked well.

{{divider}}‍

Did you design the plugin system with that in mind or is that something that you discovered after the event?

We take more time to crystallize why it works. The open-source project I was involved in prior was Java data minding tool. We discovered that Java open sourcing was something that doesn’t drive contributors magically. What it did was the other interface. We opened up this compiler so that people can change their behaviors by writing additional projects on top of it. I saw that this got the traction. If I put myself on the other side of the sheet, I can see that it worked for me as well. My use case might be doing this esoteric for you, but I didn’t have to convince you to do it. I knew I needed it and I can do it. Don’t stop me from doing that kind of attitude. I knew this would work. I made a bigger investment. I made it more real plugins, updates and all that. At the end-up process, I got exactly why this is working.

{{divider}}‍

Did you worry about the quality of those plugins? Did you do any review of them in any way? Was it an open market for whoever?

We didn’t worry about it at all. Now it’s standard. It’s crazy. I didn’t even have any authentication systems. Anybody inside the sign could make an alarm to any processes on my computer and what can go wrong inside. Back in the day, they were easy times. At some point, it needs to be successful. The problem is not the volume. It’s the clarity. If you look at the app stores and the likes, clearly lots of effort were spent to try to trim those. Later in the life of the project, we started doing it. That happened over time.

{{divider}}‍

The other interesting thing about Jenkins and Hudson is most projects or software concepts start off with a commercial version, and then someone builds an open-source version. An open-source product becomes popular and becomes more of a standard. It’s interesting in the CI/CD space. Things happened in the opposite direction almost it seems, especially around the post Docker becoming popular. It’s interesting to me. We were running our own Jenkins server. We had our own build nodes. It worked fine. It had been running for years. People are going, “Have you seen this new circle CI?” I was like, “Why would I pay for that? We are the closed source platform. I have to send my source code somewhere when I’ve got something that works.” I wonder if you had any thoughts about that.

With our bar, the expectation that we expect around the ease of use that we demand is going higher and that’s good. Back in the day when the expectation was, “You are going to open up the VI and write the XML file by looking at the website too.” After you clone the box the ease of use than being able to do that posting you needed to do the innovations. They used to have it. The staff to maintain this service keeps it up and running, updates. It adds up and doubles the work. It makes all sense of the world that if you can get that handled by somebody else, that’s worse. The circle CI certainly solves our problems. That’s great.

The other thing is developing valuable software or service. It’s a lot of engineer’s work. Back then, it can be only driven through people’s volunteer, time and passion. It only allowed so many boundaries to be spent on that collection that should be in the world. Web developer has this tendency of not knowing how to ask for more money from the Sunbelt. The normal changing of what we are is we need the tools. They have to be more productive. It’s worth that money for the business to make us more productive. It’s a good thing. That creates more investment in making tools of the trade better. That’s also encouraging in my mind.

{{divider}}‍

At what point did you realize that configuration as a code? I remember the first time I started using Jenkins, it was super simple but then a few years later, setting up a new project was hundreds of clicks. I’m not complaining. There was this moment of clarity where someone realized that you could define these jobs in your code. I can’t remember who the first real champion of that approach was.

Chef and Puppet are where they taught that idea of item potency configurations code. The idea spreads. I also think it’s a function of a volume. If you have one thing that we applied back so many times, it makes sense to go back to the code. Jenkins, when it came to what you were talking about, the multiple projects but still is the 5 or 10 range. That’s not a ton of items. Using it a few years, it went up. It’s quite common in the late 2000s to see people having thousands of set up jobs set up on their Jenkins. At that point, it’s clicking around. It’s simply not practical. We have to reinvent ourselves in Jenkins’s sense. It’s one of the reasons we did the pipeline in the Jenkins file. We also later gave the project to create this whole configuration code system in Jenkins. That’s also until now a very popular part of it.

{{divider}}‍

Was there ever an attempt by anyone or by that area of the industry to try and design a standard around it? I feel like it’s one of the real shames and unusual in modern software engineering. You feel like you have to choose a platform. Not a huge amount of lock-in but you’ve got a little bit of lock-in in terms of that non-standard configuration.

As far as I know, there was nobody who thought they would be good ideas. I wonder why. It takes special market circumstances for that thing. The station that I’m common is where the sound was in. There’s one super dominant guy, which is Microsoft taking over the world. The rest of the “smaller companies” like IBM and Oracle had to do something. The only way to compete with those is like, “We can’t fight by ourselves. We need to join hands,” in which case the demands are standardization. Maybe that didn’t happen. Maybe now is the time. Amazon, Microsoft and Google, if they are seeing any of these companies that people somewhat might see, it would be good.

{{divider}}‍

The timing around DACA becoming dominant was maybe confused things a little bit. I feel like it’s a perfect platform for running CI/CD. You don’t have to worry about what stuff is on your machine or anything like that. You can have reliably reproducible builds and things like that. Some platforms have adopted that much more strongly than others. I remember we used to have light builds fail because there was a global Python package installed on the build server or something. I feel like it’s a shame that there hasn’t been any standardization.

It doesn’t raise enough about paying I suppose or that somebody should. Maybe this is the opportunity to open up a foundation, why not?

{{divider}}‍

The project has got to be one of the most well-known open-source folks that there has been. Who chose the new name? Was that you?

At some point, we knew we need a new name. To see people including me, most senior people in the project decided partially so we liked the name Butler for the theme. That was the reason I chose Hudson in the beginning. We wanted to keep that theme. In Wikipedia, there’s a page that we saw the topic of the fictional butler, sort of the A to Z. The person’s name was Alfred, the butler for Batman. You can dispute the trademark thing. I didn’t think it’s a great idea to pick off how you turn or conform the company who cares about these things. I know it’s far out from the easiest king that uses a lot of slaves. I would rather we dodge that one. It became the Jenkins style. I believe a butler that shows up in a cartoon called Scooby-Doo. I think it’s Daphne. Every time I bring this up being conversational, I kept telling people, “I need to work to start collecting of these fictional characters.” I still haven’t done that, maybe I need to work.

{{divider}}‍

I’ve got no idea. It’s an old American Hanna-Barbera maybe. I don’t know. It must be on YouTube. Was there ever an existential concern that it might split in a way that causes a loss in damage?

It’s a risky move. We didn’t use it. We didn’t want to do it. It helped. The guy in Oracle who was creating this drama didn’t understand the open-source people psychology, how it felt. It almost came across in the cartoon. That’s easy to rally people around. Let’s also be honest. Back then, Oracle didn’t have a great fictional character. That also helped. In some sense, we knew already that long before we pull those visible two years, every person in the community was with us. In fact, we only lost one contributor and added up to 200 or so. From a substance perspective, we had everybody we needed.

Nonetheless, it created some fear, uncertainty. They generally were closely involved and you see some sign of trouble then the right reaction like, “Let’s see how it plays out.” We saw that in the adaption number 4 in 2, 3 months, it stagnated. It’s something about how that costs. At some point, it can become more of a principle. You were not doing it because the corporate overload wanted you to do something. That had to be rectified. In the end, we will be navigating and develop. That’s a painful experience of the year going through it. It’s years ago, so I can imagine.

{{divider}}‍

It’s funny that it seems this perfect reflection of that deal of Oracle buying Sun, this guy taking this popular open-source project and trying to slap a price on it. I remember reading about it. We were using the platform. It was a critical part of our company infrastructure and thinking. It always felt like there are four kits and off they go kind of thing. I was curious to know if it was slightly different on the inside.

This is probably true in the diplomacy too between the nation-states. I feel like too much is dependent on the relationship of the people who are at the top. I felt like if there was a great mediator or some adults in a sense, this whole thing might have been avoidable. Since then, when I see some high drama open-source attempt, I wonder if there’s something I can do.

{{divider}}‍

There are not many people with your experience. You can’t buy a book on managing massive open-source community projects. Maybe you can, maybe someone’s written it. Are you still involved in the project?

Not anymore. When I launched it all, we did this. My role had been assisting over time. Towards the end, you have a lot of capable people. I wasn’t primarily involved in it anymore. We did a formal ceremony years ago when we launched this company. I’m not doing anything anymore.

{{divider}}‍

In terms of the commits that are going into the projects, the ones that were going in towards the end of your tenure there, was it mainly enthusiasts or was there a larger organization?

In the beginning, I was the sole main driver and then it became more of a community-building aspect. I built these social structures. I believe they are on the commuter seat so that other people can productively write more code. Later, my role shifted further. I’m trying to shine the spotlight on their faults that are more allowed to be impactful. When this becomes a big dude and everybody can mind their own Sunbelt, it’s difficult to rally lots of people around that. This pipeline is one thing. This is shifting how users expect to use ink and say, “Patch everyone.” Unless somebody is waving the flag and try to rally the people behind it, it’s not going to happen. I feel like a lot of the role of being a visible person in the communities that look like a disguise thing, an amazing thing here. We all need to rally behind this one. There’s a different person who is driving that effort but I tried to get attention over there. That became my role.

{{divider}}‍

Were you tempted to try and cut version one complete in and start the pipeline-based projects completely from scratch? It seems like a Herculean effort to try. How many plugins did you have at that point? You must have had thousands.

The good news is that we didn’t have to bring anything. There are things for the moment then. That’s where you get the critical parts of the theme from being standing up and people stop using it. They use those bug reports. That’s the risk. We did burden the Jenkins too. I forgot when but that was also a while ago around this pipeline corporation’s code. It was difficult. The reason people were writing this code are so that they can use it on their own. They have an existing contributor, how they view the investment in the older way of doing things, trying to bring in the whole new subsystem and making everybody use it. Also, compatibility doesn’t matter. We can throw away the whole thing. It’s all a very interesting experience for me.

{{divider}}‍

Do you attribute any of the success to the fact that it was based on Java? Java always had a strong packaging and deployment story. I always thought it was amazing how it was this monolithic thing. It didn’t rely on passing off to SSH servers. It had everything encapsulated in one file.

It was helpful back then. This was in the language of this type. In the tagging systems, you do need to have this contract, which is really defined. That was helpful. Java also had the program pieces to come together up the runtime, which is another foundation of tagging system. That was crucial. I suppose also probably maybe my role was limited because I was at Sun where Java was invented. I felt like Java has a 3D dominance on the server-side, maybe except for dot-net but that helped. Towards the end, in 2010, the language became more prolific. The fact that you have to be everything in one language, that negative became a little more pronounced. Also, a long-running server having a single process there like any single plugin blowing up giving the whole thing was problematic. That also stopped the drag. They might think if people are doing this kind of system, I don’t think they will choose a single process. Everything runs in the memory model for good reasons.

{{divider}}‍

Tell us a little bit about this in terms of leaving Jenkins and starting Launchable. Was that concept in your head for a long time running up to that?

Part of that is that I used to think I need to be holding or else the sky is going to fall. I looked around, I guess I don’t need to do that anymore. I thought all these people are quite capable. Back then, I was a CTO at CloudBees. One of the things I did was to talk to the software development team. It’s almost like I extract the stories of how these actual people are using this. What more are they giving? What’s driving them to use Jenkins and all these kinds of products. I did quite a bit of those. In that process, I discovered that the challenges that you had are often quite glamorous. It’s not high volume. I wanted to do something about it. I also felt like the ten years at CloudBees have taught me a lot of things in council like the small startups growing bigger, the context in which the software gets built and sold, how that changes the software engineering itself. I felt like I wanted to put this learning into practice so I can see how it goes. That was all combined into one thing then became Launchable.

{{divider}}‍

What’s the story with Launchable? Where are you at as a company in a project?

We built the initial products working together with some customers. The challenge is to work for class tools. It’s easy to produce stuff that well. I have confidence that I could produce that works but, whether that can be sold to sustain a living, that’s same value but different matter, we wanted to verify that. From all that we work those as teams, we had the test challenges in a limited capacity. We are glad to be opening it up. We announced that we are in better taking your own and people signing up. Another thing I was passionate about this round is I’m well connected in the Japanese tech scene. I came from there. There are a lot of capable sets of people. That community is simply disconnected from the rest of the software development world.

I felt like that’s the missed opportunity. It felt like a market is disconnected. One side in the salary gets depressed. I wanted to bring those two worlds closer. I have our engineering team in Tokyo. It’s great software. We shed some spotlight on both sides. It looks like another set of the talent pool. We know how hard it is to get developers anywhere in the world. I don’t think we can afford to overlook those guides over there. It’s not here like where we are the only company coming out for different reasons. They don’t know how that works. You can go work for companies like ours. They run the software to be sold and to be open. I take that and all the deals are fair. I’m trying to play both sides.

{{divider}}‍

That’s the first time I have ever heard the phrase, “I’ve got my team in Tokyo.” It’s crazy.

I give the credit to Ruby. Those projects put the Japanese tech scene in the major arena. We need more of those.

{{divider}}‍

I realized that we are getting short on time but I did want to ask you, what was the maddest plugin that you ever saw at Jenkins? I remember that was another thing we used to have fun with at work. You would log in and someone had installed some crazy user interface hack into the platform. That was always fun.

The strangest plugin that I saw was done by one of the telco companies in Europe. They had a CEO speech on the corporate internet. Somebody grabbed them all. They use one random quote for their CEO on every face. Hopefully, it was meant as a joke, nothing serious. Somebody went to the trouble of doing it. I also heard from another person who said that he fixed this up to the 90 cams. Jenkins was driving the webcam every second, they think of a new frame and doing a comparison between the previous frames, and there are too much movement to keep it.

{{divider}}‍

That was a testament to the plugin system.

I suddenly didn’t even think this is to be used for you that we would be talking. Those were the few things that I remember.

{{divider}}‍

Kohsuke, thank you so much for your time. That has been a fascinating chat. I want to thank you again for all your work and your open-source efforts.

Thank you. This was fun.

{{divider}}‍

Take care. See you.