Ansible

Check out our open-source Feature Flagging system - Flagsmith on Github! I'd appreciate your feedback ❤️

Episode Overview

In this episode of The Craft of Open Source, I was able to catch-up with Michael DeHaan, Founder of Ansible (now part of IBM/Red Hat). We started out talking about his beginnings in programming, even back to his first computer. From there, we learned about how he got his start in Open Source while working for Red Hat and the journey he went on with Ansible. One of the lessons that I took away from the interview came when I asked Michael what his strategy was for Ansible. Here's what he said:

"I spent a lot of time thinking about the documentation. Someone gave me some good advice when we started. He was like, “If people aren't successful trying this out in about 30 minutes, they're going to move on.” You have to make somebody successful within their lunch hour."

Hope you enjoy!

-Ben

Episode Transcript

My first question is who are you and what do you do?

I'm Michael DeHaan and what I'm doing is I'm helping teach a computer science course at NC State. I have a few other side projects I was working on here and there. The future after Ansible is up in the air yet. There are lots of ways and things I could work on, and I'm still trying to figure out what that's going to be.

{{divider}}

What was your first computer?

I had a 286 when I was growing up. I was 3rd or 4th grade for me. I don't even remember how big the hard drive was. It was super small, but it had the 3.5 and the 5 1/4. I spent a lot of time trying to op, then I skipped that and then had a 486 shortly thereafter, which was a little more capable.

{{divider}}

Did you start programming on that first 286?

I don't exactly remember. It was probably Basic on the 486 because that was about the time when QBasic was available. They came with Gorillas.basic and then the other stuff. It was pretty easy to get into a video mode. I wrote some screensaver things and a few games and stuff like that. I did a lot of programming during class in high school. I got a TI-85 graphics calculator. I wrote a nice little Minesweeper clone in TI-Basic, which is funny because the screen is so incredibly tiny. It's like 6 or 8 lines or something to be able to see everything. That was my first popular program before all the dissembler for that thing.

{{divider}}

None of the teachers knew that?

No, I got in trouble. I did write a few math programs and they let the other students use it. It was a little Taylor series approximation thing or something. They got to use it on the test. That was pretty cool. 

{{divider}}

Do you remember the first time you discovered open source?

My getting into that was weird. In college, the Linux machines were starting to be available and they were vastly inferior to these hilarious boxes at the time. I wasn't a fan. In my first job out of school, I was working for a piece of IBM that later got sold off and they were doing devices to control storage and stuff. I very quickly learned that the Linux API is so much better than the Windows ones. That was where I started to like it and get involved in programming. I started to do a little bit more embedded stuff. I didn't do any open source projects until I joined Red Hat in 2005, 2006. What happened there was we had these emerging technologies group thing and you could build anything you wanted that helped customers. Some of us had different areas we were working on. 

One of the things that we noticed was there wasn't any good solution for automating a Pixie bare-metal infrastructure. Also, KVM and Xen were starting to be available. It's something to try to automate those installs. I wrote this thing called Cobbler, which got to be fairly widely used back in that time. That was my first open source project. The reason I got interested in it was my boss left the company and I didn't have a real boss for about six months. I decided I was going to let the internet and the IRC channels be my boss. I started listening to people and saying, “What do you need? How are you using this? What would be good for you?” That was a pretty good way to develop things. I don't think a lot of people were necessarily doing that inside the company at that point.

“I didn't do any open source projects until I joined Red Hat in 2005, 2006. What happened there was we had these emerging technologies group thing and you could build anything you wanted that helped customers. Some of us had different areas we were working on. ”

{{divider}}

That would have been around the time that VMware have something similar that you were then building an open source version of?

I don't know if they did or they didn't. IBM had some cluster script. I think it was xCat or something. I never used it, but everybody was writing their own Pixie automation scripts because Pixie was fairly simple, but there wasn't something you could go download that was a well-designed thing. Everybody had to build their own. It was getting together everybody's experience of what they wanted to do. You started off with Pixie, then you added virtualization and then DHCP and DNS management. Also, control over some power bar interfaces and IPI power and things like that. It got to eventually control some relatively large deployments and things. I had all these people collaborating together, using it and giving their feedback. That was a cool experience.

{{divider}}

Did that lead on to the Func project?

What happened with Func because we decided that we wanted another project and I got together with Adrian Likins. He was the guy who wrote up-to-date, which was the precursor to YAML. They were a cool crowd of people. We were like, “Let's make another project that's modular and has an interesting community around it.” The idea was to fill in some of the gaps between things that you would do bare-metal provisioning for and things that you would do configuration management and at the time Puppet was growing in popularity relative to like CFEngine, the things that filled in the gaps. I don't know how widely that got deployed, but I know Fedora was using it in some of their infrastructure, but that was a pretty fun early project. Some of those ideas ended up influencing Ansible later because I always wanted to build a config tool on it at the time, but other things happened.

{{divider}}

I'm trying to remember back because at the time when I first heard of Ansible, Chef and Puppet were already quite well-established. I went back and looked at the very first commit of the entire Ansible project, which I don't know if you remember it.

I remember what it was, the reasons for starting things. I felt that the configuration tools were following me around to an extent because I ran into a lot of Puppet users when I was working at Red Hat. I worked for Puppet for a very short period of time. I was at another company that was trying to do an integration. I wanted something better and different because it was still taking me sometimes several days or even longer to get a setup working and because of DNS problems, NTP and trying to figure it out. Another thing was I didn't like my job at the time and I wanted another project because I was enjoying the open source community and I missed that.

I was like, “Let me see if I can fix this,” because everybody says that SSH doesn't work and all these other things. Can we build a system to try this out and see if it's going to work? It got going pretty quick. One of the things that was fortunate was a lot of folks already had that same idea that I wanted this. I heard a couple of people when I was when I released it. It was like, “I would have had something in a couple of weeks,” or “I was working on something this myself,” or “We had something this internally.” It was maybe not too revolutionary to try to build something that was SSH and push-based and didn't have this whole fleet of management agents running and stuff like that.

{{divider}}

I do remember I spent a day fighting both Chef and Puppet because I was running a software agency at the time and I was like, “We don't have time for this.” I remember the first ten minutes with Ansible and it was like the light bulb went off and it was like there's so much less impedance in you trying to get stuff done. Was that something that was a design goal right from the very start?

Later we overemphasized the simplicity and the marketing of it all, but it was like, “How can I make something that was easy that I don't have to spend my entire day with?” The one company I was with exploded and I was working for Cisco for a while and Ansible was already fairly popular at the time. I was noticing that they would spend almost six months trying to cut the ties with OpenStack. By the time they got done with that content, the new OpenStack release would come out and they'd have to do it all over again. It was a full-time day job for 5 or 6 people or maybe 10.

I saw the pain of that and how much effort was in it. I was trying to make something a little bit easier that was more like a grocery list. In the end, as we added more features, people started to write playbooks that I necessarily didn't like so much because they were getting more complicated because it had more language features. It was supposed to be like, “Here's the list of packages I want to install. Here's the list of files I should copy over.” It’s not what I'm going for.

{{divider}}

You then end up with these Frankenstein monsters. That's one of the problems if you are an open-ended tool like that.

It looks like a bad programming language because it never was meant to be one. Some of the pure infrastructure as code approaches could be potentially better if you do need that level of flexibility because then you want to break into code. You could, but then you're also trying to keep people from having to do it. It's weird because there are all these other YAML tools and things. How do you remember all these different YAML dialects? They don't have much validation. We had a lot of modules for all and some other things that, but it was like how do I make something that the automation should only be a very small piece of my job, not something that I'm using every day? Of course now, even with Ansible people, there are people that are using it all day long and that wasn't the goal.

{{divider}}

I’ve worked with people whose entire job is basically CI/CD pipeline building and looking after and expanding for larger projects. You wonder whether they do stuff, but you at some point, start to think what is this for? When you wrote that first and push that first commit, was that yours? Who were you working for at the time?

I was doing it as a nights and weekends thing on the side from a small local systems management startup that was not doing too well at the point. It was an unrelated tool that I was building. It wasn't part of a company or anything like that. I would do it a couple of nights a week, then it started to be a couple of nights a week and then a day on the weekends. Pretty soon it was every day where you wake up and you'd get lots of commits and you'd have to deal with them. If you didn't deal with them, the more would pile up. It's an interesting thing. This may dovetail into another conversation. 

We were talking about a little bit of this over email. Do people have as much free time now to try out new projects and contribute to them and research new things? A lot of organizations seem to be more and more scheduled. They're in so many more status meetings. Some of their agile stuff has gotten way out of hand. People are tending to chase tools that they know or that have been pushed by developer advocates or are popular for their resume experience. Everybody is well into Kubernetes or whatever. I tried to start a few new things and I’ve gotten very little traction. I wonder if that whole way that Ansible was adopted back then part of it was timing and it was the right tool for the right time.

Also, it's very hard to get people's attention with it now because at that time there was a lot of system ends that were starting to automate things. The curve was building up. Now that everyone's already doing lots of programming, is that interesting anymore? They're also expecting something that's completely done and completely polished. Starting something new and then getting people to be interested in it as it's younger and it's smaller, it's a question of mine. Is that still something that you could replicate? I thought back in 2012, for Cobbler, this is the model that works for Func to some extent. This is the model that works for Ansible but that model has changed a lot. Before, it was identify a user community and figure out what their problems are, build some cool thing and then people try it out and you talk to them. Now people expect a much higher level of product completion.

{{divider}}

We work a lot with React and Kubernetes, and I often wonder how much money Facebook have spent on React and React Native because it's got to be close to nine figures. The amount of effort and the amount of engineering that's gone into that the open source project.

I wrote a source code scanner. This is more for the education stuff I was doing, but to see who's working on a project and what are the trends over time? React Native is 100% Facebook. That was one of the interesting examples. Frameworks with thousands upon thousands of dependencies. It seems to be a lot further away from the origins of some of the earlier open source projects where they had less dependencies and they were newer and then it feels you have to have ten years of experience and stuff so you keep up. I don't think I could keep up with what's going on in Kubernetes land now because I’ve been doing too many other different things. Maybe there will be a trend to go back and start over from scratch and say, “This was an interesting experiment, but I remember these old ways of doing things and they were fine too.”

{{divider}}

Do you think it's harder to have a project where it’s the same genesis as Ansible now?

I assumed maybe incorrectly that I could use previous Twitter followings or whatever to launch new things. I could get people to sign up to follow an account, but I couldn't get them to try anything. What I seem to get is everybody says, “I'm busy. I’ve got all these other things going on at work.” They almost assumed by default that they would have to experiment with new software in their free time after they get home. Of course, they've got families, jobs, hobbies and whatever else. Before, it didn't feel that way. It felt like people were able to experiment with stuff and try things out while they were at work. 

Our initial user base was a lot of developers that were in small startups that didn't have much of a dedicated ops team. They were mostly pure cloud and a lot of European consultants who are doing things on their own. As it got larger and more well-known, it would trickle into other kinds of traditional companies with labs, data centers and engineering teams. These days, everybody's getting more and more scheduled and also maybe even burnt out a little bit. It’s not a COVID thing, but this tech is always changing. At one point I was like, “If I did Ansible again, what would it look like?” I was trying to build a next-gen thing and everybody's like, “I’ve already put all the energy into learning this old thing. It's good enough.” Maybe that's the case.

{{divider}}

Did you have a lot of experience working with large open source communities with your experience at Red Hat and with Func and Cobbler or was that something that you picked up? When did you learn that?

It was Red Hat. That was a nice opportunity. Originally how that started was there was a mailing list for some of the early virtualization tools around Xen and KVM. I posted Cobbler there and we shared lists for a while and eventually, it split off. In that community, the mailing list might have been about 400 people when I left. I thought that was big at the time. Ansible was obviously quite a bit bigger, but I learned from trying it what worked and what didn't work. We did not have a lot of systems management projects that were run in an open way at Red Hat at all. We had satellite server was closed and they eventually did open that up.

For the most part, all the open source stuff was in the libraries and the packaging. It was a learning experience because we didn't have any examples of how to do that. How do you run a systems management project and in a way that gets feedback from a lot of people and adapts? There were some other good examples of people doing that with Puppet and Chef and elsewhere. There are a lot of monitoring tools all over the place. It's trial and error. We had a lot of IRC channel conversations back when that was more popular. Maybe it still is, but it was nice being able to hear from people and talk to them and see what works for them and get to know their particular use cases. That was definitely the way that Ansible was doing things in the early days too. It's nice to have that project management hook where you can hear from everyone. If people aren't engaging in the same way as the open source communities, you're flying blind. You're going back to these older models of running it as a traditional company or something.

{{divider}}

At what point then did you feel that, “It’s outgrown me, I need help, I need to earn a living?” How did that work?

I don't think it was that. It was more of an opportunity thing. I felt like there was a chance and it was worth trying. Originally, there were a few people that were like, “I could pay you for support for this.” Seeing that didn't happen, it had a couple of VC bites, but they were fishing or not sure about things. When you had that chance to see if it's a thing, I decided to go see what would happen. We're very lucky that that was successful, but it’s hard to replicate to know what things people are going to latch onto and pay for. We had the idea of releasing the UI as a close source was pretty good because I was about ready to build and release a free thing. I can drive the adoption. If I had released that, then I would have had a little bit of a different problem.

“Our initial user base was a lot of developers that were in small startups that didn't have much of a dedicated ops team. They were mostly pure cloud and a lot of European consultants who are doing things on their own. ”

{{divider}}

One of the things I'm interested in is some types of open source projects have a clear path to commercialization and then others don't. Was it like, “This is such an obvious thing that we can do to build into a business?”

No. I don't know exactly what order what beliefs came in, but it was a condition of like, “If there's going to be a company, we have to sell something. Here's the idea for this thing.” I had the awareness that consulting didn't necessarily scale, but with that complicated stack, a monitoring tool or a hosted application that's relatively simple, doesn't have these problems. When something that's almost a programming language, if you're building it around a consultancy, you have the incentive to keep the project too complicated because you're wanting to keep it academic. I wanted to avoid that. If it's a hosted offering, sometimes the reason that the host offering is hosted is because the project is too hard to install.

You have a reason to not make it easier to install. I wanted to do avoid those things. I think having the interface be the separation in a sold product made it a little more honest or I would say at least it allowed the core language to keep its values. It was also a good separation because not everybody needed to be successful. If you knew enough to run the tool from Jenkins or another CI/CD platform, you didn't need the gooey. It wasn't like we were crippling anything. We were trying to make it hard, but it was there for the people that needed some more enterprise requirements. Mostly it was around role-based access control and logging, which are both very unexciting.

It turned out that that was something that people were willing to pay for. That worked out pretty well. It was useful too because we have a lot of traffic going into the documentation web page for people to learn Ansible. I can have a few very small sentences. You can try this thing out for free, it's free for ten machines forever. We could not be very heavy-handed. People would download it, try it and see if they liked it, which was good. They could run it on their own infrastructure. They didn't have as many security concerns. We don't have much IT staff. If anything, I don't want to be on call, but I also don't want to create something that's going to be a giant botnet and take over all the world's computers. That would be a company ender right there.

{{divider}}

That's a bit of an occupational hazard when you're building an orchestration tool. I never thought about that. It’s the sort of thing that would keep you up at night.

I always thought that was a big liability to Chef hosted, but it never happened to them, which was fantastic. You don't want all your computers taking orders from something, especially when you don't have many people working for you.

{{divider}}

How did you broach that? Because you must have had a pretty large community at that point where you then announced that you were going to build was it a closed source product or was it a licensed store?

It was a closed source. You could download it and it was Python, but we didn't ship the source for it. It was the PYC files or whatever.

{{divider}}

What was the reaction?

I don’t remember all the conversations. I am sure that there was some controversy, but it wasn't too bad because there wasn't anything released yet. A lot of people in the early adopter audience didn't need it. This was appealing to corporations. It didn't hurt the consultants or maybe those people that were consulting could still buy a copy for their customer if they wanted to. It wasn't like we took anything away. I’ve had seen that a lot of times when somebody has a plugin and that plugin stops being part of the free distribution. You can't take stuff away. That's bad. It worked out pretty well but the reason it worked is because of the website had a lot of traffic already and then you're saying, “Go install this thing.” How you get that initial traffic is the question.

{{divider}}

Did you spend a lot of time thinking about that strategy or did it fall into place naturally?

I don't know. I spent a lot of time thinking about the documentation. Someone gave me some good advice when we started. He was like, “If people aren't successful trying this out in about 30 minutes, they're going to move on.” You have to make somebody successful within their lunch hour. There was another good lecture I remember from way back in college. This was Dr. Porter's Engineering 100 class, but he was talking about this other lecture on learning styles and how people learn things differently. Some people like things graphically, some people like things very sequentially, some people learn very random access. How do you present your documentation in a way that works for everyone? Sometimes with tech tools and it still happens now, they make stuff sound complicated or they have a marketing guy write it and it's vague.

It doesn't tell you what it does. It doesn't show practical examples. Maybe it's not organized and maybe it's not cross-linked. Being able to have all the information for somebody to be able to figure it out was important. My earlier estimations and I'm not sure if that's accurate, but it is, was I spent maybe 40% of my time total on documentation of the front of the webpage. Also telling people how it works and then making sure that those docs also said why you're doing something or what's the philosophy and making it a little more conversational rather than a reference manual. That was good because this was back when people blogged.

This is another problem. If social media engagement is low, where do people find out about new things? People would read blogs, articles, tutorials or write their own version of the tutorial, which was often 80% the same, but maybe it was a few opinions thrown in. Occasionally, those would get on Hacker News, Reddit and things like that. Now that's also a little bit of a problem because your amount of time at the top of Hacker News or Reddit is hours at best. The tendency is so much of someone to post and will respond about like, “You're not that smart,” or whatever. It's a negative audience even more so than it used to be.

That earlier strategy of having a lot of content that people could link to and share was good. I gave one presentation at a local DevOps or Linux meet up with some kind in Durham City just around the corner. I didn't do any travel for the first year, but somebody presented it at a library in Indiana, Illinois or something like that. People would randomly start presenting it to their local user group and people would tell their friends. That was made possible by the documentation helping people learn it. There wasn't a need to market it and push it. It could spread because people thought it was something that they were interested in. 

{{divider}}

Did you spend a lot of time getting feedback on that work? How did you iterate that process?

I always to write a bit, but I did hear everybody's feedback in the sense that I read every single tweet. People say don't read the comments and I read all of them. I don't remember how I changed too much of it. There was a good patch from JP Benz early on to auto-generate the documentation for the module index. That was out of the module source. That was absolutely huge. Occasionally, people would be like, “I need this to be better,” but mostly I was trying to anticipate their questions and trying to forget what you knew about the tool enough to write enough about it for that person and realizing different people are coming in with different needs and questions.

{{divider}}

Was it an incremental thing or did you ever start from scratch?

Do you mean to restart it from scratch?

{{divider}}

No, I mean the documentation. It grows slowly over time and then all of a sudden you worry that to a new set of eyes, it might be too much.

It was a constant process of always one working on it and always tweaking the webpages and stuff like that. It had to happen the entire time. The original editions, there was something pretty early and continually revising it and trying to reread it and seeing, is this still accurate? Does it still apply? The documentation was marketing. There are a lot of webpages now and they're easy to parody, but they have the three adjectives on the bottom and the three pieces of clip art. There's the about us page. There's a picture of two people shaking hands. You can't tell at all what they do. I was always trying to say exactly what things did and why you should be interested and here's how you try this out. 

{{divider}}

How did you go about managing such a large project? At what point did you start delegating and approving merge requests and things like that?

It’s been years since I’ve worked on the project. We did hire some employees and stuff and they did a great job. It was very helpful. The project wasn't a company for about the first year. Even into that, I had to do all the merging and all the reviews. It does get to be a lot and I don't remember what at all my policy was after that as we had more people, if I was still deciding what to approve. I kept creative control over the whole thing because I wanted to make sure that it was consistent because there's a danger with modules in particular of cargo culting.

If you do something and they're not consistent, then somebody's going to copy a different module to start the new one. We had some great help and a lot of people built some nice features and the community was good with a lot of contributions. If you go back and look at the stats, you do also see how overwhelming it was to manage all that. It's a lot of where the quality isn't what I wanted. It was because it took off so fast and it was very modular. When I left it, there were 400 or so modules. Realistically there should have been less. We could've kept it up. There was one particular instance where there are some cloud provisioning models added.

They were good, but it was a distraction and I wasn't ready for it at the time, but I realized that if I didn't absorb it into the project, we would lose it. That was a good vehicle for people to try it. It would've been better if we could have built something specifically for cloud and had time to design it. People started adding more clouds and there's some great contribution work from lots of places and good quality stuff. It was moving in too many directions to keep focused on one particular thing. Probably that's my advice for most projects. Less features is probably better.

{{divider}}

Did you find it quite hard to reject new work?

It's hard. You don't ever want to reject somebody's work because they'll be unhappy about it and they spend time doing it. Also, GitHub had created a problem in the industry. Before GitHub, people would usually talk about something on a mailing list. “I got this idea, how can I do this?” You'd be like, “You can help,” or this will be in maybe IRC and then they'd send you some code and then you talk about it. GitHub has this thing where you can send code to people. A lot of times that's great, especially if it's a bug fix. It's like getting a free puppy in some extent because you have to take care of it because you don't know that they're going to come back.

A lot of people do, a lot of people won't and then you also have to figure out, does it break any my other code? Can I maintain this and do I understand it? I lost one of our absolute best contributors over not being able to understand some of this code. I regret that because he was awesome. On the other hand, I couldn't maintain it and that was a problem for me. That happens. The need to do code review is, at the time GitHub's code review tools were not very good. I don't know if it's gotten much better. You have to do that and you have to do testing. There are a lot of projects that will instantly merge stuff from the web and be like, “Thank you.”

They look at contribution like a status thing. I’ve got so many contributors. We sorted that too and that wasn't good. I would say don't optimize for that. The modular thing is good, but it needs to be about code review and keeping that quality off and making sure there's a good structure for all those new modules. Communication's hard too because you want everybody on the same page. It's hard for people to contribute around the open source development or something evolving a database, a web app or anything that involves that. I need to pay attention to this project, but something that has a lot of plugins. It's pretty easy for somebody to not be paying close attention and still send you a plugin. That's good, but it's a challenge. It’s like a Wikipedia problem. Is this plugin notable? Does this work for a service that I care about? Some of them are things you can't test. Do I trust this person enough? This is a plugin to send a text message or something like that. The code looks okay, but what's the right level of testing for something that you don't have a need for?

{{divider}}

I never thought about that. That sounds like an absolute minefield. It was like if you had to design the most difficult and how to amplify those problems. Having modular components, you probably couldn't make it more difficult, could you?

It's a programming language to an extent, but I wanted to have everything in the tree so everybody could come together and maintain things. What is the right set of modules that people need versus things that shouldn't be in there? It was tricky. I’ve said it many times. My code for all my other projects is better than that one as a whole because people do great things. When you have thousands upon thousands of people coming together, they all don't have exactly the same style. It's noticeable. You get subtle inconsistencies and pieces of the tool, but it was still a good experience to see what everybody was doing and trying to do. When people find new creative uses for things, it was always fun.

{{divider}}

Are you still involved in it? Do you still commit to it?

No. I haven't been doing that for years now. I left and I had a three-year non-compete where I couldn't do anything in the systems management field for a while. I’ve tried to create a few new things since that didn't necessarily stick, but they were interesting attempts. That's where I came to this new model of what does open source contribution look like in 2020 or even two years ago. Do I still believe in it? That's changed. People do probably want free stuff to try it out. They might need to be able to write a plugin. They might want to be able to read the code, but there's less emphasis on contribution.

Hosted business models probably work the best of anything now and that's fine. It’s a little trickier to bootstrap something when it's small because people expect something to be complete. Complete is easy if it does a few small things, but if it does a lot, complete is harder. I had this crazy idea of there should be an alternative to Kubernetes and that's viable. That's a very large project to get all the features that everyone would expect to have. You can't bootstrap that one slowly because people want it to be complete but smaller tools and utilities still have a chance. There's still a lot of people trying out things in web development because it's easy to try out a new library. If something requires setting up a database, changing my infrastructure, potentially breaking production, learning how to use this new thing or even if it takes an hour, the less people are going to try it.

{{divider}}

Do you think that problem is getting worse? 

I don't know if it's getting worse now, but I felt that over the last couple of years, for sure. Maybe if the whole COVID thing is causing a little bit more burnout on people, then it could be getting worse. It doesn't feel there's as much excitement about people sharing tools that they found that I used to think that they had because there were all the blogs. I used to read Fedora Planet and stuff and that probably still exists, and then people being excited about new things. Now all I see are people are more like the Hacker News and Reddit, it's more cynical about, “Here's why I shouldn't have to try your thing,” or “Why didn't you use this?” I’ve posted that several times and I’ve gotten a, “Why didn't you use this?”

If I had listened to them, all these other things would have never been filled. It's hard to not be cynical about that. My interest in open source has changed because people are willing to say that they're interested in trying something, but they don't necessarily have time to follow through anymore. They don't have necessarily have time to contribute. It's not their fault. It's a fault of the way that we're running software engineering practices at most companies.

{{divider}}

I hadn't considered that because obviously, GitHub has centralized that to a huge degree. Also, I hadn't considered that the actual mechanism of communicating with people, which you're right. That's when Google Reader was big and you'd follow 50 tech blogs and that would be like how you learn about new projects and things, but that was obviously completely decentralized. Now you've got a mixture of Twitter, Reddit, Hacker News and that's basically it.

Your Google Reader thing was perfect. That was the high point. That and I was using IRC cloud and whatever. Everybody was producing their own stuff. There were lots more sharing, but then now there's this consolidation theme. Everybody is looking at what Facebook is doing. The people that get the most interactions and the most likes are not necessarily people that have built tools. It's the developer advocates that sell them. People are looking at the Google, the Facebooks and VMware advocate people for advice and not building those tools themselves. Instead of having these small local clusters where people are sharing information among themselves, they're all consuming the same information. Where it was decentralized, maybe it allowed more variety to come up with more competing completely different ideas. Whereas now we've settled into what might be a local maximum of some kind.

{{divider}}

Maybe it wasn't Google Reader that was canceled, but RSS as a syndication method.

Everybody gave up on it at that point. If something stayed on Twitter, it stayed up there for seconds. That's too much of a fire hose and doesn't have topics and it's not categorizable as easily. I noticed that too in that less people are engaging or commenting on those ideas. The whole dopamine reward cycle is completely different on social media over time. What that means for open source, I don't know. There does seem to be interested in new things and new UI tools. If more companies tried to create, GitHub is in the ideal position to do this. Not just a popularity rank system without start, but some dashboard where they're like, “This is the new stuff that our editors found.”

Talk about it and then highlight some new interesting projects in lots of different categories and writing more content. Tech journalism is so bad with all those writers that basically are regurgitating press releases. If there was more content about, “These things are cool and this is what I found this week dashboards for new projects and things that,” maybe that could get that spark going again. There has to be this move back towards people writing more. The little 140 characters, you can't talk about tech and that space.

Back when people blogged about stuff and shared about it. I always wanted to see more engineering blogs out of companies too but that never happened. High Scalability was the website that used to describe some of the architectures like Facebook and Netflix. I asked them to do a guest blog when I dropped Ansible for the first time. That was pretty much one of the original things that got the spike going on Ansible. I wish more engineering companies will do regular tech blogs. What they were doing is they do it with marketing prior to them, but they weren't keeping it up. If people did that more as a part of their job, maybe that's a way to find time to talk about new tech, new ideas and share architecture between people versus consuming it from the vendor or living in a very vendor-forward first world where the vendors tell us what we should consume and then we consume them. All the good ideas have always been with the end-users and the customers.

With Ansible, they always knew more than I did. I didn't run an infrastructure of any size. They had all the experience. How can you learn from them? Those are the people that you want to hear from. What's happened in the IT world is instead of hearing those people talk about their architectures and their engineering challenges and what's useful, we've got this DevOps convention thing instead where people are talking about the bona fides of a particular religion and their processes. They were like, “Break down the silos.” Everybody goes up, “Break down the silos and then release more often.” There are no specifics. It's become those developer advocates that go from show to show and keeps saying the same things. We don't learn. We have to reclaim that somehow and make engineering more about actual engineering once again versus the labor assembly line.

{{divider}}

I feel like there's a constant yearning to emulate the big successful companies, but the ones that are operating at a global internet scale. It's like, Airbnb did 200 builds to production a day, but we don't and I don't feel bad for it. Sometimes you're made to feel bad because you're not behaving like Airbnb, but we've got a thousandth the size of their team or what.

If you think about it too, yes, they have some scale, but did they need the complexity? That could probably be done with a three-tier architecture and a lot less nodes and a lot bigger metal. They probably didn't need to be 400 microservices and whatever. They've made that complicated too. Everybody's trying to make their infrastructure, Kubernetes is the big one. You're on a cloud and you're running another cloud on top of your cloud. You didn't have to do that because they were already running your cloud for you. Now you've got the management headaches plus the cloud bill. I'm more of a developer. The small number of moving parts and be simple so I can concentrate on the app, what's specific to this business. Your point about chasing scale is how we're getting there. Maybe that's legitimate, maybe everybody wants to learn.

They want to see how somebody is doing these other things. A lot of times you get complexity because people want to learn. I started out, I was doing a lot of Java and a little bit of C, but I got into Perl for a while because I enjoyed it because it was complicated. People sometimes enjoy Java architecture because it was complicated and not happens in your board. Puppet was that way too. People were writing bash scripts and they're like, “This is a language. It's got this operator that looks a spaceship.” They enjoyed the hieroglyphics. It's not wrong, but maybe that's an example of where we're at now. Everyone's made the infrastructure complicated because they got bored. 

{{divider}}

Also, 98% of people build work on relatively simple CRUD applications. There's nothing wrong with that. That so happens to be the thing that solves most problems. When you have an issue where you suddenly have to write an endpoint that can do 10,000 requests a second, that's an unusual situation and engineers, unless you're happened to live in Silicon Valley, people outside of that sphere don't have to deal with that problem that often. When it comes up, they're like golden eggs that everyone wants to work on.

It feels like there's a lot of knowledge because everything is going cloud, whatever, nothing wrong with that. There's a lot of knowledge that gets locked up in the cloud provider. People don't know how these things work anymore because there are so few people that know how to work on them and they're all out, in Silicon Valley and Seattle or whatever. That's a weird complexity. It feels like instead of where before I was thinking about even the projects in my college class I helped with. Now they're basically assembling things out of all these libraries and they don't necessarily know as much about how the internals work. Before, we didn't have any libraries. We had to build everything. Now it's moving to the fact of what I'm learning is the APIs that Amazon made for me. I'm not doing much but plugging together these things. The whole nature of the craft is very different. The number of people that can write a high-performance web server and see, which was never me anyway, is extremely low and extremely specialized. The vast majority of it is it's CRUD and the existing services and glue. That's the nature of the beast.

{{divider}}

Have you got any projects but not necessarily ones that you've worked on that you've excited you and you've found enjoyable and liked working with?

From a software perspective, I'm still doing a lot of Django here and there. Nothing super, highly exciting, but it's been rock solid and stable. A lot of Pythons still. I’ve got a music sequencer project I’ve been working on in my spare time for providing some alternatives to Ableton Live and stuff. That's been cool and that's all been trying to think of the projects I’ve used to make that possible. It's all relatively simple things. I'm going back to more basic programming versus a bunch of libraries and architectural things. What apps that I can build that are interesting, but that the music sequencer project is called WARP. It’s at WARPSeq.com. I'm going to be trying to release that. UI is probably going to evolve a little bit.

{{divider}}

Is that a pure software sequencer?

It's software, but it can talk to heart like a USB MIDI interface or something if you want to talk to hardware. It’s been fun trying to get Python timing to work like I want it to. It gives me an excuse to relearn some web development stuff that has been very rusty after having teams of great people that were doing that for me for a while.

{{divider}}

Have you got anything else you want to shout or any URLs you want to give out?

No, I'm good. Thank you very much. I apologize for all the complaints on the state of open source.

{{divider}}

I don't think people talk about it enough, to be honest. People talk about burnout and stress on maintainers and stuff, but the periphery around why that is and how we got there, I’ve never thought about it.

We need a large player like a Microsoft or GitHub to make some good dashboard and some dynamic content. What is the website that everybody reads for programming news? Replace Hacker News, replace Reddit, have some good articles and coming in from a lot of places. There's an opportunity to get people exposed to new things and start collaborating with each other between companies more. We also have to look at our project management, give your engineering teams more time to research, try things on their own and stop over-scheduling them. Basically, run software like we did many years ago.

{{divider}}

Thanks for your time again, Michael. That's interesting.

Thank you.