Shepherd Pro

Buckle up for a journey into Shepherd! In this episode, Chuck Carpenter, the founder of Shepherd Pro and host of Whiskey Web and Whatnot, joins Ben Rometsch to delve into Shepherd and Shepherd JS, an influential open-source library for creating interactive guides in JavaScript. Chuck shares his deep involvement over seven years as a maintainer, highlighting Shepherd’s evolution from its origins at HubSpot to becoming a robust tool with broad community adoption. Don’t miss Chuck’s insights on Shepherd’s edge in the user experience space, its business potential, and his vision for leading the developer community forward.

---

Introduction

I'm pleased to welcome Charles Carpenter to the show. Charles and I have known each other over the last several months, which will all become clear as we go. Charles, welcome.

Thanks for having me. Ben calls me Charles because only my friends call me Chuck.

I thought this introduction needed to be a little bit formal.

Yeah, sure. Whatever you need from me.

We are both big football fans, and we've had some tumultuous goings at our respective clubs over the last few weeks. For those who have no idea what I'm talking about, I support Ipswich Town, which has been in the real doldrums over the last several years. They won promotion back into the Premier League in the UK, which is easier and the most competitive league in the world. Manchester United managed to win some silverware.

That's the funny and controversial part because there's been a lot of stick thrown at United throughout the season. Justifiably so in many ways. At the end of the day, for most fans, they want to see some silverware. Second place isn't going to do much. That means you didn't win anything. That's the irony across it. Manchester United came in eighth place in the league. It looked like we were going to miss European competition against all odds. It was 9 to 1 odds that United would win a cup final.

I don't bet that often, but I did see that odd, and I was like, “That is insane.”

I don't bet at all because it would have lost had I bet. I avoid any involvement there. Against all odds, they came back and beat Manchester City, 2 to 1. It was an exciting game. The big drama is if the coach is going to stay. I certainly hope so. It's been a challenging season. Conversely, on the other side, speaking of Ipswich town, because the coach there formerly worked at Manchester United. There were talks like, “Are we going to swoop in and steal him back?” It would be a terrible move. He should stay committed to the project that has gotten him here. You were remiss not to mention the fact that it was two consecutive promotions.

For people who aren't aware, they were in the third division, which is called the first division because of money, stuff, and nonsense, mainly money. They were in an illogical race starting with one. They were in the third division, and they got promoted in two consecutive seasons, which hasn't happened for quite some time on account of the competitiveness of the sport in the UK.

There are four professional leagues. There have been a couple of managers that have done three consecutive back-to-back promotions. One of them was the English Moneyball story, in which data started coming into the sport more, quite a while ago. Some analysts realized that if you get the keeper or the defenders to hoof the ball as far up the field as they can at every opportunity and get a massive striker who can run fast and keep hammering the ball to him. There was a team that did that and ruined the game for several years because the spectacle was shocking.

I imagine it's not that exciting to watch, but promotion is as exciting as a fan. I wonder how challenging that would be in the modern era because of the amount of money involved now. It’s not to say there wasn't always a lot of money, but the size of budgets and payments with nation-states and crazy owner groups and everything else, everything is tenfold over. Every time you move up a league, the amount of money involved in establishing squad salaries and everything else is such a massive jump. To come in and try to compete on the next level, and being able to do it again is an incredible feat in this day and age.

My favorite player on the Ipswich side is Wes Burns, who is a Welsh international winger. He scored the goal of the season. If you search Twitter or Google for West Burns, it switches the goal of the season. He’s an absolute real humdinger. He was brought to the club for 60,000 pounds sterling. There are players earning five times that per week in the Premier League. That's how much we pay for him. It's wild.

I'm pleased that our manager is sticking around. I've become jaded with the sport, particularly around the money and investments that have been made by various groups who aren't doing it for the love of the sport. He seems to be a beacon of hope. We'll score nine points in 2025 and get relegated. He'll pull the parachute, and we'll be back where we started, but at least it was fun.

Was it a reference to the parachute payments coming back?

Project Genesis And Evolution

Let's start talking about open-source stuff. What the hell do you work on?

I work on the web. That's a large ambiguous net, but I have been for over twenty years now doing various web software projects for the man, for myself, and so on and so forth, but the one we're here to talk most about is Shepherd and Shepherd JS, which is an open-source guide creation library written in JavaScript. It has a few framework-specific wrappers that have a little bit of sugar around them. For the most part, we're talking about Shepherd itself specifically.

Shepherd has been around for nine years. I've been one of the main maintainers of the project. It's a long-running battle-tested, rewritten a number of times, completely accessible, keyboard accessible, and a lot of accessibility features built into it. It’s a teamable and open API that makes it state-agnostic. It's dumb. You can do whatever you want with this series of dialogues that can interact and react to user inputs.

What was the genesis of the project? Did you have involvement and commit one?

I did not. The project was kicked off originally nine years ago. It was a HubSpot project at the time that we inherited after a year or so. They had two particular open-source projects. One was Tether, and one was Shepherd. Tether was about attaching elements to another element. There are floating UIs in the most recent one.

There have been a few different libraries that do this thing. That's become less necessary over time as the web itself, and those NAT built-in APIs get better, and we've been able to reduce the size of the library based on the web itself getting better. We do not commit to zero early on. I used to write a bunch of Ember JS. I’m looking for something like this for an Ember project and creating an Ember wrapper around that. Ember wrappers are helpers for installing into those specific kinds of applications. That was the gist of it. It was about attaching this popup or dialogue next to another element. Over time, we've done a lot more things in terms of its highlight API and things of that nature. That was the impetus and the gist of it.

I'm trying to rack my brains as to how many of these podcasts have people who won't commit one or commit zero.

It's often that a story of open source success is around, “I was scratching my own itch. I made this thing, and it's blossomed from there.” That wasn't necessarily the case. I found the library by scratching my own itch. It’s the evolution from there. It has been more involvement of Robbie the Wagner and me. He's the other major maintainer, and he is getting the project given to us because they didn't want to maintain it anymore.

Let's talk a little bit about that. They were using it themselves within a HubSpot and open-sourced. That's around the time Dan Lyons’ book came about. Have you read the HubSpot book?

I have it, no.

It's a great read because it's an interesting biography about how he took this job. He was a journalist, and he took this job as this crazy startup. He spends a lot of the book trashing how ridiculous the business is. Let me look up what the share price is now because he has a market cap of $30 billion. They were using it themselves and open-sourced it because that was what people of that nature started doing around that time.

GitHub is becoming more pervasive. Open source means having contributors from the community share in that and having more of a culture. It's almost to a degree, especially for a big company, that having an open source is almost like a marketing channel. I've used HubSpot on and off a number of times throughout my career. They have tours. They have this all in their application, but they write it all internally.

Open source is almost like a marketing channel.

Culturally, there was a shift in how they were going to do software, and they pulled out from open source projects and let the heavy maintainers of those projects take them over. That's how it ended up going down for us. I don't have the specifics. I know they weren't going to be doing HubSpot-associated open-source projects at that time. It was a natural. Do you want this? Sure, yes. We can do whatever we want without going through a formal process because it was in the early days.

It'd be good to try and find someone in a large organization to talk about that because I wonder whether that feels like quite a common story where those organizations don't realize or don't have sight of what the maintenance burden of a project like that could be. All of a sudden, they've got 2 or 3 full-time engineers who are doing that maintenance job. That must be a common story.

I experienced a version of that at National Geographic because I recall we were creating a component library in the early days of React, and we were putting a lot of one-off assets across a bunch of properties for the society and the media projects. We started working on this open-source project because we thought it would be cool.

It gives our engineering group some visibility, and people don't think about the photography, but there's neat tech going on in various parts of society. We went down that path for less than a year. It's an old organization. They would start freaking out about any potential legal implications there and what exposure. We don't know. We don't understand it. We have to shut it down. It's a no because we don't get it.

I've spoken with FINOS, a financial services member-owned organization that tries to help financial services organizations contribute to open-source. The vast majority of those larger organizations, like every line of code, go through a non-engineering review, which is like, “Is this going to get us into legal hot water? Is the regulator going to get upset?” It is nuts, but different organizations have different levels of that.

I'm curious to know. Can you set the scene for that passage of time where it started with them going, “We don't want to be the key holders of this repository,” and ending with you having those keys? We see this with projects where there can often be a bit of a bun fight amongst the maintainers because there's potentially a valuable asset or some asset there. How did you traverse those potential hazards?

I could see where that could be a case, where there's value to the efforts that they put into it. Even in this spot, it was part of their full-time jobs. They still were responsible for the inception and things of that nature. It was fairly drama-free. It was more of a secondary artifact of them having to pull back from being more involved and being gatekeepers and limiting PRs and inputs not purposely, but not having the same bandwidth that they originally had. Robbie is proposing to them that we take it over, which makes more sense. Let's make sure it's okay. They asked about some internal things. We never signed anything. They transferred the repost over, and it all progressed from there.

What's funny is that in the case of the Ember JS wrapper, we already had that. These wrapper frameworks and reposts were already with us. That was an agency at the time called ChipShape, which does exist but not quite at full throttle as it was earlier on. We had some projects where we worked with Shepherd JS and in doing custom implementations and were using that with an Ember Wrapper. For them, you're already working, using, and contributing a lot. We can't quite get the same energy there. Everybody says it's fine. This makes sense to us.

How mature was the community around the projects at that point?

I would say less than half of where it is now. We're at 13,000 stars. A lot of our trajectory came early on from growth associated with Ember JS, having a natural package that worked in those applications when it was at the height of greater popularity. That worked a lot in our favor. Conversely, once the Reacts-specific version came out, it was fairly popular. This Shepherd JS is so popular, and it scaled quite a bit from there. The community as a whole and contributions at that time were less than half of where they are now.

When you collected the keys, how clear was the future in your mind of where you wanted things to go?

There were a couple of different larger initiatives that we wanted to do right out of the gate at that point. I'm trying to think where we released 5.0. A lot of those changes were related more to ECMAScript module support versus common JS support. We’re trying to upgrade things in the build toolchain and change some import paths. These are not major features, but we went through rewrites to reduce bundle size quite a bit.

We changed from Tether to Popper at one point. We started adding accessibility in trying to reduce the size of the components themselves. We ended up eventually integrating Spelt because felt outputs HTML, which reduced bundle sizes and dependencies. We had goals of making it smaller and embracing web APIs as much as we could, and as with those releases, we incrementally added things. Accessibility was another artifact of where the Ember community was, which is they were much into having PWAs with offline first ideology and thinking about like, “How do we make some open APIs to make that easier?” To a degree, those things were scratching our own itch.

At the time that you took it on, the agency was still going full throttle. Did you consider this to be a business in a project in its own right? Was it more you like hacking on it and happy to pick up the reins?

It’s more the latter. I don't think I had 0% thought around trying to build some business around it. For quite some time, I was happy to be an open-source contributor, trying to do a thing well as part of that and utilizing that in a lot of our projects moving forward. That was also a win, having it as a standalone. I'm good at talking myself out of any good and bad ideas. It's like, “Yeah, but I could do this, but then there's ABC already in the market. How is this different?” I can build things, but it's hard for me to have a lot of conviction in an idea for a startup.

In terms of the project and the entity behind it now, where is that?

There's a twofold part to that. There are various times when I've thought about things like, “We could try and do a little something with this.” I’m not sure we're builders, getting involved with folks who are not builders but have a lot of conviction about going to market and looking at things in the marketplace. Having a lot of conviction around open source is great.

I tend to try and choose and pick products that have an open-source component to them. Most of the time, I don't host it myself, but I love seeing that that's an option. Maybe that's me as a builder. Those are the businesses I like to support rather than some unicorn VC-backed monolith, “I'm in here, and I can never escape.” That's their point because they want to get as many users as possible. Getting involved with folks who have those same ideologies gave me a lot more conviction about being a player in the space.

We haven't mentioned it, but there is now a pro component to Shepherd. That is building an application around it that interacts with the Shepherd Library and captures all of the user events. Over time, it will let you do more customized targeting and a number of integrations there to understand that these user journeys and these guides are effective. If they're not, why and what's going on?

Business Potential And Strategy

In terms of the maturation of that from a repository to a business, can you talk a little bit about when you start to think, “There's an opportunity for it to become a self-sustaining entity?”

I would say a year ago. There have been various times when, internally, we had some considerations or had been engaged by outside entities taking a look at the project and taking a look at having conversations with us. Would you be interested in building something around this? I'm like, “I'd be interested in hearing what you have to say.” That's a good first start whether I think that is something worth my efforts or not.

I was engaging with a portfolio business around trying to do that. The difference is, what are the other components? What are we talking about building a business? Is it that you hand me some money, sit off in a room, type away, and try to come up with things, or is there support? What does this look like in your view and my view? Do we align on that? I started having those more serious conversations last summer. I started putting that into effect at the end of last year and started building not long after.

In terms of the competitive landscape, are there direct competitors with what you're doing as opposed to a part of an analytics platform or part of a user experience platform? What does the competitive landscape look like?

There are a number of them out there. More often than not, it's part of a holistic user experience platform. Competitors that exist out there are getting into your application and letting you build onboarding experiences or funnel-driving experiences and do it with a multitude of components. They have more robust component-level offerings than we do.

Most things seem to be focused on providing a WYSIWYG for non-engineers to build these with. In turn, they let you use that to drive user-based experiences. Sometimes, there's an analytics component in there and integration associated with it. Most times, there's no open source component to it and no self-hosting option to that either. I do believe that those two elements are differentiators in our case. We have some catching up to do feature-wise, but having that conviction first is a strong one that gives us an option.

‍

In terms of the ideas around how you could build a sustainable revenue stream, it's one of those interesting projects that have multiple avenues in that regard. With Flagsmith, we didn't put any thought behind it at all. We did the stimulus-response answer, which was paid API. It wasn't a wrong answer, but it wasn't the absolute standout right answer. Did you spend more than 200 milliseconds thinking about that as a strategy or not?

Probably only 300 milliseconds. A lot of ideas, all things considered. We're in an alpha phase. We're taking signups for the SaaS-level product. We are open to having conversations and have had some conversations with companies looking to do something more specific, like a white glove setup and/or having the service on-prem because they prefer that, or they have a closed system, or there are some regulatory reasons around that.

To that end, I'm waiting to get feedback from the market about that. A common charging principle in this is about monthly active users. It’s API access. It's doing a lot of monitoring in that. Its usage is based on SaaS. I'm not convinced that's the direction that we want to go, but I think it's going to take a little more input into who our customers end up being and where they find the value. I'm leaning more towards less usage-based, flat fee-based. I don't have clear answers on this, but going more of the enterprise license route of things.

The direction that we want to take will depend on our customers and where they perceive value.

Self-hosting these things has been one of the interesting things about Flagsmith. Several years ago, especially in enterprise deployments, there were 4 or 5 different paths in terms of the orchestration framework. Now, we never talk about anything other than Kubernetes. Can you describe what the project, or if any, is closed source as opposed to open source and what you are thinking about that?

As of now, nothing is gate-kept in that realm as I continue down the featured development path. A lot of the value there is in my expertise with the library and application. Everything else is completely open. Overall, it's a simple container for the application itself. None of that is quite gate-kept at this point. Because it is still fairly early days, and we are out in the alpha level of features, the more advanced features that get requested will start to become more of the enterprise license and different sign-on avenues and things where security has to get scaled up. What do I need to add? That's something that is of higher value. For now, it's all open.

What are the metrics of the projects in the business that you are looking at and considering as they stand? Is it still open source adoption and contribute account and things like that? Are there other things that you're trying to work on?

We're in a transition around that. Having up to doing the alpha release was a lot more around open source contribution, post-alpha release, and having users sign up, start to bang on the product, and look at doing a number of user interviews. I'm trying to get more conviction around the roadmap because a couple of things that I'm planning to build through in 2024 are about advanced targeting and cohort syncs from other analytics platforms so that the work is simplified and targeting some level of configuration through the application.

All configurations are done in the Shepherd JS way, which is JSON objects and code. We want to dynamic size some of that without making a complete WYSIWYG experience. What integrations are the most important to folks? We have a couple of them on our roadmap. Right now, more of it is about getting as many users of the product as possible and feedback to help drive my direction.

In terms of the project as it is on GitHub, it's quite a beast. You got a large number of contributors and star activity. How did you go about managing all of that?

I’m compartmentalizing those times. I'm still heavily involved in the repo itself. I'm there quite a bit to engage in issues, feedback, and PRs. It is a beast, and it can be quite time-consuming at various bits. The plus side is that it is very mature. Emergencies don't come along quite that often. We know what we're going to get for the most part. We have a ton of testing and a ton of guardrails there protecting us through contributions. It's not too bad in that sense. It's about compartmentalizing time to give the community help and feedback.

In terms of the shape of the contributions, one of the common themes from talking to people on this show is it's interesting. It'll be interesting to hear if this is different for Shepherd, but 90% of the code that's written for Flagsmith is written by people who are paid to write that code. How does that look for you guys?

That's pretty consistent, 80% to 90% of the contributions come from Robbie and me, and the rest from the community. A lot of times, those contributions will be an edge case bug, something that someone found, or a small feature they want to suggest, small UI changes. Every once in a year, we'll get a nice, healthy contribution that changes some major functionality and simplifies something. It’s something like, “This is amazing. We appreciate it.”

PRs are free. We welcome them every time you open an issue. Consider submitting a pool request. That can expedite your feature getting in. That's sometimes a misnomer with different folks. Maybe they don't want to, or they don't have time to empathize with that. If you want to expedite a need you have in a library, I can't say enough if you bring an open PR to it. Even if we have to work together to take that to fruition, that helps prioritize and expedite things.

One of the things I'm jealous of with Flagsmith is when someone hits an issue. If they're starting from zero, getting to the point that they can potentially reproduce that issue locally is. It's not a huge amount of work, but it's not a five-minute task. However, I would expect that for quite a few of the issues that come up with Shepherd, people could get up and running in terms of trying to reproduce it and digging into the browser console quite easily.

Even more so with things like CodeSandbox and things like that. CodeSandbox is great for spinning up this React app. You can, through the gooey, add dependencies and fairly easily do a replication of your issue rather than having to fork at GitHub, make things happen there, and someone else has to spin it up somewhere else. That is the great thing about this being a JS library. You can spin up a simple note app to replicate your issues. That's what we say. If issues come up, I'll take a look. Can I easily replicate it based on their feedback? If I can't, I'll ask them to provide a CodeSandbox with this in place.

There's an opportunity there, but there are a lot of people trying to solve that problem. Some of the class of issues we get, especially now, as products and code base is mature. We did hit one where most of the development team was like, “That's not a bug,” and it was. One of the things that I've learned working as an engineer is that anyone certain of something should be kept to arm’s length because engineers should never be certain of anything.

I do wish that it was easier for us to turn around and spend 25 minutes getting the thing set up. Python doesn't help either because of the total mess of packaging managers. In terms of framework support, there's a corollary problem of the constant churn and fashion of JavaScript frameworks. How do you shield yourself from getting framework-specific stuff into your code base?

There's a massive caveat to that. It's only useful if you're using it. We've run into that because we have a React wrapper, Ember wrapper, and View and Angular. Neither of us has touched Angular in a while. To viably support that, it's a challenge. What are the benefits back? What framework-specific workflow or something can this help with?

It's only useful if you're using it.

For the React one, most people are going to do a provider context pattern and/or ask for a hook. I'll create those couple of things for you. You could do that in your own project every single time, but I can see where that gets old. This is a little time saved. Go down that path. For some, trying to test across every JavaScript framework that exists is a zero-sum game.

That's where we're asking, “If you're using this, you know more about this. These are where we need some contributions.” Otherwise, I suggest the most stable version of Shepherd is Shepherd JS. It's JavaScript across the board. You can use it in Vanilla JavaScript. You can use it for everything from HTML to whatever your favorite framework is. It should work in React native for the most part and everything in between. I would always suggest that path because how many of those flavors can you pursue? It's all JavaScript. We're not saying to use this in your Python project. That's the most direct.

You've steered away from having a provider or plugin design? Is there React-specific code in the core project as it stands?

Not in the core project. Only in the React wrapper. That's the only place where we can try to provide some helpers along the way. On the Ember side, these Ember packages have an install path or a blueprint that automatically allows you to put whatever things you might need in your application. The React one gives you a hook. Offhand, I'm not sure what the Angular one does for you, but they're trying to provide some slight shortcuts but no specific functionality. It doesn't exist in the core library.

How do you go about managing those wrappers? Are they all under the umbrella of the Shepherd organization? Are there community-contributed ones? How does that work?

Some of both, mostly under the Shepherd umbrella because there's no one in the community that's stepped up and said, “I want to maintain and keep this view wrapper going for the time being.” That's something we'd be open to. The core library is the important part, and the sugar is on top. If someone can be committed to that and they're saying, “I work in view all the time.” If somebody wanted to create a new one that was solid JS or Spelt specifics, I would welcome that. I have no problem with that.

‍

Future Plans And Innovation

In terms of plans for the future, where do you see things in the next twelve months?

Most of the focus is going to be functionality within the primary application and evolving this connection from the client and the user back to the main application, dispersing that data out as much as our users need and giving them a better interface to get smarter with their guides that's, I would say the majority of the effort is going to go into the application itself and continuing to push that forward.

There were lots of discussions about various things. If I have a Python project and I somehow want to add this, is there another path of SDKs we need to consider? It's worth considering. Some of that is maybe on the roadmap right now, but primarily, all the conviction is about moving the main application forward, making it make sense to self-host this if you'd like, and getting feedback on the community.

Has anyone thought of or started doing any work around supporting a completely different language and framework like Flutter or Swift UI?

Since it's me, no one has followed those paths, but those doors aren't closed. There's a real possibility, especially in the mobile world, where it could make a lot of sense to have some path there. It would be the squeaky wheel thing. If the community spoke up and said, “This path is something we want and need,” we try to plan out a way in that direction. I'm open to it. No one is working on it yet, but I can see that within the next year. It will make sense at some point. It's about what the strongest path is, what the clearest feedback is, and where there's either a gap in the marketplace or some obvious massive need that comes to fruition and gets proven.

I've always been surprised talking to some of our customers. Flutter is a popular framework, especially for internal applications and tooling. It's frustrating because it's a completely different paradigm, language, and framework. It's quite a piece of debt that you're taking on doing that. It's at that point where it gets mentioned often that it's taken for a feature-flagging platform. It's table-stakes stuff. They do things like introduce an entire desktop platform class, which brings in these new primitives. It's tough, but it's a case of finding someone on the team who wants to learn dart and figure this out.

There are twofold there. Didn't Google let go of a bunch of the Flutter team? They've reduced that even though they stay committed to the product. Who knows because you can't trust anything? If you're not writing Objective C or Swift or you're not developing apps natively in that way, you're in a React native or Flutter situation to enter the space. That makes sense to a lot of degrees, but it doesn't cover all your bases. It's pluses and minuses many things.

Closing Thoughts

Chuck, last question. If Manchester United could sign any player for the following season, who would it be and why?

I have a lot of conviction about getting a more senior striker, but not senior-senior as in over 30s. We all know that's old. Victor Osimhen is interesting to me. Ivan Toney is also interesting to me. This shows what a physical holdup presence can do for our game, and our ball control is huge. We're solely reliant on this twenty-year-old kid to go and score us 30 goals in 2025. That's unlikely going to happen.

We have two positions that we need. We need a center-back who doesn't have legs of glass. We need a more established center or striker, another target man who can play a billion games with us and also be a good mentor to Rasmus. If I only have to pick one, I'm going for a striker for sure. We need that, and one of those two names that I mentioned is someone who's physical and can score some goals.

We've got one pure striker in the squad. He's small. This is the thing about the Premier League. I went and saw United play for the first time in the last season. He was injured in that game. All of the fans were like, “We are done.” It was before the game started. It's such a tough sport at that level.

We're depending on wingers for goals. It's great. Garnacho scored some incredible goals. Rashford, for the previous season, was able to come from the wing and score a bunch of goals. It's interesting, but the problem is when we have a target man, we need to retrain our wingers to put crosses in and not try to cut in and take a shot. Stop doing that every time.

Chuck, thanks so much for your time. It's been great to hear your story. I look forward to seeing where it goes.

Thanks. I appreciate it. Thanks for having me.

Take care.

‍

Important Links

• Charles Carpenter