Video.js

Check out our open-source Feature Flagging system – Flagsmith on Github! I’d appreciate your feedback ❤️

Episode Transcript:

I'm super interested to say hello to Steve Heffernan. For those who don't know, Steve's probably typed FFmpeg into his console more than most people in the world. He created Video.js, Zencoder, and is the Founder of Mux, which is on a bit of a juggernaut journey, as I've been reading. Welcome, Steve.

Thank you. It's good to be here. It’s good to chat with you.

{{divider}}

My agency already uses Video.js and Zencoder. I feel like video, especially a lot of the technical formats of it and stuff are still mysterious to me. Has that world got simpler in the last several years or is it as complicated?

No. It's as complicated and maybe even getting more complicated. Video is this thing. I've started to say that video is sneaky hard. As we think about other developer services like Stripe and Twilio, payments to me and text messages naturally sound hard. I don’t know how to even start without Twilio. With video, I can record a video on my phone.

I can download the MP4, upload it to a server and embedded it in a video element on the webpage and it looks like it works. From there, it’s like, “It's okay, this must be easy.” As soon as you get into scaling that out to support all the different browsers, devices, all of the different networks across the world, supporting low bandwidth and high bandwidth, from there, it gets infinitely more complex.

{{divider}}

Back when you were going through Y Combinator, the device support wasn't quite so crazy. Talk to me about what came first, Video.js or Zencoder?

Zencoder came first. We built Zencoder to address a need that we saw. We had a web development shop. We were building video applications for other companies. A number of them were trying to do a video. This was like a timeframe where everyone was trying to be the “HD YouTube.” This was before YouTube supported HD and then YouTube became HD YouTube and all of those projects died. Video took off at that time.

We were talking to lots of people who needed the scalable transcoding problem solved. At the same time, the cloud was a new thing in 2009 or 2010. We were still convincing people to trust the cloud and move from closets full of servers to this scalable thing that we can dramatically increase the efficiency of your transcoding by scaling up and scaling down.

Zencoder. We started with YC at the beginning of 2010. We had a few iterations before that but went heads down on it at the beginning of 2010. It was while we were going through Y Combinator. We rented a place near Mountain View, which is what you do for Y Combinator. You go near Y Combinator for three months. You have to figure out your own lodging and all that, but you're meeting with them on a regular basis. We rented a place up in the Santa Cruz Hills, right near Mountain View at this a house that was like you take a small road to a smaller road and you’re in the middle of the Redwood trees. It was totally secluded.

That was great. It was a nice area where we could like focus on building the product. While we were doing this, two Cofounders, Jon and Brandon, had families at the time that they would go to visit on the weekends. They leave me in the middle of the woods in this house with the wild turkeys and the banana slugs to figure out what to work on. I remember one Friday, Brandon made a comment about potentially building controls in Web technologies as opposed to Flash because, at that time, the only way to do a video on the Web was Flash.

I had built controls in Flash a number of times over. In general, I'm a big fan of the Web platform and its openness. From my first experience building a webpage and realizing I could look at other people's web pages and see how they did it and learned from it. I liked Flash, but that element of Flash was that it was closed source, you couldn't learn from it and there are other issues, but it had a lot of good things too.

{{divider}}

Flash is amazing at playing video compared to everything else around. It was good at solving that problem.

They built it for those middle use cases and because it was a plug-in to the browsers, what it had going for is it worked the same on in every browser. There are still elements that we deal with HTML5 video that is not the same across browsers. That was a big element of it. That's why I like Flash ultimately. It used to also have to do QuickTime and Windows Media Player and things like that in the browser. We got to a point where it's like, “Thank God. We can finally use one plug-in to play a video,” and HTML5 came along.

I was excited to build controls in HTML with CSS and JavaScript and took the weekend to go heads down and do that. By the end of the weekend, I had basically a tutorial. I wrote a blog post, “Here's how you build controls.” That got some traction. I turned it into a quick library. You can drop JavaScript on the page. I remember the first weekend of having a website for Video.js.

I had Google Analytics on it at that time and I looked at the stats after the weekend and 20,000 people had visited the site. This was after it got mentioned on Daring Fireball. I remember thinking at the time, “That was like half a baseball stadium that came and visited this website.” Now, those feel like small numbers, at least in comparison to what Videojs.com gets now. It blew me away at that time.

{{divider}}

How old was HTML5 as a ratified standard at that point? Were yours the first blog posts or library that did that?

It was the first open source set of controls. HTML5 was brand new. Less than 30% of browsers in the wild supported HTML5. It was one of those things where I gave talks on convincing people that HTML5 video is the future, especially in professional video. Most people were like, “There's no way we're going to supplant Flash.” Everything we’ve built into Flash, including supporting things like DRM, adaptive streaming, advertising and things like that. No one was convinced that HTML5 video is going to be the future. This was early on, and not too many people were playing with it at that time.

{{divider}}

Was that timing serendipitous or was it planned? Maybe you didn't know at that time that you were escaping to where the puck was going rather than where it was. Was the idea of Video.js’ encoder predicated on this idea that HTML5 was going to become more prominent?

It was probably a mix of lucky timing and seeing where the puck was going. With Zencoder in the cloud, it was like, “Yeah, obviously. The cloud is going to be a thing. We just need to get over this hump and get people on board and this space will win.” It’s the same with Video.js. It was harder to convince people at that time, but as a developer myself, I preferred to use web technologies and the benefits of not using a plug-in in the browser. There were lots of downsides at that time, but as the browser development process caught up, it made a lot of sense. At least the default use case was going to be an HTML5 video, if not the long-term professional use case as well.

{{divider}}

Have you got any experience running an open source project that was on a rocket prior to that?

No, I'd put out some terrible unused tools in the past, some things around the Ruby community or with the Rails community and working with CSS files. I put out this project called RESTful CSS that people tore apart. I was like, “No, this is the wrong way to do it.” Nothing before that had actually got any traction or usage.

{{divider}}

GitHub was a few years old at that point because this was the early days. Did you realize that you had this thing that you could nurture and that would help bring traffic to the corporate side of the business and things like that? Did you have a plan behind it or did you see how it panned out?

It evolved over time. There's the understanding that at first, like any other blog post, this can bring some awareness to Zencoder and the projects that we're doing on that front. It took some time to understand that the project could have a life of its own as opposed to being something that was like, “We built this. We'll make some patches here and there to keep it up to date and evolve it over time that it could become this thing with its own community and plug-in ecosystem. It would require at least an engineer's worth of time constantly on the project, if not more, to support what was going on.”

{{divider}}

In terms of the commercial side and the people investing in Zencoder, it's an interesting situation. It's commercial open source now. It seems like you have a paid SAS API or closed source features or something like that. I feel like the Video.js thing is interesting as well because it's related to Zencoder as a business. I can imagine that 99% of the developers who were using Video.js had no interest or knowledge of Zencoder. Is that fair to say?

It's a delicate balance to walk where you want this open source project to feel like it's community-based. You want to build a community around it that doesn't feel they're at the service of some bigger company. They're contributing to something that they own as much as anybody else owns. At the same time, we have engineers that were paying from Zencoder that are investing their time. We need to be able to justify that investment. What was beneficial for us there was that we took a pretty light approach from a business standpoint. It’s probably debatable what the best business strategy there might've been.

At least it started with and continued with a light approach where we set up Zencoders to clear sponsor of that project. You see it in a lot of open source projects. This is the corporate shepherd of the project. Zencoder by the standard now, like a relatively small logo link on the Video.js homepage at that time. That drove a lot of traffic. I don't know that we said like, “Get your video and coding here.” It was just a logo and it still drove 10% to 20% of the traffic that we got to Zencoder. Think about the jQuery Project. For the longest time, they still have a Media Temple link on their homepage. At that time, that must've been the valuable link out there as far as driving developer traffic.

{{divider}}

When you started looking at them, there are ways that you could jive with that balance because you wouldn't have started Video.js or jQuery hoping to make it rich? They're not the reasons to do those things, are they?

We've continued to develop a core philosophy. The things that we're going to build on the client-side should be open source like web clients and things like that where it's being delivered into user’s browsers and looking for opportunities to open source things, build communities and pull people into our development process.

The client-side is the easy line to draw there, whereas the services that need to be scaled require a lot of backend technologies where you can put an API in front of and monetize with some utility billing, things like that. Certainly, with Video.js, we were not planning on a direct monetization path for that from the start. We were as much solving our own problem. We need to embed videos ourselves and between HTML5 and Flash, there's no obvious way to do this. Why don't we solve it for everybody else too, at the same time?

{{divider}}

In terms of the stewardship and the ownership of that project, is that still something that's under the ownership of Zencoder or is it more of a community-managed project now?

It’s still a community-managed project. The longer story there is Zencoder was acquired by a bigger video company called Brightcove at the end of 2012. Brightcove essentially took over as the corporate shepherd of the project. They put me full-time on the project. Before Brightcove, I was split between Zencoder efforts and Video.js. Brightcove put me full-time on Video.js, which was awesome.

Brightcove had an existing player team that they started to work on the project as well. At the end of 2015, I handed off the lead maintainership of the project to Gary Katsevman, who is one of the Brightcove engineers. He's been leading the project since then and doing a great job stewarding the community there and keeping this place where other developers can jump in, build plug-ins, help out and things like that.

{{divider}}

You mentioned Flash having DRM and some of that DRM stuff is baked into the browsers now. My understanding is, for example, Netflix did a lot of work with Chrome, etc., to add a DRM layer on top of that specification. Is that right?

Yeah. There's been a lot of interesting evolutions, especially since 2010. The next major evolution was called media source extensions, which allowed you to do the original version of the Video Elements. You still do this day, but you would give it a URL to a video, and it would play that back. As far as doing something like adaptive streaming or more interesting, like processing of the video on the client, there was no way to do that until this technology media source extension was developed.

Now, as JavaScript developers, we can pull the video bytes ourselves through the network. Process them if we need to or choose which bytes to pull down based on the network of the clients. Feed those bytes into basically a pipeline into the video element. Encrypted media extensions came after that, which is then building on that concept where you can mix in like DRM keys and encryption into the workflow, and support security from that standpoint.

{{divider}}

Is the project has been on this constant cycle of implementing these new specifications, waiting for the browsers to catch up, then waiting for new extensions to come along? It's a never-ending process by itself. I’m curious as well because there's someone who's a naive layperson. For me, the video works. I don’t need a real player plug-in. I haven't considered a lot of Windows Media extensions for a long time. There's a lot of brainpower being put against this because Netflix works everywhere. They're happy with the DRM side. Other people in the industry have been plugging away to get to that point. I don't even know how my TV runs Netflix. I've got no idea. What the video run time of my TV is, I have no idea.

A video player is an interesting thing, especially an open source video player, because it’s not a project like React or a big framework that you're going to be constantly using, pushing the edges of and building new components on top as part of your main project. At the end of the day, it is a widget. Once it does what you need it to do, you're on to other things.

You're not constantly developing this thing. You may be if you're a Netflix or YouTube. You've got a team behind this, but regularly you think about it as a widget. While at the same time, video is this powerful medium and there's a lot of ways that you want to interact with it when you get into accessibility, security and things like that.

In the browser platform, in general, there's not a lot of things that don't impact video at the end of the day. As browsers and network technologies evolve, video is this interesting first interaction layer of how the controls work and things like that. It touches every part of the network stack. It's such a big file format. It's as heavy compared to images and texts. It doesn't take a lot of scale to be hitting issues on network scaling and things like that. It's interesting as far as on the surface, it's like a widget, but when you dig down deep into it, you could spend your whole life learning everything that’s involved in that.

{{divider}}

It goes down and down. It's a thankless task in a way because it's a bit like taking a penalty as a striker. You expect the video player to play the video. If it doesn't do the video, you expect it to strike a score and if they don't score. I can't imagine people, engineers having lunch and going, “Did you see this new commit in Video.js?” They would do like React or Rails or something. It's hard to delight developers, is what I'm trying to get to you.

That can impact basically building the community around the player. It's easy to get jealous of some of these bigger projects and see the sheer number of people that they have contributing, which I'm sure has its own downsides too. With Video.js, we tend to have a profile of people who pop in and pop out. I'm working on the video player this month so I'm going to pop in, make some beneficial changes, build an interesting plug-in and then pop out. Now, I'm on other projects and the video player is working. Building a community off of that has been probably more challenging and as an open-source maintainer, there's always this question of, “How much time do I spend fixing the issue or building the feature versus nurturing a potential contributor?”

You can build a thing fast, but then you'd be potentially losing out on this contributor who could come in and fix other things and feel ownership around the project. A type of project like Video.js makes that harder because I spent a lot of time where I probably over-indexed on trying to nurture community members. I wanted people to feel that they had ownership around the project, feel that they could contribute and understand how the project worked. I spent a lot of time on issues and educating people. Unfortunately, at the end of the day, we had bigger fires to put out after the video was working.

{{divider}}

Have you got any idea on how your contributors and the community generally skew in terms of professionals or people who work in a professional capacity? I would imagine that it would maybe be a higher ratio than other projects. Do you get a lot of people who are like, “I'm John from Samsung and you're crashing our browser? Let’s try to fix this thing.”

Absolutely. Not having run another project in the same way, it's hard to compare necessarily a profile, but I wouldn't be surprised by that. We definitely had a mix. We had more novice developers and I was helping them understand JavaScript at the same time as understanding how Video.js worked. The people who dug in tends to be more professional developers working with video in a more professional environment. The heaviest contributor is probably LinkedIn. A team from LinkedIn came in and contributed a good chunk of work because they built their video player on top of Video.js. From that standpoint, that profile probably leans that way.

{{divider}}

Has it had the same license throughout its life or has that changed?

That's been an interesting story. The project started as LGPL because I didn't know a whole lot about licenses at that time. I don't know even how prominent something like MIT or Apache was at the beginning of 2010. Having worked with FFmpeg, the most prominent license for us at that time was GPL. GPL is pretty restrictive as far as any contribution you make. You have to contribute back to the project, and it didn't seem to fit the model that we were going for. The next evolution that we could see was LGPL, which was a lot laxer on contributions and freer as far as how you used the project.

{{divider}}

Classically, open source libraries tended almost always to be LGPL like Java JAR or RubyGems.

That sounds right. It's blurry in my memory as far as what else was happening in the ecosystem. At that time, I assume like other developers, I didn't pay a whole lot of attention to the license until I was actually going to put it into some professional capacity. The license was, “We'll worry about this when we need to worry about this, but otherwise, let's see if we can get it working.”

{{divider}}

I expect if you were going through that experience through Y Combinator now that you have a team of people who, “I am going to give you the laydown on how to choose a license and how to build your community.” Back then, none of that existed, or did it?

None of that existed. I don't know if Y Combinator has done that. Y Combinator does have a legal team that's helping out. I'm sure this is a topic that has come up.

{{divider}}

In terms of the experience of building a big open source project and community, leveraging it and etc., I do assume they do.

I remember at some point at Brightcove, we were reviewing Apache 2.0 versus MIT, deciding where a license should be on Video.js. It was probably in 2014 or sometime around then. Corporations were becoming keener. Brightcove's a technology corporation, so they're probably more on the forefront anyway, but starting to understand licenses at that level of corporate legal and what they need to do with that.

I remember at that time, Apache 2.0 was more attractive for the general corporate legal entity because it covered more things. It was more verbose. It covered a lot more details. Legal could feel more comfortable that things were being covered versus MIT, which is simple and probably left some questions unanswered. That's where the project is now. It's Apache 2.0. We definitely years later considered switching to MIT because it feels people don't care as much these days, but these days between Apache 2.0 and MIT, but there was certainly a timeframe where people were almost demanding MIT for the simplicity of it.

{{divider}}

Do you mean your users and contributors?

Yeah. Maybe demanding is a strong word, but the idea is that people understood MIT. It was so prevalent for a while there. It was the new thing. The new license that you put on everything unless you were intentionally trying to restrict it in some way. As people ran into your Apache 2.0, I almost hit a point where corporate legal now understood what MIT was and would sign off on that because, like, “I get the project, yes, sign off on it.” If you presented them with Apache 2.0 or something else, they're, “I haven't seen this before. We got to go through rounds of editing and understand what we're getting ourselves into here.”

{{divider}}

How did that work? Did you push to commit to change the license or what happened at that point? Was it people freaking out?

That's a funny thing. We pushed the license change with a major version update assuming that if anybody had a problem with it, we can at least point to the old version and see how we need to move forward after that. At the end of the day, nobody will seem to care. I'm trying to remember if we had any conversations with anybody external about the license, but if there's anything there, people were positive about the move because of the flexibility that it added. I don't think we've run into any real problems that I can point to you since. I don't know if that's the right way to do it. If you can push to commit and say, “This is now under a new license.” I’m not a lawyer.

{{divider}}

That's the beauty of it. They can still go back to the previous commit, run that and fight that and do whatever they want with that.

I don't know how that applies backward or forwards or anything on those lines. When contributors are involved in how their code is wrapped into that, I hope that's figured out within Apache 2.0 and MIT. We certainly haven't run into any issues on that front yet, at least.

{{divider}}

In terms of your trajectory with a project, you are working for Brightcove and you're working on Video.js and you left Brightcove a few years ago.

At the end of 2015.

{{divider}}

Was that the end of your professional collaboration with the project?

No, it wasn't the end. It was certainly a significant drop in my involvement. I formally handed off the lead maintainership to Gary at that time. That was a hard thing to do because this is a baby that you've built over the last several years. It felt like a significant chunk of my life at that time. Handing that off was a pretty hard thing to do, but handing it off to Gary, who I knew would do a great job with it and having it still have a significant corporate shepherd underneath the Brightcove umbrella made that a lot more comfortable. I knew it wasn't going to be one of these projects that dies because it's got no backing or anything. That meant a lot to me to have the project continue to have that backing.

Now, me and one of the other lead contributors at that time, Matt McClure, stay involved at the technical steering committee. We have conversations with Gary every couple of months to understand higher-level details of the project and pop in from time to time. Otherwise, I'm pretty busy with Mux and everything we're doing there. It's hard to stay involved as much as I would like but still excited for where that project is and where it's going.

{{divider}}

Tell me about starting Mux. You found it in a video business full of juggernaut open source projects and you've still not got enough for video.

There are two sides to that. One is that me and my cofounders of Mux are deeply passionate about video and developer tools for video. The space is endlessly interesting. You can go so deep into video from the UI perspective and get into interesting accessibility details from the encoding side. If you want to go super deep, you're getting into the crazy, alien math that goes into encoding and coding videos.

You can go as deep as you want forever in video. That's part of what kept me so engaged in the topic of video technology. The other side of that is the amount of knowledge, network and everything that I built up in that space. It's unfathomable to think about going to completely different technology and trying to build that from the ground up again.

{{divider}}

What was the kernel for the IDR of Mux? Was this something that you've always been thinking, “This needs to exist?”

We started Mux knowing that we wanted to continue to build the original vision of Zencoder, which was a great developer tool around video. There's still a lot more to solve on that front. My Cofounder Matt started the San Francisco Video Technology Meetup back in 2013 for the benefit of this awesome community of video engineers. A lot of our friends work on YouTube and Netflix.

Especially at that time, 2015, those two companies, YouTube and Netflix, had another level of data and insight into what was happening in their video players around the world. They can make a change to their adaptive algorithm or their video UI in some way and understand the impact on engagement, watch time and things like that.

We knew we wanted to build a suite of developer tools and we also saw this as superpower of data. We ultimately landed on the first product we'd build would be a video monitoring platform that gives you that data, that superpower to make these decisions that help you build better video. It would ultimately help power these other developer tools that we wanted to build and make those even better. That was the Genesis there and how we jumped into things.

{{divider}}

What's the experience of starting a business years later compared to Zencoder? It feels the amount of collateral flying around the West Coast is crazy at the moment.

It's one of those things where we got to the end of our time at Brightcove. You generally forget the painful sides of doing a startup. It’s like, “That was all fun. There's no challenge with doing a startup. Let’s do that again.” You immediately get into it and it’s like, “This is hard. We have a lot of work to do.” A lot of work that's outside of even building the product itself. We went into it at least excited and that drove us through the first few years of doing a startup which can be the hardest years to validate that you're doing something that's worthwhile for users.

It's hard to say how dramatically different. The fundraising environment is different from 2010 to 2016, 2017 and 2018, when we've been raising money. Certainly, on the video side of things, in 2010, half the investors we talked to, one of the first questions was, why don't people use YouTube? There's this mentality that there are two video platforms in the world, YouTube and Netflix and why does anybody care about anything else? That was a hurdle we had to get through with Zencoder.

That is not the case with Mux. There's still an element of that. There's still an element of helping people understand how big a video is going to be because it is becoming a fundamental element of the web that every company is becoming a video company essentially. For most cases, investors are in the know and they understand that video is not those two companies now. There's a lot happening in that space. That only made it easier for us to raise money. Having a previously successful startup makes it easier in general. It is hard to compare from that side of things, but I'm not complaining in any way. It's been helpful.

{{divider}}

When you founded and designed the business, were you thinking we could snap this technology off an open source? Was that in the back of your mind at any point?

My Cofounder Matt, through the San Francisco Video Technology community, grew the Demuxed Conference, which is still community-run. It’s a big conference around video technology, sharing the challenges that you're facing and sharing the solutions that you're coming up with. The area that we focus most is publicizing how we're solving the hard video problems, how we're doing interesting things like per-title encoding and adaptive encoding. We're looking at the content of the video itself, the network of the users that are going to be watching the video and making tweaks to our strategies based on those things and using machine learning to do those things.

It's as much sharing our approaches in an open way. I would say that we haven't yet got to a tool that's easy to open source yet. We solved heavy infrastructure with our video API. We've solved heavy data infrastructure with our data product and we haven't yet solved the client-side well for end-users. From that, we're talking about, as our customers want to build live stream recording into their apps and things that. They have to figure that out themselves and then plug it into our video API or our data product. Ultimately, we will solve those problems for our users. It'll be default open source and we'll be doing a lot in that space there.

{{divider}}

The client's side strategy, structurally, it's difficult to open any of this stuff that you've talked about. You need several Yaka bytes of data to do anything useful with it. It’s a shame that we don't have an opportunity to do something open here. I don't think you necessarily have to. You’re emotionally attached to the projects in this community that you've been working on.

There's a line in my mind between open sourcing small tools and utilities and open sourcing “product” essentially. We have actually open sourced a good number of open sourced things and GitHub. In my mind, those are all independently valuable. Also, we're not productizing those. They're out there. They're there for people to use. We're continuing to use them ourselves and keep them up-to-date.

They're smaller utilities or tools that we haven't necessarily had productized versus something like Video.js. Even though it started as one of these other utilities, it grew into something that we had a product mindset around, at least to understand how we were going to resource it and put engineering behind it and things like that. It's interesting to realize that there's that difference in my mind between the two.

{{divider}}

Sometimes you are working on something and something like that falls in your lap. You didn't design it. You didn't sit down in your forest house going, “This is going to be this huge project.” It’s the timing and the opportunity. I feel to me, that's what's changed a little bit around open source. People don't have a problem with it. If it’s not the technical opportunities to do it and the commercial opportunities to do it, then I don't see that there's any reason why you should necessarily. What calamitous or hilarious or any crazy stories?

It is from the Video.js days. One funny story related to open source development in general is I gave a talk once at the O'Reilly Conference around open source. It’s a big, massive conference with tens of thousands of people. At that time, it was up in Portland, Oregon. I am putting together this whole talk around contributing to contributors. It is what I called it. The idea was this concept of nurturing your community and helping them understand how to use your product. How to set up your open source project to enable people to contribute more, learn how it works and become a good community member.

I spent all this time on it and I got accepted. I'm super excited and it ended up in the keynote track. They had this big auditorium that they'd split up, but they had me on the stage of the keynote thing. This was 5,000 chairs in the audience and 40 people showed up at the dock. They were littered around these 5,000 chairs. It was, “You guys can come forward.” I have to look back and see, “Did I frame that poorly or something?” Maybe I was over-indexing on this idea.

{{divider}}

That must be a tough job trying to figure out what the demand is going to be for a specific tool. They have data around that on their website and stuff. I’ve been at where they put George Hotz in some tiny little room and he had this mad talk. That must be pretty tough. Steve, thanks so much for your time and thanks for the player. I remember using it years ago. It's great to see that it hasn’t fallen by the wayside and is still active. That eleven-year and counting run is something to be proud of.

I appreciate that. It’s great chatting, Ben.

Good luck with Mux. I'll keep an eye on how you are getting on.