This is the textual version of my June 7 2021 online talk at Upstream Live: "Sidestepping the PR Bottleneck: Four Non-Dev Ways To Support Your Upstreams", 23 minutes. Video is now up.

Intro

Hi, I’m Sumana Harihareswara. I founded Changeset Consulting, which provides targeted, short-term project management services to open source projects and the organizations that depend on them.

Just so you know, I won’t be sharing any slides today, so feel free to look away from the screen. Do some situps, look out the window, fold some laundry, or something. And: If you prefer to read rather than listen, the written version of this is now up on my blog.

If you want open source projects you depend on to succeed/thrive, then you want to support them.

This is both philosophical (you want to be a good open source citizen) and pragmatic (and ensure the health of projects you depend on).

It’s useful and important to do that through patches. One of the first times I spoke at an open source conference was at Open Source Bridge 2010: “The Second Step: HOWTO encourage open source work at for-profits”. I spoke on freeing up people in a company to contribute changes back and gave steps to widen your internal bottlenecks.

But now, even if your developers have time to polish up changes and submit them upstream, when you submit a patch, the bottleneck is on the maintainer’s side, because they don’t have a lot of time to do code review. You’ve contributed pull requests but they’re languishing. If a project is stuck, the gift you have given them isn’t helping – it’s like you’ve given them a gift card to a store that’s really hard to get to, so you’re not really helping them substantively.

Lack of time to do code review is common, so it’s likely your PRs aren’t helping as much as you’d thought!

And that’s if your developers even have time to make and polish PRs; maybe your developer resources are slammed, but you have other folks on your staff whose time is more flexible.

So let’s go into how you can develop a deeper toolbox, and how you can help projects you depend on, including expanding their code review bottleneck. Which benefits you and them.

So today I’ll share case studies and discuss 4 ways to help:

• testing infrastructure – one of the most valuable ways you can sponsor an open source project is through your ops department.
• money – whether you go with Tidelift or another sponsorship option, direct donation can move the needle.
• secondary mentorship – you can help new contributors get situated and stay unblocked.
• coaching and cheerleading – I’ll talk about a time when being an encouraging sidekick made a huge difference.

Testing infrastructure

First off, let’s talk about how your operations department can help developer throughput for a project you depend on – by providing access to infrastructure for use in testing.

As an open source project matures, it comes to a crossroads where maintainers have to ask: what platfoms are we going to support, and how robust will our support be? And their access to testing infrastructure, especially for continuous integration, becomes a key turning point that determines how robust it will be.

Let’s say they want to support the three basic OSes: Windows, Mac, and Linux. But wait, which VERSIONS of those platforms will we support, and which Linux distributions? So now we already have probably at least six or seven things to test.

But what else are we going to support? AMD and Intel chips? People who do and don’t have a particular soft dependency installed? People who run Python 3.5, 3.6, and 3.7?

You can see how these options multiply out, until the project has a test/support matrix with 56 environments.

And, often, projects are limited enough that they can’t rigorously and consistently test their whole matrix – a lot of open source projects can’t afford paid continuous integration services, and they don’t have enough virtual or physical computers to test all the environments they want to support. So maybe 56 environments is too much to handle, and the project ends up with two tiers of support – rigorously tested environments, reflecting the developers’ personal machines, and then everything else. If none of the developers use Windows then Windows is in that “might work” category.

Or the CI services they use limit how many simultaneous jobs they can run, and that slows down the revision and review cycle. Instead of taking 30 seconds or a minute to run CI it might take 10 minutes, or even an hour during peak contribution times like hackathons. This also creates pressure to reduce the number of tests that run on every patch, which means that maintainers have to take more time to review every patch before merging it.

But you can help with this. As a company, you can help your upstream open source projects by providing infrastructure. Give them access so they can use your testing setup for their continuous integration. This is especially useful if your cloud includes the ability to spin up proprietary OSes.

Broader and faster CI helps developer throughput and, in particular, helps the code review cycle go faster.

Money

Next: money.

You’re here at Upstream, so you know that subscribing to a project via Tidelift gives you better assurances of a project’s security, reliability & legal compliance.

But you might be saying: Sumana, how does money specifically help widen the code review bottleneck?

So let’s talk from the point of view of maintainers & what they can do, using lifter income.

First off, with enough income, if a maintainer is spending their time at a job working on something else, sometimes they can quit, and spend time on their open source project.

I’m gonna take a moment now to talk about why that is SOMETIMES. And it has to do with health care and health insurance.

But that’s SOMETIMES. People in a lot of countries, when they quit a job, or go to part-time, they don’t have to worry about how that’ll affect their health insurance. Or, if you’re a tech worker in the US who doesn’t depend on your employer for health insurance, then you have some flexibility, and the more lifter money you get, the more time you can spend on open source. Like if you’re a student getting health insurance through your school, or it’s coming through your spouse or your parents, or you’re a veteran or a senior citizen and you get health insurance through the government.

A lot of tech workers in the US, though, would need to make A LOT of money from lifter income to quit their jobs, buy health insurance on the open market, AND be confident that they could continue to afford that even if the prices go up. So they have less flexibility, and money might not translate so directly into more hours of work.

OK, so, back from that sidebar.

But even so, even if someone can’t quit their day job, so the maintainer still has the same number of hours to spend on their open source project, money can still translate into BETTER work.

A huge example of this is EQUIPMENT. A lot of people volunteering in open source have old or inadequate computers at home, or they don’t have a proper desk, chair, secondary monitor, and stuff like that. A more efficient home office setup – or even JUST a second monitor – can help the quality and speed of someone’s work. Some one-off donations can pay for that, and then over time, a steady lifter income (of a few hundred bucks a month) can help someone move to a bigger place, pay more in rent or mortgage, so they can have a proper home office, with a door that closes, and faster Internet.

Another example is EDUCATION. A lot of open source developers, when they want to learn a new skill, default to looking for what they can read for free online. But someone who’s getting some steady lifter income can pay for books or one-off tutorials to level up, or subscribe to a paid content platform. Like, O’Reilly’s learning platform is $50/month.

And part of how we educate ourselves and each other is with conferences. But most conferences cost money to attend. Sometimes prices for conferences assume all the attendees have their registration costs covered by an employer or a university, and registration costs hundreds of dollars, even over a thousand dollars. Not to mention, if it’s an in-person conference, travel and lodging costs. Lifter income can make the difference, and make going to a conference affordable.

And another example is PAYING INTERNS and FREELANCERS. For example, a maintainer can save up lifting income to pay a freelance writer or designer to improve their website or their documentation, which will save them user support time. Or, for example, a large one-time donation can help break a review and release bottleneck. Last year, donations made it possible for my company to hire a GNU Autoconf expert to get a release out the door, 8 years after the last release. Zack Weinberg took more than six months to review and merge 157 patches, most of which were bugfixes, and make several beta releases, and finally release Autoconf 2.70.

Or a maintainer can save up income to pay an intern, and to teach them to become a co-maintainer. For example, the Outreachy internship program costs a mentoring organization $6500 per intern.

And speaking of interns:

Secondary mentorship

Let’s talk about interns and secondary mentorship. Here’s another way a company can help- you can help a project develop new contributors, and help them get further along the road to becoming co-maintainers.

Think about the engagement funnel for an open source project that wants more contributors and more co-maintainers. An individual contributor starts off not knowing project exists, then becomes interested, and starts contributing. Then, if the project can retain and grow them, they may grow skilled and trustworthy enough to be promoted to become co-maintainer of the project. And the more maintainers you have, the more time is available to do code review.

You can help with a few parts of that funnel. For one thing, if the project isn’t getting new people at the start of the funnel, your marketing department could help with that, with attracting new contributors in the first place. But a lot of the projects you’re working with are actually bottlenecked further along than that – maintainers don’t have enough time to mentor the contributors they have, and grow them to the level where they can promote them.

So I recommend that you help provide SECONDARY mentorship.

When an opensource project is trying to grow new contributors, they often work with internship programs, such as Google Summer of Code or Outreachy. Someone from the project mentors an intern, who’s often new to open source, for three months. But that mentor is usually doing this on top of their other responsibilities. And they’re often pretty new to management, and they’re almost always managing remotely (which is harder than in-person when you’re training someone new), AND sometimes that intern has never had a professional job before and has to learn basic professional skills along the way.

So, that mentor, one of the project maintainers, has a lot of domain knowledge about the project’s codebase, but could probably use an extra pair of hands. This is where you can come in. You can be a secondary mentor.

A secondary mentor does not need to be an expert in or even a contributor to the project codebase. Here’s an example: when I was working on MediaWiki a few years ago, I worked with an Outreachy intern named Frances Hocutt – this was his first programming job, as he was switching careers out of chemistry. Frances’s project was to evaluate and improve the web API client libraries for MediaWiki, in Java, Perl, Python, and Ruby.

One of his mentors was me, but I gathered a co-mentor and two technical advisors: engineers who have different strengths, live in different time zones, and who all promised to respond to questions within two business days. Frances was reading and writing code in four different languages, and was able to get guidance in all of them. The other guys had very different perspectives. Tollef has worked in several open source contexts but had never worked on MediaWiki, so he could approach MediaWiki’s API with learner’s mind and help Frances by modeling how he reasoned about it. Brad had hacked on the API itself and maintained a popular Wikipedia bot that uses it. And Merlijn is a maintainer of an existing client library that lots of Wikimedians use. I brought deep knowledge of our technical community, our social norms, and project management. And I was in charge of the daily “are you blocked?” communication so we could avoid deadlocks.

This worked really well. It made it possible for people to take vacations without Frances getting stuck, and to have more guidance available – from day to day, to help him stay unblocked and always have a next action he could take, but also on bigger topics, like about programming careers and engineering standards, how to judge whether his work was good enough. And after his internship, Frances actually got hired by the Wikimedia Foundation.

Outreachy now highly recommends that every intern has at least two mentors -- and in fact, when I run these mentorship programs, I often MANDATE that every intern has 2 mentors.

See if you can complement a primary mentor by being in a different time zone, and bringing different SKILLS – like project management, technical communication, or testing. You do not have to be a developer to do this. As long as you are reasonably literate in open source process in general, and you actually like helping people learn, you can really improve the odds that an intern will succeed at their internship, want to stick around, and get closer to becoming a project co-maintainer.

Coaching and cheerleading

Finally, let’s talk about coaching and cheerleading – I’ll talk about a time when being an encouraging sidekick made a huge difference.

I did some volunteer work last year, helping rejuvenate pipenv (a command-line tool that some people use to help handle Python packages they make and use).

Pipenv’s maintainers had not released a new version since November 2018, and users were concerned (in many cases switching to competitors). In early March 2020, someone suggested that perhaps the official Python Packaging User Guide should stop recommending it. I saw that suggestion and went into the relevant Internet Relay Chat channel to nudge one of pipenv’s maintainers and to ask: what do you need? What’s blocking you?

Dan knew what was blocking him. He said: “if you’re purely evaluating ‘how do we release the code’, yeah I might just be the main roadblock?” and said he needed “someone to yell at me to stop doing things that are not related to the goal?”

I said: “if you JUST want someone to yell at you to stop doing those unrelated things …. would you actually listen to that person?”

he said: “historically speaking, I’d insist I was doing something important briefly but probably reassess, I do know what needs to happen”

So, over the next three months, I donated about 15 hours of my time. Given my current hourly rates, this constitutes a donation worth a few thousand dollars, which is infinitesimal compared to the value unlocked by expediting a pipenv release. And with those 15 hours of work, I enabled Dan to do fix bugs, do code review, merging several pull requests in, and to make the release – releases mean your patches get not only reviewed but also released so you don’t have to maintain a fork! All in all, with my help, Dan released two betas, then a stable version in May.

Dan needed someone to help him with prioritization, release management, and communications. So I:

• rewrote the release checklist and kept it updated
• updated GitHub issues and tweeted to keep users apprised, and to ask them for help testing the beta releases (including suggestions of specific cases to test, and requests for specific expertise when Dan ran into nasty continuous integration bugs)
• created draft outlines for him to fill in so he could write a status update and the beta release announcement for mailing lists/forums
• closed duplicate "is pipenv dead?" issues and updated a longer-term issue about maintainer needs, roadmap, etc.
• made a few documentation pull requests: helped move Pipenv docs from a nonfunctioning domain to a better one, fixed a bunch of obsolete links, and edited Dan's "how to release" docs, and made a pull request to move them into the repo and out of an ephemeral hackpad
• every few days, checked in with Dan via IRC or email (we also had a few videocalls) to help him stay on task

That was 15 hours in total.

Those 15 hours of coaching and communication and cheerleading enabled Dan to review pull requests, fix bugs, and get that release out. And since then pipenv has made releases on a pretty steady clip.

So: An external perspective can help a lot. You can be that person. Whether you call yourself a sidekick, a project manager, a cheerleader, a coach, or something else, you can be a supportive accountability partner who helps with the bits that maintainers are not great at, or don’t have time for. And you don’t have to know the project codebase to do this, or be a feature-level developer - the only pipenv code I touched was the docs.

If you/your company depends on an open source project and you’re getting annoyed or worried because the release cadence has slowed to a standstill, there’s a strong chance you can turn that around. If someone on your team can spend a few hours complementing the existing maintainers and helping unblock them, that could save you a bundle compared to forking or switching dependencies. Try talking with the maintainers about what they need. And I do mean talking, as in, synchronous conversation via voice or chat, so you can build some trust and get the kind of conversation I had with Dan.

Conclusion

So: testing infrastructure, money, secondary mentorship, and coaching and cheerleading can all move the needle, to increase the health of projects you care about – and decrease the code review backlog so your patches get into the mainline.

I’m Sumana Harihareswara, and I can help you make this happen through my consultancy, Changeset Consulting. And let’s talk more in the chat after this talk – I’d love to hear your stories about what works.