If you work on open source software, especially command-line tools, I want you to know about newly available research reports and design guidance, and a user research HOWTO, that you can pick up and reuse.

The pip work and how you can learn from it

Back in 2020, during grant-funded work on the next-generation pip resolver, the consultancy Superbloom (previously Simply Secure) did fascinating user experience research and design work. I was the project manager on this work, and can attest that working with UX experts on pip was crucial and valuable. The more we knew about users’ experience, the better decisions we could make.

During that work, they wrote several useful documents that took a while to get merged, but now live in pip’s documentation! Like:

• how users thought pip should react to dependency conflicts
• what users used pip search for
• how pip users thought about and dealt with package security
• how to design a survey for pip

Even if your work has nothing to do with pip, consider skimming those. The most exciting thing about them isn't their literal content, but the fact that this is possible. Open source projects -- including command-line tools -- can get help systematically understanding users and improving to meet their needs better.

Questions you can answer

As I mentioned last year, I recently talked with David Lord, a Pallets maintainer, who wants to know for his projects:

• He puts effort into consistent deprecation warnings. Are they effective? Are people seeing those warnings early enough to avoid unpleasant surprises when functionality changes? If not, what could he do to communicate this information to them more effectively?
• Who’s using this software? That is: do their needs cluster around a few major use cases/workflows? If so, what are they?
• Which, if any, of the new features from the past few years do they use?
• How do they seek help? Do they make an effort to start with the official docs, or just search the web and often find old blog posts written by enthusiasts?
• What level of detail do they need in an error message spouted to the command line, and what would be so long that they’d be more likely to skip reading it?

He's not alone! So many maintainers would like to know this kind of information, but they fundamentally assume it isn't possible. (And I could spend a few paragraphs on why they make that assumption, but I'll skip that today.)

Yet this is the kind of information we got through Superbloom's work.

Example: their research findings helped us improve the format and content of the ResolutionImpossible error message, so users could actually work out what went wrong and how to fix it. UX analyses and training have helped massively improve pip’s user experience design, including error messages and docs, as well as the direct ergonomics of invoking the right commands and having them do what users expect. Another example that hasn't been completely merged in yet: an analysis of some confusing error messages and a suggested template for clearer error message formats.

Join forces

We didn’t limit ourselves to only researching the UX of pip, because that wouldn’t make sense; pip is an interdependent part of the Python package creation and distribution ecosystem, and we needed to understand how users reasoned about, researched, and learned about packaging broadly. And – I believe – learning these tools and facts has been, and can be, helpful to other packaging tools maintainers beyond pip.

And I see this as a model that we can replicate in other ecologies, especially groups of related command-line tools. Pool funds and invest in UX research for a suite of tools that people often use together, and learn surprising and helpful things that help you all improve your projects’ developer experience and user experience.

More resources

I hope this work can inspire maintainers elsewhere to do likewise. You can apply for grants or recruit sponsors to fund this sort of work. Or, if it's easier to learn UX research skills than to hire them, you could learn. Last year, Superbloom published more resources to help you start to do user testing and usability testing in your projects, including:

• a user testing HOWTO, also available as an interactive guide on itch.io
• a HOWTO for synthesis (“stage of research in which you reorganize information to make sense of what you observed and heard”)

And the pip work now joins other guidance and case studies in improving command-line UX, such as Command Line Interface Guidelines, and the Rust compiler error style guide.

Hope this is helpful! I’m Sumana Harihareswara, and I can help you with this sort of thing through my consultancy, Changeset Consulting; I do project management, coaching, training workshops, and more. Anyone reading this is eligible for a short free 30-minute consultation call; email me.

And I’m working on a book on managing existing open source projects, so you can learn how to get them unstuck. You can read three sample chapters by signing up for my newsletter.

(This post is a remix of my Fediverse thread and my Python Discourse forum post on the same topic, in case you want to comment in one of those places.)