Blog by Sumana Harihareswara, Changeset founder
Links and References For My PyCon US Keynote
Today I’m giving the closing keynote address at PyCon US 2024, sharing “Untold stories from six years working on Python packaging.”
I aim to post a fuller transcript with slides within the next several weeks. But, in advance, here are a bunch of links and other references so you can follow up.
[Edited 5 June 2024 to add: here are the slides and transcript: PyCon US 2024 keynote slides and transcript ; I presume the video will be up at PyCon's YouTube channel this month, and indexed at PyVideo at some point as well. And: in my thank-yous, I forgot to mention the audiovisual/tech staff at PyCon US, who were phenomenal. Thank you, A/V staff!]
[Edited 25 October 2024 to add: Link to video.]
Python packaging specifically
[Edited 5 June 2024 to add: Nathaniel Smith's site, and his NumPy work.]
The grant-funded Python packaging projects, most of which I managed or helped obtain funding for. I was the project manager for and/or a grant proposal writer for:
- 2017-2018: replacing legacy PyPI with the new Warehouse codebase (thanks, Mozilla Open Source Support program)
- 2019: security, localization, and accessibility improvements for Warehouse (thanks, Open Tech Fund)
- 2019-2020: further security improvements to PyPI (I didn’t manage this one) (Thanks, Facebook)
- 2020: overhauling the resolver at the heart of pip, and improving user experience (Thanks, MOSS and Chan Zuckerberg Initiative)
- 2021-2022: more improvements to packaging security (not listed on the wiki page) (Thanks, National Science Foundation)
(My keynote goes into more depth on the legacy PyPI replacement project and the pip resolver project, relative to the other projects)
Dustin Ingram’s 2018 talk “Inside the Cheeseshop” which goes into more detail on the history of PyPI and packaging more broadly
My simplified diagram of Python packaging is adapted from this more complex (yet still simplified) diagram
A simplified diagram illustrating some of the important tools in Python packaging and how they relate to each other. On the left, an end user uses pip to download and install cool software on a computer. They draw information from the cloud-based Python Package Index. That in turn draws information from developer packaging tools to the right, which the developer's cool software feeds into.
NumPy, “The fundamental package for scientific computing with Python”
The Python packaging platypus mascot
Donald Stufft’s 2016 blog post on legacy PyPI's architecture and the under-resourcing of PyPI as further context for the funded work that started in 2017
From the 2017-2018 work overhauling PyPI: announcing the Mozilla grant and the coming switchover from legacy PyPI to Warehouse, then asking for project maintainers to test the new site, then announcing the beta, then announcing that the old site was shutting down
Current Python Software Foundation staff, including multiple staffers who work on packaging infrastructure, and the announcement that they’re hiring one more
The original video we made to publicize changes coming to pip in 2020; I edited it to improve sound and timing, and to update Ee Durbin’s name.
More on Mel Chua's digital infrastructure research and the encapsulation paradox; Mel Chua is currently very ill, and financial contributions are welcome
From pip documentation: Things to watch out for in the new resolver
Blog post, March 2020, announcing rollout of the new pip resolver
Release announcement, November 2020, of pip 20.3 (the new resolver became default)
the pypi-announce mailing list, a low-traffic list to receive announcements of major changes to PyPI and other major packaging tools
an example “where do you get your news?” comment and publicity and outreach work I did for a pip beta release
The user experience research and design documentation for pip and Python packaging specifically, developed in 2020; Superbloom later created a HOWTO on user testing
Before-and-after screenshots of pip’s redesigned error messages: pull requests number 10959, 10795, and 10703 – thanks to Pradyun Gedam for pointing to those. (Also: Pradyun Gedam’s new library “diagnostic”, further developing his interest in improving error messages in command-line tools more broadly.)
Additional resources
Superbloom’s HOWTO on user testing
Rebecca Solnit’s book Hope in the Dark: Untold Histories, Wild Possibilities which makes the point that you just don’t know what your work will lead to. You don’t. You plant seeds and you may never get to see them sprout. And failures can ready the ground for success.
Making Software: What Really Works, And Why We Believe It edited by Andy Oram and Greg Wilson – a compendium of research on software engineering, one of the books I recommend most frequently, and here I’m referring to the “Code Talkers” chapter (what programmers often need to understand about a codebase and can’t gather without asking another person) and the “Novice Professionals” chapter (recommends a practice of “feature interviews” as part of new developer onboarding). I also recommend the followup project Never Work in Theory, highlighting useful software engineering research to “be a bridge between researchers and practitioners.”
(Also, you might find that the best way to extract “design rationale” and similar knowledge from someone’s head is to have them give you a “live tour”.)
Chelsea Troy’s blog post on the effectiveness of tech conferences. “concentrate the right groups of people into a space to catalyze conversations that lead to Big Things…. Cons become worth it from the conversations that happen between folks who otherwise might not have met, that endure beyond the event itself…. I (and you) have almost certainly benefited from cons we’ve never been to and never heard of.”
The next Volunteer Responsibility Amnesty Day is on the next solstice, June 20th, 2024
My 2017 MetaFilter blog post about AEDs, “Restart a heart near you”
The 2023 press release announcing the passage of The HEART Act
An example of the Wellerman sea shanty as shared on social media in late 2020/early 2021
And: some additional song lyrics. [Edited 5 June 2024 to explicitly mention that the song is "Solidarity Forever", lyrics by Ralph Chaplin, circa 1915.]
Comments
