New software releases can bring bug fixes, new features, and faster performance. For example, NumPy 1.20 added type annotations, and improved performance by using SIMD when possible. If you’re installing NumPy, you might want to install the newest version. Unfortunately, if you’re using an old version of pip, installing the latest version of a Python package might fail—or install in a slower, more complex way. Why? The combination of glibc versioning, the CentOS end-of-life schedule, and how pip installs packages. […]

You probably don’t want to be an asshole. Being an asshole, as Siderea’s classic essay The Asshole Filter points out, is about being transgressive, about violating social boundaries and rules. And so within the cultural norms of our society, most of us try to avoid being an asshole, by sticking to the expected social boundaries. In programming as in social life, there are boundaries we try not violate: we build software with abstractions, boundaries between the complexity beneath and the […]

If you run a security scanner on your Docker image, you might be in for a shock: often you’ll be warned of dozens of security vulnerabilities, even on the most up-to-date image. After the third or fourth time you get this result, you’ll start tuning the security scanner out. Eventually, you won’t pay attention to the security scanner at all—and you might end up missing a real security vulnerability that slipped through. This is not your fault: the problem is […]

No one enjoys waiting, and waiting for your software to build and tests to run isn’t fun either—in fact, it’s quite expensive. And if you’re building your Docker image in a CI system like GitHub Actions with ephemeral runners—where a new environment gets spinned up for every build—by default your builds are going to be extra slow. In particular, when you spin up a new VM with a new Docker instance, the cache is empty, so when you run the […]

Somebody is always wrong on the Internet, and bad Docker packaging advice is quite common. But one particular piece of advice keeps coming up, and it’s dangerous enough to merit its own article. In a whole bunch of places you will be told not to install security updates when building your Docker image. I’ve been submitting PRs to fix this, so it’s up in fewer places now. But previously this advice was given by the official Docker docs’ best practices […]

You have some data in a relational database, and you want to process it with Pandas. So you use Pandas’ handy read_sql() API to get a DataFrame—and promptly run out of memory. The problem: you’re loading all the data into memory at once. If you have enough rows in the SQL query’s results, it simply won’t fit in RAM. Pandas does have a batching option for read_sql(), which can reduce memory usage, but it’s still not perfect: it also loads […]