Necromancer's notes

Even though I’m mostly a linux user, I sometimes have to deal with windows (and, heck, well even mac sometimes). And not only to fire up steam and relax playing some game or wandering in VR worlds (unfortunately).

So this time I had to deal with distributing a python app that should be available for windows users as well. And windows, just as it usually happens, was a huge pain. At this moment on would ask – why not just use pip? And there goes the long read.

Why not just use pip? (Needs huge downloads)

The app is opensource, so not problem with that. I have submitted the project to pip long ago, so linux users would normally just run ‘pip install rumboot-tools’ and things will just work. Why not do the same in windows?

Well, turns out there are few critical ‘native’ python dependencies (netifaces) that have to compiled in windows. That need at least Visual Studio Build Tools. And the latter is a 6+GB download. And some of the consumers of the app work behind a proxy that limits speed (no porn at the office!) and a huge download will be a huge problem.

The things were made even worse by those ‘effective managers’ at M$ that put the download link so far in the download page, that few users really found it out. Finally, what you can download is not full installer, but a small file that downloads everything from the internet while installing. Is it bad?

On a test run less than a half of the volunteers found the needed download. The remaining ones failed to make it work via proxy. And M$ regularly tells us from the ads that linux is not user-friendly!

Trying out venv (Didn’t work out at all)

The next idea was about getting all the needed stuff in a venv, zip it and send it. Unfortunately this didn’t work. venv hardcodes absolute paths for a python installation with all the minor version numbers. So the resulting package would be very fragile an will require precisely a certain python version installed in a certain directory. And mind that 32/64 bit differences windows users still have to deal with! And since the app comes with a bunch of command-line tools, the end-user will have to deal with adding all that to the PATH as well.

PyInstaller (Not the solution you’re searching for)

Long story short – it didn’t work out of the box. Again a certain python version was needed and the resulting package had a whooping 10% chance to run properly out of the box. Unfortunately it didn’t provide the option to bundle the whole python along. Besides you’ll have to write and maintain a manifest file and test it extensively in order for that thing to work.

Standalone python + hack the needed packages (Why not?)

The idea is neat. Just take the standalone python zip, install the packages and bundle it for distribution. But that’s not that simple. Standalone python has no pip (although you can enable it in a few hacks, though it’s not officially supported). Next install packages, zip and ship. This is the most idiot-proof ™ solution, since we bundle everything in one package. But it’s hard to make, since even after you install pip, standalone builds don’t have all the needed headers so pip can’t compile native extensions. And you’ll still have to make an installer.

WheelHouse (The silver bullet we’ve been searching!)

Turns out the solution was pretty much built inside pip and in the plain sight, but very few people on the internet really considered it or discussed. Perhaps it was that much obvious. pip can build so-called ‘wheels’. Binary packages. Some stuff you download from pip come in those pre-built wheels, but a few of my dependencies didn’t have them.

Even better, pip can build you a set of wheels for your package and all dependencies! Just type inside your package directory

pip wheel .

And magic happens! Now I just needed to run that for all major python versions supported by my app (3.7, 3.8, 3.9) and 32 and 64 bit versions of windows. And generate a quick-n-dirty install.cmd that will install all those wheels. Next – just ‘zip and ship’. Looks like it’s a job for CI.

This set of wheels is called a ‘wheelhouse’ by the python crowd. There’s another pitfall when installing it. Since pip will handle all dependences when generating the wheels, you will normally have to tell pip not to try to download anything from the internel (–no-index) and not process dependencies, when installing every wheel in the list (unless you want to keep the list organized in the order of dependence).

So an install.cmd script will be a huge list of:

pip install --no-index --no-deps pkg1.wheel
pip install --no-index --no-deps pkg2.wheel
pip install --no-index --no-deps pkg3.wheel
pip install --no-index --no-deps pkg4.wheel
...

This can be done either globally (no need to add anything to the PATH) or in a venv. Simple as that and no need for the build tools on user machines any more! Now, for the CI.

For windows I picked appveyor. I see this CI service increasingly often for windows projects. “Visual Studio 2019” image they provide had all the python versions I needed for both 32 and 64 bits. And with a little bit of deployment magic, it can build zips and attach them to the releases page of the github project for every tag. Neat!

Example

For those interested – check out rumboot-tools.