Packaging Python dependencies, a not-yet-guide

Several times i have found myself wanting to make a Python app, most often to leverage some existing nice libraries. These generally come properly packaged (in a Python sense) and with dependencies.
And that has been where it stops - i have no idea how to distribute that along with my app, because being well-packaged often makes the python files themselves not necessarily usable right off the bat.

And if i am struggling with this now, i can only imagine what beginner app developers must feel.

This changes now! (ehm… soon)

First things first
Recently i learned that one can install things to a directory, like so:
pip3 install --target=deps <some package>
That’s all well and fine, but despite what you may think and Python may think of itself, you often get architecture-dependent packages and native libraries.

What if we could… yes we can!

sfdk tools package-install SailfishOS-4.4.0.58-aarch64.default python3-pip
sfdk build-shell pip3 install --target=deps <some package>

Q: Why do i have to care about the target name here, but never during more normal operations?
But anyway, doing it properly is as easy as this:
BuildRequires: python3-pip

RPM spec
Now this is basically witchcraft…

After some more convoluted attempts, I just added this in my install section in the rpm spec:
pip3 install -r requirements.txt --target=%{buildroot}/%{_datadir}/%{name}/qml/pages/deps
Q: how do i make an arbitrary file like requirements.txt available in my shadow(?) build environment?
This is driving me nuts! .pro-file ansers does not count.

TODO: have a mode-switch for the rpmbuild where it will upgrade all packages and write a new requirements.txt

Q: I seem to be getting a lot of fake auto-detected “requires”, including python stuff, in my rpm, so on top of what the FAQ says to do for provides:
%define __provides_exclude_from ^%{_datadir}/.*$
Does this seem somewhat reasonable to add?
%define __requires_exclude_from ^%{_datadir}/.*$

Q: I’m also getting warnings about .so files in the wrong place and shipping executables… Sure, the executables i could post-process to throw away, but do i really need to put libs in %{_datadir}/lib? I fear i might not be able to convince python about that.
But anyway; ideas for a good way of automating that as post-processing would be welcome.

The Python side
A bit of a hack, but i just do:

import os
import sys
sys.path.append(os.path.join(os.path.dirname(sys.path[0]),'deps'))
from mydep import something

(Import order, and path concatenation now fixed)
Arguably the QML should do addImportPath() but i find the above to be a better separation of concerns where the Python side self-serves.

Q: How do i put the “main” python file not under qml?
(Changing the import path, and updating the .pro-file was not enough)

Ideas? Suggestions? Any answers and improvements are much appreciated.

This is a Wiki… Musings will gradually, hopefully, give way to instructions. We can re-home this to a better more docsy place when it is done if that is more suitable.
Kudos to @Thaodan for moral support and bearing with me.

6 Likes

sfdk should pick the snapshot of the target by default, you shouldn’t need to directly pick the snapshot.

1 Like

Import sys first, read wrong-import-order / C0411 - Pylint 2.14.0-b1 documentation.

1 Like

This hurt me for years, I am so ready to try this later and get it finally fixed. Thanks!

1 Like

JUST to play devils advocate, don’t get carried away with pip :slight_smile: I have one example (pydub audio library), where it makes MORE sense just to unpack the files, read the code and include them as files. This probably ISN’T the general case, but I really believe you should take a look at source. As soon as I seen nifty library and the dependency graph begins to look like a coral reef, I sigh deeply and start reading.

But, great effort. I hope to contribute soon.

1 Like

Hah, ha. Now that I started playing advocate to the devil, I noticed how many ‘library functions’ I have in my pythonic projects and realized I need to break those out, build packages, endless dependency graphs and … @attah will end up laughing last. …

2 Likes