Packaging Python dependencies, a not-yet-guide

[attah]

Several times i have found myself wanting to make a Python app, most often to leverage some existing nice libraries. These generally come properly packaged (in a Python sense) and with dependencies.
And that has been where it stops - i have no idea how to distribute that along with my app, because being well-packaged often makes the python files themselves not necessarily usable right off the bat.

And if i am struggling with this now, i can only imagine what beginner app developers must feel.

This changes now! (ehm… soon)

First things first
Recently i learned that one can install things to a directory, like so:
pip3 install --target=deps <some package>
That’s all well and fine, but despite what you may think and Python may think of itself, you often get architecture-dependent packages and native libraries.

What if we could… yes we can!

sfdk tools package-install SailfishOS-4.4.0.58-aarch64.default python3-pip
sfdk build-shell pip3 install --target=deps <some package>

Q: Why do i have to care about the target name here, but never during more normal operations?
But anyway, doing it properly is as easy as this:
BuildRequires: python3-pip

RPM spec
Now this is basically witchcraft…

After some more convoluted attempts, I just added this in my install section in the rpm spec:
pip3 install -r requirements.txt --target=%{buildroot}/%{_datadir}/%{name}/qml/pages/deps
Q: how do i make an arbitrary file like requirements.txt available in my shadow(?) build environment?
This is driving me nuts! .pro-file ansers does not count.

TODO: have a mode-switch for the rpmbuild where it will upgrade all packages and write a new requirements.txt

Q: I seem to be getting a lot of fake auto-detected “requires”, including python stuff, in my rpm, so on top of what the FAQ says to do for provides:
%define __provides_exclude_from ^%{_datadir}/.*$
Does this seem somewhat reasonable to add?
%define __requires_exclude_from ^%{_datadir}/.*$

Q: I’m also getting warnings about .so files in the wrong place and shipping executables… Sure, the executables i could post-process to throw away, but do i really need to put libs in %{_datadir}/lib? I fear i might not be able to convince python about that.
But anyway; ideas for a good way of automating that as post-processing would be welcome.

The Python side
A bit of a hack, but i just do:

import os
import sys
sys.path.append(os.path.join(os.path.dirname(sys.path[0]),'deps'))
from mydep import something

(Import order, and path concatenation now fixed)
Arguably the QML should do addImportPath() but i find the above to be a better separation of concerns where the Python side self-serves.

Q: How do i put the “main” python file not under qml?
(Changing the import path, and updating the .pro-file was not enough)

Ideas? Suggestions? Any answers and improvements are much appreciated.

This is a Wiki… Musings will gradually, hopefully, give way to instructions. We can re-home this to a better more docsy place when it is done if that is more suitable.
Kudos to @Thaodan for moral support and bearing with me.

[Thaodan]

@attah

sfdk tools package-install SailfishOS-4.4.0.58-aarch64.default python3-pip

sfdk should pick the snapshot of the target by default, you shouldn’t need to directly pick the snapshot.

[Thaodan]

@attah

import pyotherside
import sys
sys.path.append("deps")
from mydep import something

Import sys first, read wrong-import-order / C0411 - Pylint 2.14.0-b1 documentation.

[thigg]

This hurt me for years, I am so ready to try this later and get it finally fixed. Thanks!

[poetaster]

JUST to play devils advocate, don’t get carried away with pip I have one example (pydub audio library), where it makes MORE sense just to unpack the files, read the code and include them as files. This probably ISN’T the general case, but I really believe you should take a look at source. As soon as I seen nifty library and the dependency graph begins to look like a coral reef, I sigh deeply and start reading.

But, great effort. I hope to contribute soon.

[poetaster]

Hah, ha. Now that I started playing advocate to the devil, I noticed how many ‘library functions’ I have in my pythonic projects and realized I need to break those out, build packages, endless dependency graphs and … @attah will end up laughing last. …

[poetaster]

@attah did you ever get any answers to your questions ? I’m still doing this by hand which for some dependency graphs is becoming a bit tedious.

[attah]

No, i’m afraid i didn’t get anywhere. But that’s mainly because the program i wanted to package was not modularized enough to be reusable, so there was nothing driving the effort.

[poetaster]

Ah, ok. I have a medium sized one which might be worth it. librosa is candidate to replace pysub for Audioworks. hard reqs on libsndfile python, numpy and cffi. I ‘can’ use my ‘just unpack em’ in folders approach but this might be a candidate. hmmmm. Thanks anyway!

@Thaodan do you have any advice for getting around the issue of installing with requirements.txt files?

[Thaodan]

@poetaster

@Thaodan do you have any advice for getting around the issue of installing with requirements.txt files?

Hm I’m not exactly sure besides pulling in the packages through rpm dependencies but I would try using submodules since they will be fetched when running tar_git and are kept track through git.
The only downside is that you have to track dependencies manually.

[attah]

@Thaodan

pulling in the packages through rpm dependencies

Wouldn’t that mean someone would have had to package them as such first?
And it wouldn’t exactly be harbour compliant? Or am i missing something?

@Thaodan

submodules

Correct me if i’m wrong, but many/most Python projects don’t exactly sit in a usable state in their repos anyway. I.e. they’d still need to be pip-installed.

Is there really no way to expose files from the repo to the shadow build environment?

[takimata]

@attah

These generally come properly packaged (in a Python sense) and with dependencies.

I strongly second packaging your dependencies as real RPM packages. When it comes to OS package managers, Python packages are not proper packages.

I’m doing this and yes, this approach is

• slow
• painful
• dependency hell is haunting you
• and again, it slows you down

But,

• you start to avoid overly complex libraries, thus reducing bloat
• you make use of existing dependency management functions
• you’re making your work reusable for others
• it’s the proper way how it should be done

I don’t think it will fly with Jolla Harbour, though. That’s why we have Chum.

[attah]

…and last but not least out of reach to almost anyone that does this as a hobby only.
Please come down from your high horse. You’re not helping.

[poetaster]

@attah

Correct me if i’m wrong, but many/most Python projects don’t exactly sit in a usable state in their repos anyway. I.e. they’d still need to be pip-installed.

I’m not in a position to correct, but for many repositories that don’t have further dependencies, you can use this approach just copying the src files into your library location. I have one relatively flat example where I’m testing a library and it only has 3 deps that are flat. On the other hand this:

install_requires =
    audioread >= 2.1.9
    numpy >= 1.20.3
    scipy >= 1.2.0
    scikit-learn >= 0.20.0
    joblib >= 0.14
    decorator >= 4.3.0
    numba >= 0.51.0
    soundfile >= 0.11.0
    pooch >= 1.0
    soxr >= 0.3.2
    typing_extensions >= 4.0.0
    lazy_loader >= 0.1
    msgpack >= 1.0

You can forget. And having that packaged for sfos as rpms would take quite a bit of time, too.

[Thaodan]

@poetaster

I’m not in a position to correct, but for many repositories that don’t have further dependencies, you can use this approach just copying the src files into your library location.

That was the idea, with some scripting around pip you could extend this further.

@poetaster

You can forget. And having that packaged for sfos as rpms would take quite a bit of time, too.

Not really most of the heavy lifting for python packages is done in rpm packaging macros.

[poetaster]

@Thaodan

Not really most of the heavy lifting for python packages is done in rpm packaging macros.

I meant in the sense of testing things. In the case above, the python wrapper around libsndfile has obvious complexities with CFFI and numpy, soxr and so. I’ve been avoiding that by just leaning on ffmpeg

[poetaster]

@attah, it’s not really a solution but another tangent. I’m trying to make some repairs to GitHub - lainwir3d/sailfish-rpn-calculator: A RPN calculator for Sailfish OS and came up with this hack.

with dependency directory in rpm/python_modules_src/{sympy,fastcache-1.0.2}

 >> macros
%define __provides_exclude_from ^%{_datadir}/.*$
%define __requires_exclude ^libc|libdl|libm|libpthread|libpython3.7m|libpython3.4m|python|env|libutil.*$
# << macros

....

cd fastcache-1.0.2
python3 setup.py install --root=%{buildroot} --prefix=%{_datadir}/%{name}/

This works quite well. I’m just going to make the rpm directory a submodule and make the python modules submodules of that