Packaging Python dependencies, a not-yet-guide

Several times i have found myself wanting to make a Python app, most often to leverage some existing nice libraries. These generally come properly packaged (in a Python sense) and with dependencies.
And that has been where it stops - i have no idea how to distribute that along with my app, because being well-packaged often makes the python files themselves not necessarily usable right off the bat.

And if i am struggling with this now, i can only imagine what beginner app developers must feel.

This changes now! (ehm… soon)

First things first
Recently i learned that one can install things to a directory, like so:
pip3 install --target=deps <some package>
That’s all well and fine, but despite what you may think and Python may think of itself, you often get architecture-dependent packages and native libraries.

What if we could… yes we can!

sfdk tools package-install SailfishOS-4.4.0.58-aarch64.default python3-pip
sfdk build-shell pip3 install --target=deps <some package>

Q: Why do i have to care about the target name here, but never during more normal operations?
But anyway, doing it properly is as easy as this:
BuildRequires: python3-pip

RPM spec
Now this is basically witchcraft…

After some more convoluted attempts, I just added this in my install section in the rpm spec:
pip3 install -r requirements.txt --target=%{buildroot}/%{_datadir}/%{name}/qml/pages/deps
Q: how do i make an arbitrary file like requirements.txt available in my shadow(?) build environment?
This is driving me nuts! .pro-file ansers does not count.

TODO: have a mode-switch for the rpmbuild where it will upgrade all packages and write a new requirements.txt

Q: I seem to be getting a lot of fake auto-detected ā€œrequiresā€, including python stuff, in my rpm, so on top of what the FAQ says to do for provides:
%define __provides_exclude_from ^%{_datadir}/.*$
Does this seem somewhat reasonable to add?
%define __requires_exclude_from ^%{_datadir}/.*$

Q: I’m also getting warnings about .so files in the wrong place and shipping executables… Sure, the executables i could post-process to throw away, but do i really need to put libs in %{_datadir}/lib? I fear i might not be able to convince python about that.
But anyway; ideas for a good way of automating that as post-processing would be welcome.

The Python side
A bit of a hack, but i just do:

import os
import sys
sys.path.append(os.path.join(os.path.dirname(sys.path[0]),'deps'))
from mydep import something

(Import order, and path concatenation now fixed)
Arguably the QML should do addImportPath() but i find the above to be a better separation of concerns where the Python side self-serves.

Q: How do i put the ā€œmainā€ python file not under qml?
(Changing the import path, and updating the .pro-file was not enough)

Ideas? Suggestions? Any answers and improvements are much appreciated.

This is a Wiki… Musings will gradually, hopefully, give way to instructions. We can re-home this to a better more docsy place when it is done if that is more suitable.
Kudos to @Thaodan for moral support and bearing with me.

9 Likes

sfdk should pick the snapshot of the target by default, you shouldn’t need to directly pick the snapshot.

1 Like

Import sys first, read wrong-import-order / C0411 - Pylint 4.0.0-dev0 documentation.

1 Like

This hurt me for years, I am so ready to try this later and get it finally fixed. Thanks!

1 Like

JUST to play devils advocate, don’t get carried away with pip :slight_smile: I have one example (pydub audio library), where it makes MORE sense just to unpack the files, read the code and include them as files. This probably ISN’T the general case, but I really believe you should take a look at source. As soon as I seen nifty library and the dependency graph begins to look like a coral reef, I sigh deeply and start reading.

But, great effort. I hope to contribute soon.

1 Like

Hah, ha. Now that I started playing advocate to the devil, I noticed how many ā€˜library functions’ I have in my pythonic projects and realized I need to break those out, build packages, endless dependency graphs and … @attah will end up laughing last. …

2 Likes

@attah did you ever get any answers to your questions ? I’m still doing this by hand which for some dependency graphs is becoming a bit tedious.

No, i’m afraid i didn’t get anywhere. But that’s mainly because the program i wanted to package was not modularized enough to be reusable, so there was nothing driving the effort.

Ah, ok. I have a medium sized one which might be worth it. librosa is candidate to replace pysub for Audioworks. hard reqs on libsndfile python, numpy and cffi. I ā€˜can’ use my ā€˜just unpack em’ in folders approach but this might be a candidate. hmmmm. Thanks anyway!

@Thaodan do you have any advice for getting around the issue of installing with requirements.txt files?

1 Like

Hm I’m not exactly sure besides pulling in the packages through rpm dependencies but I would try using submodules since they will be fetched when running tar_git and are kept track through git.
The only downside is that you have to track dependencies manually.

1 Like

Wouldn’t that mean someone would have had to package them as such first?
And it wouldn’t exactly be harbour compliant? Or am i missing something?

Correct me if i’m wrong, but many/most Python projects don’t exactly sit in a usable state in their repos anyway. I.e. they’d still need to be pip-installed.

Is there really no way to expose files from the repo to the shadow build environment?

I strongly second packaging your dependencies as real RPM packages. When it comes to OS package managers, Python packages are not proper packages.

I’m doing this and yes, this approach is

  • slow
  • painful
  • dependency hell is haunting you
  • and again, it slows you down

But,

  • you start to avoid overly complex libraries, thus reducing bloat
  • you make use of existing dependency management functions
  • you’re making your work reusable for others
  • it’s the proper way how it should be done

I don’t think it will fly with Jolla Harbour, though. That’s why we have Chum.

1 Like

…and last but not least out of reach to almost anyone that does this as a hobby only.
Please come down from your high horse. You’re not helping.

I’m not in a position to correct, but for many repositories that don’t have further dependencies, you can use this approach just copying the src files into your library location. I have one relatively flat example where I’m testing a library and it only has 3 deps that are flat. On the other hand this:

install_requires =
    audioread >= 2.1.9
    numpy >= 1.20.3
    scipy >= 1.2.0
    scikit-learn >= 0.20.0
    joblib >= 0.14
    decorator >= 4.3.0
    numba >= 0.51.0
    soundfile >= 0.11.0
    pooch >= 1.0
    soxr >= 0.3.2
    typing_extensions >= 4.0.0
    lazy_loader >= 0.1
    msgpack >= 1.0

You can forget. And having that packaged for sfos as rpms would take quite a bit of time, too.

That was the idea, with some scripting around pip you could extend this further.

Not really most of the heavy lifting for python packages is done in rpm packaging macros.

I meant in the sense of testing things. In the case above, the python wrapper around libsndfile has obvious complexities with CFFI and numpy, soxr and so. I’ve been avoiding that by just leaning on ffmpeg :slight_smile:

@attah, it’s not really a solution but another tangent. I’m trying to make some repairs to GitHub - lainwir3d/sailfish-rpn-calculator: A RPN calculator for Sailfish OS and came up with this hack.

with dependency directory in rpm/python_modules_src/{sympy,fastcache-1.0.2}

 >> macros
%define __provides_exclude_from ^%{_datadir}/.*$
%define __requires_exclude ^libc|libdl|libm|libpthread|libpython3.7m|libpython3.4m|python|env|libutil.*$
# << macros

....

cd fastcache-1.0.2
python3 setup.py install --root=%{buildroot} --prefix=%{_datadir}/%{name}/

This works quite well. I’m just going to make the rpm directory a submodule and make the python modules submodules of that :slight_smile:

1 Like

Thanks for sharing! I have been facing the same issue and adding the pip install command in my.spec file really helped.

A question though: how do you manage to get the requirements.txt file available to the RPM build step?
I ended up asking an LLM for help and eventually duplicated my requirements.txt file inside the /rpm folder and adding it as Source1 in the spec file, but I am pretty sure there must be a cleaner way, for example by leveraging the .pro file in one way or another… So far, all the examples I checked (gpodder-sailfish or the Tidal app referenced in the previous post) integrate their Python dependencies directly in git or through submodules and install them along the app itself in the .pro file.

Tidal player is not really doing installs, just copying files to the apps library directory.

I still use the old school setup.py for a couple of apps:

 cd %{buildroot}%{_datadir}/%{name}/lib/sympy-1.9
 python3 setup.py install --root=%{buildroot} --prefix=%{_datadir}/%{name}/
 rm -rf  %{buildroot}%{_datadir}/%{name}/lib/sympy-1.9

This is evil. The goal is to get the libraries up on chum and then just use packages.

1 Like

Thanks!
I should have been clearer: by ā€œinstallingā€, I meant bundling it in the RPM, not doing an actual pip install.
I completely agree that this approach is evil, but in my case, I do not consider that getting the libs on chum would be worth the effort. I am trying to make an app to ease migration from iOS to SailfishOS (out of personal necessity), something which by design is pretty much a single-use app (install the app, migrate the data, remove the app and enjoy your new phone). The dependency chain looks like this:

 ios_backup_core (not even a proper package on PyPI...)
      - ios-backup-decrypt (on PyPI, only required for encrypted backups)
          - pycryptdome (on PyPI as well, builds native libraries, there is pycryptdomex on Chum though)

In other words, two highly specific packages, and another one which would definitely belong to chum but which I will never have the availability to maintain…