Embedded Youtube-DL: RFC from App developers

Hi all!

As some of you may know, I have been packaging a version of youtube-dl over on Openrepos.

That version was built using the default build process, which results in only a binary, and no installation of python site-packages.
Which is fine for users, but a bunch of packages by other people also depend on youtube-dl and usually package their own version - which depending on circumstances may get outdated requiring the dev to update their package just to update the dependency.

Now with the advent of SailfishOS Chum, I have moved the package there, with the intent that it can be available in a more standard location, and hopefully a place where updates happen quickly.

I have now also split the package into the youtube-dl standalone binary, and youtube-dl-python, with the latter being a regular python package intended for embedding into applications.

To be found here:

So my question to you fellow devs of applications which embed YTDL here are:

  1. Is having this package available from Chum useful to you, and are you willing/planning to use it?
  2. Is the current state of the package suitable for you to embed it into your app?
  3. Currently this is built using /usr/bin/python, which results in a python2.7 package. I’d actually like to switch to python3 for this – which versions would your apps require? ← fixed, py2 and py3 package available for recent SFOS versions.
  4. Anything else?
4 Likes

There is a huge ecosystem of python packages on pypi.org. Makes me wonder why we need to repackage this to rpm files and not simply resort to pip for application dependencies, which is common practice elsewhere.
If python virtual envs are used, it also becomes easy to have multiple versions installed on a system if needed.
I guess one of the reasons is that it is not allowed in the Jolla store and a pip -r install requirements.txt usually is a command line step. A lot of developers (including myself) decided to include their python libraries in the app rpm itself, to be sure it is the correct version and it won’t cause any conflicts. It’s a bit of a mess and I do not see this will get solved anytime soon.

1 Like

Thank you for your comment! :slight_smile:

I’d argue that it’s not common practice if you look at it in the context Linux distributions. Where it would be bad practice to rely on an ever-changing third-party repository for system packages. Which is why all distributions repackage things from repositories like pypi (or CPAN, or CTAN or whatever) into native packages and manage their releases.
Of course these repos are fine for users and developer machines.

But that is a bit beside the point here.

Youtube-DL is a bit special in that upstream releases rather often as breaking websites are fixed.
In the past, there have been several apps (e.g. Quickddit, VodMan) which at one point broke through such a website change, and the app developer had to bump their package just to update the packaged youtube-dl, with no changes to the app itself.

Having a separate package devs can rely on to be updated independently could improve this situation.

Of course, it means users of those apps will have to use Chum - and we’ll have to see whether that gains traction with users.

1 Like

For the record: I am not talking about python system packages. Those are already available and packaged by Jolla. But YTDL is not a example of a system package. And I won’t call pypi.org (with package versioning) a ever changing third-party repository. It adds new package versions without any effort for the jolla community.

Don’t get me wrong, I applaud any attempt to improve the current situation (and this centralization could help), but I think it does only address a part of the python package issues. But let’s call that an issue beyond this post :wink:

1 Like

I’d use it. I’m working on updates / enhancements for Videoworks, one idea being find on youtube, edit ! :wink:

I’m also looking at doing a sub-directory based install for Imageworks and Videoworks since there is inconsistency in available packages (PIL, in 3.4 is way too old, and just perfect in 4.1, sigh). Are your build files at Peter G. / Youtube Dl · GitLab current for chum? Looks like custom make files?

The obs branch has the chum versions, yes.
There are no custom Makefiles, just one patch, everything else is upstream.

  1. Anything else?

I forked and maintained youtube-dl-lite, and here is the performance part of the README, before it was taken down (as it was forked from youtube-dl).

The version tested is 2020.02.16. I ran the tests on the devices using local terminal, running the command time python3 ./youtube-dl for the scripts and time ./python-dl.bin for the Nuitka compiled versions. For the unzipped Python script runs, the first run always generates the *.pyc files, and this run takes more time. Naturally, these first runs were discarded. With that out of the way; I ran the command five more times in a row and took the average.

My desktop computer was running up-to-date Manjaro Linux with kernel 5.5.2-1. All other devices were running Sailfish OS 3.2.1.

First, the execution times in seconds:

Device \ Variant ytdl-full.py
zipped
ytdl-lite.py
zipped
ytdl-full
unzipped
ytdl-lite
unzipped
ytdl-full
Nuitka
ytdl-lite
Nuitka
AMD Ryzen 7 2700 1.111 0.275 0.114 0.114 0.219 0.116
Sony Xperia XA2 Ultra 6.051 1.741 1.399 0.516 1.287 0.643
Sony Xperia Z3 Compact Tablet 6.366 1.658 1.658 0.666 1.369 0.715
Jolla Phone 12.17 3.138 2.976 1.218 2.422 1.195

And then the same data as percentages. For each row, 100% is the largest execution time, which is the vanilla, zipped full version of youtube-dl.

Device \ Variant ytdl-full.py
zipped
ytdl-lite.py
zipped
ytdl-full
unzipped
ytdl-lite
unzipped
ytdl-full
Nuitka
ytdl-lite
Nuitka
AMD Ryzen 7 2700 100% 25% 10% 10% 20% 10%
Sony Xperia XA2 Ultra 100% 29% 23% 9% 21% 11%
Sony Xperia Z3 Compact Tablet 100% 26% 26% 10% 22% 11%
Jolla Phone 100% 26% 24% 10% 20% 10%

The desktop computer is by far the fastest, as expected. Interestingly, just uncompressing the full version gives the maximum reduction in execution time, whereas the mobile devices still benefit greatly using the uncompressed lite version.

For the mobile devices, the results are practically identical. Just uncompressing the vanilla Python zip cuts ~75% off the execution time, which is very close with using the zipped lite version. The best time was achieved with unzipped lite version, with ~90% of the execution time cut off.

TL;DR Unzipped files let caching work, and boost performance quite a bit. Please consider releasing a second package, which unzips the contents for maximum performance. Thanks!

1 Like

Thanks for the useful feedback. I’ll replace/repackage the unzipped version as well. Probably should have done that in the first place, but if zipped is the default for upstream I figured they’d know better.

Alright, new variant available for testing now.

Chum now has the following:

  • youtube-dl: meta package which depends on one of the -bin variants
  • youtube-dl-bash-completion: just the bash completion file
  • youtube-dl-python: py2 module (for app devs)
  • youtube-dl-python3: py3 module (for app devs)
  • youtube-dl-script: youtube-dl as script (unzipped), depends on the py2 module
  • youtube-dl-standalone: the zipped binary version, as has been shipped previously

installing the meta package will pull in the -script and -python packages per default.

Please test.

3 Likes