Embedded Youtube-DL: RFC from App developers

Applications
#1

Hi all!

As some of you may know, I have been packaging a version of youtube-dl over on Openrepos.

That version was built using the default build process, which results in only a binary, and no installation of python site-packages.
Which is fine for users, but a bunch of packages by other people also depend on youtube-dl and usually package their own version - which depending on circumstances may get outdated requiring the dev to update their package just to update the dependency.

Now with the advent of SailfishOS Chum, I have moved the package there, with the intent that it can be available in a more standard location, and hopefully a place where updates happen quickly.

I have now also split the package into the youtube-dl standalone binary, and youtube-dl-python, with the latter being a regular python package intended for embedding into applications.

To be found here:

So my question to you fellow devs of applications which embed YTDL here are:

  1. Is having this package available from Chum useful to you, and are you willing/planning to use it?
  2. Is the current state of the package suitable for you to embed it into your app?
  3. Currently this is built using /usr/bin/python, which results in a python2.7 package. I’d actually like to switch to python3 for this – which versions would your apps require? ← fixed, py2 and py3 package available for recent SFOS versions.
  4. Anything else?
4 Likes
#2

There is a huge ecosystem of python packages on pypi.org. Makes me wonder why we need to repackage this to rpm files and not simply resort to pip for application dependencies, which is common practice elsewhere.
If python virtual envs are used, it also becomes easy to have multiple versions installed on a system if needed.
I guess one of the reasons is that it is not allowed in the Jolla store and a pip -r install requirements.txt usually is a command line step. A lot of developers (including myself) decided to include their python libraries in the app rpm itself, to be sure it is the correct version and it won’t cause any conflicts. It’s a bit of a mess and I do not see this will get solved anytime soon.

1 Like
#3

Thank you for your comment! :slight_smile:

I’d argue that it’s not common practice if you look at it in the context Linux distributions. Where it would be bad practice to rely on an ever-changing third-party repository for system packages. Which is why all distributions repackage things from repositories like pypi (or CPAN, or CTAN or whatever) into native packages and manage their releases.
Of course these repos are fine for users and developer machines.

But that is a bit beside the point here.

Youtube-DL is a bit special in that upstream releases rather often as breaking websites are fixed.
In the past, there have been several apps (e.g. Quickddit, VodMan) which at one point broke through such a website change, and the app developer had to bump their package just to update the packaged youtube-dl, with no changes to the app itself.

Having a separate package devs can rely on to be updated independently could improve this situation.

Of course, it means users of those apps will have to use Chum - and we’ll have to see whether that gains traction with users.

1 Like
#4

For the record: I am not talking about python system packages. Those are already available and packaged by Jolla. But YTDL is not a example of a system package. And I won’t call pypi.org (with package versioning) a ever changing third-party repository. It adds new package versions without any effort for the jolla community.

Don’t get me wrong, I applaud any attempt to improve the current situation (and this centralization could help), but I think it does only address a part of the python package issues. But let’s call that an issue beyond this post :wink:

1 Like
#5

I’d use it. I’m working on updates / enhancements for Videoworks, one idea being find on youtube, edit ! :wink:

I’m also looking at doing a sub-directory based install for Imageworks and Videoworks since there is inconsistency in available packages (PIL, in 3.4 is way too old, and just perfect in 4.1, sigh). Are your build files at Peter G. / Youtube Dl · GitLab current for chum? Looks like custom make files?

#6

The obs branch has the chum versions, yes.
There are no custom Makefiles, just one patch, everything else is upstream.

#7
  1. Anything else?

I forked and maintained youtube-dl-lite, and here is the performance part of the README, before it was taken down (as it was forked from youtube-dl).

The version tested is 2020.02.16. I ran the tests on the devices using local terminal, running the command time python3 ./youtube-dl for the scripts and time ./python-dl.bin for the Nuitka compiled versions. For the unzipped Python script runs, the first run always generates the *.pyc files, and this run takes more time. Naturally, these first runs were discarded. With that out of the way; I ran the command five more times in a row and took the average.

My desktop computer was running up-to-date Manjaro Linux with kernel 5.5.2-1. All other devices were running Sailfish OS 3.2.1.

First, the execution times in seconds:

Device \ Variant ytdl-full.py
zipped		 ytdl-lite.py
zipped		 ytdl-full
unzipped		 ytdl-lite
unzipped		 ytdl-full
Nuitka		 ytdl-lite
Nuitka
AMD Ryzen 7 2700 1.111 0.275 0.114 0.114 0.219 0.116
Sony Xperia XA2 Ultra 6.051 1.741 1.399 0.516 1.287 0.643
Sony Xperia Z3 Compact Tablet 6.366 1.658 1.658 0.666 1.369 0.715
Jolla Phone 12.17 3.138 2.976 1.218 2.422 1.195

And then the same data as percentages. For each row, 100% is the largest execution time, which is the vanilla, zipped full version of youtube-dl.

Device \ Variant ytdl-full.py
zipped		 ytdl-lite.py
zipped		 ytdl-full
unzipped		 ytdl-lite
unzipped		 ytdl-full
Nuitka		 ytdl-lite
Nuitka
AMD Ryzen 7 2700 100% 25% 10% 10% 20% 10%
Sony Xperia XA2 Ultra 100% 29% 23% 9% 21% 11%
Sony Xperia Z3 Compact Tablet 100% 26% 26% 10% 22% 11%
Jolla Phone 100% 26% 24% 10% 20% 10%

The desktop computer is by far the fastest, as expected. Interestingly, just uncompressing the full version gives the maximum reduction in execution time, whereas the mobile devices still benefit greatly using the uncompressed lite version.

For the mobile devices, the results are practically identical. Just uncompressing the vanilla Python zip cuts ~75% off the execution time, which is very close with using the zipped lite version. The best time was achieved with unzipped lite version, with ~90% of the execution time cut off.

TL;DR Unzipped files let caching work, and boost performance quite a bit. Please consider releasing a second package, which unzips the contents for maximum performance. Thanks!

1 Like
#8

Thanks for the useful feedback. I’ll replace/repackage the unzipped version as well. Probably should have done that in the first place, but if zipped is the default for upstream I figured they’d know better.

#9

Alright, new variant available for testing now.

Chum now has the following:

  • youtube-dl: meta package which depends on one of the -bin variants
  • youtube-dl-bash-completion: just the bash completion file
  • youtube-dl-python: py2 module (for app devs)
  • youtube-dl-python3: py3 module (for app devs)
  • youtube-dl-script: youtube-dl as script (unzipped), depends on the py2 module
  • youtube-dl-standalone: the zipped binary version, as has been shipped previously

installing the meta package will pull in the -script and -python packages per default.

Please test.

5 Likes
#10

Since youtube-dl proper has pretty much stalled development, I have now done an experimental version of one of its successors, yt-dlp:

https://build.sailfishos.org/package/show/home:nephros:devel:python/yt-dlp

Please test, if you care. :wink:

If I get at least one or two positive replies, I’ll submit to chum.

2 Likes
#11

Not really, AFAICS:

Side note: yt-dlc (which was a starting point for yt-dlp) ceased to be maintained more than a year ago.

The reason why I denote that, is that I am afraid to loose a recent yt-dl (as the mainline thing, which most people use) on SailfishOS.
While yt-dlp states, “The main focus of this project is adding new features and patches while also keeping up to date with the original project”, it is still an more experimental (“development”) project with far less users. It also is not a fully compatible drop-in replacement, so it might cause issues, depending on how the calling app uses it.

So, while I appreciate you initiative to provide a alternative to the original yt-dl, I hope that you will maintain the original yt-dl for SailfishOS as long as it is maintained upstream.

#12

Not to worry, once rpm spec files are done staying up to date with upstream is not much more than a git pull in the packaging repo.

So it should be easy enough to maintain both.

(And both can be installed alongside each other.)

1 Like
#13

Cool. I’ll see if I can get to some python tests for yt-dlp. I was meaning to add a feature for Videoworks to pull/source video from youtube.