The announcements of 4.4 releases mentiones numerous times that the next SFOS version will switch rpm compression (from xz to zstd).
Does this change affect existing rpms using the old compression, i.e. will they still install on 4.5?
Or is ist that third party apps will have to rebuild the world to continue to be available?
I only ever understood it to mean that 4.5 can now optionally use the new fancy compression (and will do so for jolla packages).
Do you have reason to believe otherwise?
It appears that Fedora and RedHat did the same thing… and surely the internet would have been up in arms if it was a compatibility break.
Yes, RetHat et al also do things like provide backwards compatibility (e.g.by supplying -compat packages for ABI changes), which Jolla traditionally does not. E.g. ncurses, openssl
If this is a package-breaking change I’d like to know so I can schedule time or rebuilding all my published stuff.
Ditto, although it seems like that’d be an option. I’ve re-iterated the question on the .72 page.
Um, where DOES one set the compression option?
It’s also interesting to know if we have to brace for another round of having to pick up working but unmaintained apps (like happened with introduction of aarch64, and SailJail).
I mean @poetaster will do 90% of that in that case as usual (I think he might be an AI to be frank), but still even digital beings have limited resources.
It’s some macros in the rpm(build) environment. And I guess probably a compile-time option for (lib)rpm as well, haven’t checked that.
This is probably going to be important to change back to xz, iff one builds on OBS with a 4.5 target, but wants to publish the result on OpenRepos (for users of older releases).
Nah, just re-discovering my love of computing after a 10+ year period of stagnation (with the exception on one crazy year building electronic musical instruments). And I hate to see perfectly functional code go to waste 
Next rpm issue, how to push all those heavy data parts to ~.local given the current mechanisms of rpms meeting the limits of sailjail.
Thanks for bringing compression up!
RPMs really shouldn’t install anything to user homes.
BUT, you can install into /home/.system/{usr,var,...}, which is located on the /home partition and thus has less space constraints. Only caveat it that apps must ensure they only start after /home has been mounted properly.
I’m aware of the many reasons why (er, multi-seat systems, for one), but in practice? I’m working on stellarium (and a rebuild of machinesvs.) moving the data out of the application area, so I’ll start a new thread once stellarium is done (I fixed all the bugs).
Is there documentation about that somewhere? Should we document that somewhere?
I don’t know. The first time I came across it was when the release notes of MLS Manager mentioned it. (@abranson might have some more information?)
I think it’s main purpose is to move “confidential” but system-wide data (like e.g. wifi passwords) from plain old unencrypted /var into an encrypted location.
Hence the existence of
/home/.system/var/lib/connman-vpn
/home/.system/var/lib/sailjail/settings
/home/.system/var/lib/connman
and similar.
Thanks! It’s actually a reasonable solution, you are correct.
It turns out that I have to re-write some of the data loading code in Stellarium anyway. Sigh. Or I make symlinks from /home/.system/usr/share/stellarium → /usr/share/stellarium. Hmmm. I just tested this route (manually moving and symlinking) and it works fine.
If if weren’t such a lot of data, I’d already have implemented loading on first run, but it’s one of those finding a balance between what is sufficient for the first run is, well, tricky. It’s the stars, after all!
Yes that’s about right. /home/.system is a place where system-wide data is kept on the home partition, without becoming inaccessible when another user logs in.
Btw, the addition of the new compression format for rpms is a reproducible builds thing. I think the old one wasn’t always producing the same result, which meant that builds were being triggered when they shouldn’t be, and deltarpm didn’t work very well.
To make it clear, rpms created with the 4.5.0.16 toolchain can no longer be installed on e.g. a Jolla 1 phone, because the rpm version in SFOS 4.3 cannot handle zstd compression it seems.
@nephros, do I understand correctly that you are suggesting to put
%define _binary_payload w2.xzdio
in spec files to set the compression to xz -2 (as it might have been before)?
Notes:
xz -2 is slow on decompression, but better than bzip2 -9 (w9.bzdio).gzip by %define _binary_payload w9.gzdio (= gzip -9): Slightly worse compression ratio, similar compression speed, but much faster decompression. OTOH, SailfishOS 2.2.1 provides xz / liblzma 5.0.4 (released 22 June 2012), so I assume it has been in SailfishOS since its inception (first releases 0.99.x.y in November 2013).xz's default compression level 6 is very slow, memory- and CPU-intensive for both compression and decompression, and its levels {7,8,9} extremely so (even more than zstd's “ultra”-levels {20,21,22}). See xz's man-page, sections “Memory usage” and “-0 … -9” for details (!).%define _source_payload w2.xzdio or something similar (as often suggested), because “SRPM payload compression should stay at gzip (there’s almost no benefit in changing the compression, because SRPM’s contents is compressed already)”. This also implies that any %define _<*>_payload w<?>.gzdio continues to work fine (although only 9 makes sense for substituting <?>).References:
xz -2 to zstd -19: Changes/Switch RPMs to zstd compression - Fedora Project Wiki
rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
zstd -15 would have been slightly more desirable: Much faster compression speed, faster decompression, only slightly worse compression ratio; this can be achieved by setting %define _binary_payload w15.zstdio in a spec file for SailfishOS ≥ 4.4.0 (requires the corresponding SDK release for building).zstd / libzstd is not available for Fedora < 28, RHEL < 8.2, *buntu < 18.10 etc.; I assume it has been introduced into SailfishOS only recently (v4.4.0?). BTW, Linux integrates zstd for kernel space users (no, this does not address persons 
) since 4.14 (November 2017).xz -2 or bzip2 -9 compresses only slightly better than gz -9 at roughly the same compression speed but much slower decompression; but gzip's default 6 is compressing much quicker and usually yields the same compression ratio: Now I remember why I stubbornly kept using tar -cvzf <tarball> and never switched to anything else, also for compatibility’s sake. This is also the reason why Debian 9 / Ubuntu 18.04 started using xz / lzma compression for packages and used gzip prior to that. OpenWRT and other embedded Linux distributions continue to use gzip compression for their packages due to its lightness (WRT CPU, memory and decompression speed).Can someone please try to install an RPM package built with the Sailfish-SDK for 4.5.0 (or the sailfishos-4.5.0 DoD repository at the SailfishOS-OBS) on SailfishOS ≤ 4.3.0, to ensure that nothing else changed. It must emit at least rpmlib(PayloadIsZst) is needed by <package name>-<version>-<release>.<arch>, but any other rpmlib(…) is needed by <…> would be something to also take into account.
Background: With the switch from w9.gzdio to w2.xzdio in RHEL 5→6, also the filedigest was altered from md5 (to sha1, IIRC), see here (it is historic as for RHEL 5→6, the write-up is erratic, the suggestions flawed, it is just to provide a reference and evidence what happened).
Did a few tests with both, “noarch” and true binary RPM files and SRPM files (….src.rpm) build by rpmbuild and checked by rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
| Built on / with |
binary RPM file | noarch RPM file | SRPM file |
|---|---|---|---|
SFOSÂ 2.2.1rpmbuild
|
xz 6 | gzip 9 | |
| SFOSÂ 3.1.0 SFOS-OBS |
xz 6 | xz 6 | |
SFOSÂ 3.2.1rpmbuild
|
gzip 9 | gzip 9 | |
| SFOSÂ 3.2.1 SFOS-OBS |
xz 6T | xz 6T | |
| SFOSÂ 3.4.0 SFOS-OBS |
xz 6T | xz 6T | |
| SFOS 3.4.0 Sailfish‑SDK   |
xz 6T | xz 6T | |
| SFOS 4.0.1 Sailfish‑SDK |
xz 6T | xz 6T | |
| SFOSÂ 4.4.0 SFOS-OBS |
xz 6T | xz 6T | |
| SFOS 4.4.0 Sailfish‑SDK |
xz 6T | xz 6T | |
| SFOSÂ 4.5.0 SFOS-OBS |
zstd 19 | zstd 19 |
Contribution are welcome, specifically information about binary RPMs built locally (i.e., on a SailfishOS device) by rpmbuild.
No I’m not suggesting anything, but the information from @adekker above is important for packagers who provide packges for older SFOS versions.
I guess setting the compression scheme back to the previous one is a feasible approach for such packages.