SFOS 4.5.0 switches RPM compression. What does that mean for existing RPMs?

thigg · 3 October 2022 09:54

Is there documentation about that somewhere? Should we document that somewhere?

nephros · 3 October 2022 10:14

I don’t know. The first time I came across it was when the release notes of MLS Manager mentioned it. (@abranson might have some more information?)

I think it’s main purpose is to move “confidential” but system-wide data (like e.g. wifi passwords) from plain old unencrypted /var into an encrypted location.

Hence the existence of

/home/.system/var/lib/connman-vpn
/home/.system/var/lib/sailjail/settings
/home/.system/var/lib/connman

and similar.

poetaster · 3 October 2022 11:48

Thanks! It’s actually a reasonable solution, you are correct.

It turns out that I have to re-write some of the data loading code in Stellarium anyway. Sigh. Or I make symlinks from /home/.system/usr/share/stellarium → /usr/share/stellarium. Hmmm. I just tested this route (manually moving and symlinking) and it works fine.

If if weren’t such a lot of data, I’d already have implemented loading on first run, but it’s one of those finding a balance between what is sufficient for the first run is, well, tricky. It’s the stars, after all!

abranson · 3 October 2022 13:15

Yes that’s about right. /home/.system is a place where system-wide data is kept on the home partition, without becoming inaccessible when another user logs in.

Btw, the addition of the new compression format for rpms is a reproducible builds thing. I think the old one wasn’t always producing the same result, which meant that builds were being triggered when they shouldn’t be, and deltarpm didn’t work very well.

adekker · 11 February 2023 09:57

To make it clear, rpms created with the 4.5.0.16 toolchain can no longer be installed on e.g. a Jolla 1 phone, because the rpm version in SFOS 4.3 cannot handle zstd compression it seems.

olf · 14 February 2023 01:57

@nephros, do I understand correctly that you are suggesting to put
%define _binary_payload w2.xzdio
in spec files to set the compression to xz -2 (as it might have been before)?

Notes:

xz -2 is slow on decompression, but better than bzip2 -9 (w9.bzdio).
The only real alternative is to use gzip by %define _binary_payload w9.gzdio (= gzip -9): Slightly worse compression ratio, similar compression speed, but much faster decompression. OTOH, SailfishOS 2.2.1 provides xz / liblzma 5.0.4 (released 22 June 2012), so I assume it has been in SailfishOS since its inception (first releases 0.99.x.y in November 2013).
xz's default compression level 6 is very slow, memory- and CPU-intensive for both compression and decompression, and its levels {7,8,9} extremely so (even more than zstd's “ultra”-levels {20,21,22}). See xz's man-page, sections “Memory usage” and “-0 … -9” for details (!).
Do not set %define _source_payload w2.xzdio or something similar (as often suggested), because “SRPM payload compression should stay at gzip (there’s almost no benefit in changing the compression, because SRPM’s contents is compressed already)”. This also implies that any %define _<*>_payload w<?>.gzdio continues to work fine (although only 9 makes sense for substituting <?>).

References:

Excellent documentation with background information for Fedora 30→31 switching from xz -2 to zstd -19: Changes/Switch RPMs to zstd compression - Fedora Project Wiki
- How to test an installed package (requires RPM ≥ 4.14.0, which is in SailfishOS since at least v2.2.1):
  rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
- IMO zstd -15 would have been slightly more desirable: Much faster compression speed, faster decompression, only slightly worse compression ratio; this can be achieved by setting %define _binary_payload w15.zstdio in a spec file for SailfishOS ≥ 4.4.0 (requires the corresponding SDK release for building).
- BTW, RHEL 8→9 followed this switch, as SailfishOS 4.4.0→4.5.0 does, Ubuntu 22.04 did and Arch Linux did in January 2020.
- But zstd / libzstd is not available for Fedora < 28, RHEL < 8.2, *buntu < 18.10 etc.; I assume it has been introduced into SailfishOS only recently (v4.4.0?). BTW, Linux integrates zstd for kernel space users (no, this does not address persons ) since 4.14 (November 2017).
- Using xz -2 or bzip2 -9 compresses only slightly better than gz -9 at roughly the same compression speed but much slower decompression; but gzip's default 6 is compressing much quicker and usually yields the same compression ratio: Now I remember why I stubbornly kept using tar -cvzf <tarball> and never switched to anything else, also for compatibility’s sake. This is also the reason why Debian 9 / Ubuntu 18.04 started using xz / lzma compression for packages and used gzip prior to that. OpenWRT and other embedded Linux distributions continue to use gzip compression for their packages due to its lightness (WRT CPU, memory and decompression speed).
Need new option to control RPM compression level · Issue #1675 · jordansissel/fpm · GitHub
1715799 – Re-enable zstd compression support in RHEL-8 rpm
https://github.com/eclipse-archived/packagedrone/pull/133#discussion_r228995829
Historic (on RHEL 4), but informative: rpm - rpmbuild change compression format - Stack Overflow

Can someone please try to install an RPM package built with the Sailfish-SDK for 4.5.0 (or the sailfishos-4.5.0 DoD repository at the SailfishOS-OBS) on SailfishOS ≤ 4.3.0, to ensure that nothing else changed. It must emit at least rpmlib(PayloadIsZst) is needed by <package name>-<version>-<release>.<arch>, but any other rpmlib(…) is needed by <…> would be something to also take into account.
Background: With the switch from w9.gzdio to w2.xzdio in RHEL 5→6, also the filedigest was altered from md5 (to sha1, IIRC), see here (it is historic as for RHEL 5→6, the write-up is erratic, the suggestions flawed, it is just to provide a reference and evidence what happened).

olf · 14 February 2023 03:19

Did a few tests with both, “noarch” and true binary RPM files and SRPM files (….src.rpm) build by rpmbuild and checked by rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>

Built on / with	binary RPM file	noarch RPM file	SRPM file
SFOS 2.2.1 `rpmbuild`		xz 6	gzip 9
SFOS 3.1.0 SFOS-OBS	xz 6	xz 6
SFOS 3.2.1 `rpmbuild`		gzip 9	gzip 9
SFOS 3.2.1 SFOS-OBS	xz 6T	xz 6T
SFOS 3.4.0 SFOS-OBS	xz 6T	xz 6T
SFOS 3.4.0 Sailfish‑SDK	xz 6T	xz 6T
SFOS 4.0.1 Sailfish‑SDK	xz 6T	xz 6T
SFOS 4.4.0 SFOS-OBS	xz 6T	xz 6T
SFOS 4.4.0 Sailfish‑SDK	xz 6T	xz 6T
SFOS 4.5.0 SFOS-OBS	zstd 19	zstd 19

Contribution are welcome, specifically information about binary RPMs built locally (i.e., on a SailfishOS device) by rpmbuild.

nephros · 14 February 2023 13:04

No I’m not suggesting anything, but the information from @adekker above is important for packagers who provide packges for older SFOS versions.

I guess setting the compression scheme back to the previous one is a feasible approach for such packages.

olf · 14 February 2023 19:02

Well, after researching the topic RPM compression much more in-depth than originally intended, I can come up a conclusive suggestion.

but the information from @adekker above is important for packagers who provide packges for older SFOS versions.

I do not see that anybody questioned that, rather the opposite: Jolla’s and @dekker’s information, plus your “non-suggestion” made me research and document this.

I guess setting the compression scheme back to the previous one is a feasible approach for such packages.

Well, “the previous one” is not really conclusive (see my prior post), and I do think that RedHat’s / Fedora’s former choice of xz -2 -T (= %define _binary_payload w2T.xzdio in the spec file) is a better fit for large RPM files (> 100 kbytes), especially for mobile devices. Only for multi-megabyte RPMs I would choose (at most) xz -6 -T (= %define _binary_payload w6T.xzdio), but do understand that Jolla had chosen this as a “one size fits all” setting, because it covers the worst cases well. Note that multithreading for the option -T was introduced in xz 5.2.0, hence it cannot be used for RPMs which may be installed on SailfishOS < 3.2.0!
For small RPMs (< 100 kbytes) and all SRPMs, gzip -9 (= %define _binary_payload w9.gzdio and %define _source_payload w9.gzdio; the latter seems to be the default on most Linux distributions including SailfishOS) is absolutely fine, even its default compression level 6 is almost equivalent (and compresses much faster) and in some cases even 1 (but not 0, because it means “no compression”).

P.S.: On SailfishOS ≥ 4.5.0 Jolla exactly followed RedHat’s / Fedora’s choice of zstd's compression level 19, which is a reasonable as a general default setting, because it covers the worst cases well. I would add (in line with Jolla’s former choice for xz) the parameter -T (= %define _binary_payload w19T.zstdio), which defaults to 0 if -T is set, and to 1 if not (see man-page). For RPMs < 1 Mbyte zstd -10 -T (= %define _binary_payload w10T.zstdio; at most 15) appears to be a better choice. Note that zstd's default at the command line is a compression level parameter of merely 3, but that was likely chosen because of its much faster compression speed, which is much nicer for interactive use (though the compression ratio is lower and decompression slightly slower).
But as the switch to zstd with the backward compatibility issues it causes was the principal point here, xz and / or gzip are the proper choices to obtain backward compatibility.

poetaster · 20 June 2023 18:37

I just wanted to boost this thread again, since for those of us keeping track of installations < 4.4 it’s necessary. Unless we want to stop supporting everything but latest greatest.

You may find your packages being available in harbour but failing to install if you do not specify. It seems that 4.5 + 3.10 SDK yields rpms that won’t install on older SFOS.

olf · 21 June 2023 00:18

Yes.

Ultimately I settled on either adding …

the line
%define _binary_payload w2.xzdio
to a spec file
or, if Spectacle is used, the line
- _binary_payload;w2.xzdio
to the Macros: section of its yaml file

persmule · 27 July 2023 03:43

Hello! I am trying to “zypper up” my sfos running on scorpion-windy within 3.4.0.24, but cannot proceed because many packages are affected by seemingly zstd conpression,

( 1/31) Installing: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl ..........................................................................................................................................................[error]
Installation of droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl failed:
Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed: cpio: Bad magic
error: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl: install failed
error: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.9.jolla.armv7hl: erase skipped

Abort, retry, ignore? [a/r/i] (a):

[nemo@sfos-scorpion-windy ~]$ rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" /var/cache/pk-zypp-cache/packages/adaptation-community/armv7hl/droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl.rpm 
zstd 19

and even if I update rpm to 4.16.1.3 (which should have supported zstd) could not make these packages installable:

[nemo@sfos-scorpion-windy ~]$ LANG=en_US.UTF-8 zypper lu
Loading repository data...
Reading installed packages...
S | Repository                  | Name                                                 | Current Version                                         | Available Version                                       | Arch   
--+-----------------------------+------------------------------------------------------+---------------------------------------------------------+---------------------------------------------------------+--------
v | adaptation-community-common | bluetooth-rfkill-event-hciattach                     | 1.0.9-1.2.8.jolla                                       | 1.0.9-1.2.9.jolla                                       | armv7hl
v | adaptation-community-common | community-adaptation-devel                           | 0.0.3-1.9.6.jolla                                       | 0.0.3-1.9.7.jolla                                       | armv7hl
v | adaptation-community        | droid-hal-version-scorpion_windy                     | 0.0.1+master.20180915104950.5c60ac8-1.11.9.jolla        | 0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla       | armv7hl
v | adaptation-community        | geoclue-provider-hybris-hal                          | 0.2.22-1.5.17.jolla                                     | 0.2.22-1.5.18.jolla                                     | armv7hl
v | native-common               | gpsd                                                 | 3.23+git2-1.8.5.jolla                                   | 3.23+git2-1.8.6.jolla                                   | armv7hl
v | native-common               | gstreamer1.0-vaapi                                   | 1.18.5-1.6.1.jolla                                      | 1.18.5-1.6.3.jolla                                      | armv7hl
v | adaptation-community        | hybris-libsensorfw-qt5-hal                           | 0.11.4-1.6.11.jolla                                     | 0.11.4-1.6.12.jolla                                     | armv7hl
v | native-common               | kernel-headers                                       | 5.4.23-1.2.11.jolla                                     | 5.4.23-1.2.12.jolla                                     | armv7hl
v | native-common               | libdrm                                               | 2.4.110+git1-1.7.4.jolla                                | 2.4.110+git1-1.7.5.jolla                                | armv7hl
v | native-common               | libgps                                               | 3.23+git2-1.8.5.jolla                                   | 3.23+git2-1.8.6.jolla                                   | armv7hl
v | adaptation-community        | libhybris                                            | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libEGL                                     | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libGLESv2                                  | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libOpenCL                                  | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libhardware                                | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libnfc                                     | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libsync                                    | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libvibrator                                | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libwayland-egl                             | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-tests                                      | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | native-common               | libva                                                | 2.6.1+master.20200301232537.48b1b74-1.5.35.jolla        | 2.6.1+master.20200301232537.48b1b74-1.5.37.jolla        | armv7hl
v | adaptation-community-common | mce-plugin-libhybris                                 | 1.14.1-1.5.10.jolla                                     | 1.14.5-1.3.1.jolla                                      | armv7hl
v | native-common               | qt5-qtdeclarative-import-multimedia                  | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia                                     | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-gsttools                            | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-audio-pulseaudio             | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-mediaservice-gstcamerabin    | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-mediaservice-gstmediacapture | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-mediaservice-gstmediaplayer  | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-resourcepolicy-resourceqt    | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qtscenegraph-adaptation                              | 0.7.6-1.3.18.jolla                                      | 0.7.6-1.3.20.jolla                                      | armv7hl

You can see that many shared repositories are affected. Could you have packages in these repositories rebuilt to a compatible format, or is there a way to update rpm and related tools to make zstd-compressed packages installable on old systems?

poetaster · 27 July 2023 08:02

Which port ist that? Sounds like a question for the maintainer, but I’m really uncertain about this since I’ve only had the issue appear with 4.5 and I only deal with the apps I maintain as advised by @olf

@olf, are you aware of an version rpm (pkcon, too?) which might help resolve this?

kan_ibal · 27 July 2023 14:16

Try that rpm.

persmule · 27 July 2023 15:13

Thanks. Upgrade complete by upgrading rpm and rpm-libs to 4.16.1.3+git8-1.10.1 first, as well as related packages. In detail:

$ sudo ssu re 4.4.0.58
$ sudo zypper ref
$ sudo zypper up rpm rpm-libs store-client patchmanager deltarpm zypper sailfish-installationhandler
$ sudo ssu re 3.4.0.24
$ sudo zypper ref
$ sudo zypper up

attah · 27 July 2023 15:32

You are skipping about a gazillion stop releases… no wonder it didn’t work for you.

persmule · 27 July 2023 15:33

But sadly the system became broken after upgrading. The unlock screen cannot kick in after reboot.
I have to restore the system from a backup made with twrp recovery.

persmule · 27 July 2023 15:36

But with packages within common repositories built with incompatible conpression, it is even impossible to update packages under the same release.

attah · 27 July 2023 15:45

I don’t follow what you are saying… Do you understand what i mean by “stop release”?
For the rest of us we got the prerequisite package in 4.X and only in 4.X+1 was the new compression actually used.

persmule · 27 July 2023 15:50

The Issue I met is not related to “stop release”, but what poetaster assumes occurs on packages within common repositories.