SFOS 4.5.0 switches RPM compression. What does that mean for existing RPMs?

Did a few tests with both, “noarch” and true binary RPM files and SRPM files (….src.rpm) build by rpmbuild and checked by rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>

Built on /
with
binary RPM file noarch RPM file SRPM file
SFOS 2.2.1
rpmbuild
xz 6 gzip 9
SFOS 3.1.0
SFOS-OBS
xz 6 xz 6
SFOS 3.2.1
rpmbuild
gzip 9 gzip 9
SFOS 3.2.1
SFOS-OBS
xz 6T xz 6T
SFOS 3.4.0
SFOS-OBS
xz 6T xz 6T
SFOS 3.4.0
Sailfish‑SDK   
xz 6T xz 6T
SFOS 4.0.1
Sailfish‑SDK
xz 6T xz 6T
SFOS 4.4.0
SFOS-OBS
xz 6T xz 6T
SFOS 4.4.0
Sailfish‑SDK
xz 6T xz 6T
SFOS 4.5.0
SFOS-OBS
zstd 19 zstd 19

Contribution are welcome, specifically information about binary RPMs built locally (i.e., on a SailfishOS device) by rpmbuild.

3 Likes

No I’m not suggesting anything, but the information from @adekker above is important for packagers who provide packges for older SFOS versions.

I guess setting the compression scheme back to the previous one is a feasible approach for such packages.

Well, after researching the topic RPM compression much more in-depth than originally intended, I can come up a conclusive suggestion.

but the information from @adekker above is important for packagers who provide packges for older SFOS versions.

I do not see that anybody questioned that, rather the opposite: Jolla’s and @dekker’s information, plus your “non-suggestion” made me research and document this.

I guess setting the compression scheme back to the previous one is a feasible approach for such packages.

Well, “the previous one” is not really conclusive (see my prior post), and I do think that RedHat’s / Fedora’s former choice of xz -2 -T (= %define _binary_payload w2T.xzdio in the spec file) is a better fit for large RPM files (> 100 kbytes), especially for mobile devices. Only for multi-megabyte RPMs I would choose (at most) xz -6 -T (= %define _binary_payload w6T.xzdio), but do understand that Jolla had chosen this as a “one size fits all” setting, because it covers the worst cases well. Note that multithreading for the option -T was introduced in xz 5.2.0, hence it cannot be used for RPMs which may be installed on SailfishOS < 3.2.0!
For small RPMs (< 100 kbytes) and all SRPMs, gzip -9 (= %define _binary_payload w9.gzdio and %define _source_payload w9.gzdio; the latter seems to be the default on most Linux distributions including SailfishOS) is absolutely fine, even its default compression level 6 is almost equivalent (and compresses much faster) and in some cases even 1 (but not 0, because it means “no compression”).

P.S.: On SailfishOS ≥ 4.5.0 Jolla exactly followed RedHat’s / Fedora’s choice of zstd's compression level 19, which is a reasonable as a general default setting, because it covers the worst cases well. I would add (in line with Jolla’s former choice for xz) the parameter -T (= %define _binary_payload w19T.zstdio), which defaults to 0 if -T is set, and to 1 if not (see man-page). For RPMs < 1 Mbyte zstd -10 -T (= %define _binary_payload w10T.zstdio; at most 15) appears to be a better choice. Note that zstd's default at the command line is a compression level parameter of merely 3, but that was likely chosen because of its much faster compression speed, which is much nicer for interactive use (though the compression ratio is lower and decompression slightly slower).
But as the switch to zstd with the backward compatibility issues it causes was the principal point here, xz and / or gzip are the proper choices to obtain backward compatibility.

I just wanted to boost this thread again, since for those of us keeping track of installations < 4.4 it’s necessary. Unless we want to stop supporting everything but latest greatest.

You may find your packages being available in harbour but failing to install if you do not specify. It seems that 4.5 + 3.10 SDK yields rpms that won’t install on older SFOS.

1 Like

Yes.

Ultimately I settled on either adding …

  • the line
    %define _binary_payload w2.xzdio
    to a spec file
  • or, if Spectacle is used, the line
    - _binary_payload;w2.xzdio
    to the Macros: section of its yaml file
1 Like

Hello! I am trying to “zypper up” my sfos running on scorpion-windy within 3.4.0.24, but cannot proceed because many packages are affected by seemingly zstd conpression,

( 1/31) Installing: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl ..........................................................................................................................................................[error]
Installation of droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl failed:
Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed: cpio: Bad magic
error: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl: install failed
error: droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.9.jolla.armv7hl: erase skipped

Abort, retry, ignore? [a/r/i] (a):

[nemo@sfos-scorpion-windy ~]$ rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" /var/cache/pk-zypp-cache/packages/adaptation-community/armv7hl/droid-hal-version-scorpion_windy-0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla.armv7hl.rpm 
zstd 19

and even if I update rpm to 4.16.1.3 (which should have supported zstd) could not make these packages installable:

[nemo@sfos-scorpion-windy ~]$ LANG=en_US.UTF-8 zypper lu
Loading repository data...
Reading installed packages...
S | Repository                  | Name                                                 | Current Version                                         | Available Version                                       | Arch   
--+-----------------------------+------------------------------------------------------+---------------------------------------------------------+---------------------------------------------------------+--------
v | adaptation-community-common | bluetooth-rfkill-event-hciattach                     | 1.0.9-1.2.8.jolla                                       | 1.0.9-1.2.9.jolla                                       | armv7hl
v | adaptation-community-common | community-adaptation-devel                           | 0.0.3-1.9.6.jolla                                       | 0.0.3-1.9.7.jolla                                       | armv7hl
v | adaptation-community        | droid-hal-version-scorpion_windy                     | 0.0.1+master.20180915104950.5c60ac8-1.11.9.jolla        | 0.0.1+master.20180915104950.5c60ac8-1.11.10.jolla       | armv7hl
v | adaptation-community        | geoclue-provider-hybris-hal                          | 0.2.22-1.5.17.jolla                                     | 0.2.22-1.5.18.jolla                                     | armv7hl
v | native-common               | gpsd                                                 | 3.23+git2-1.8.5.jolla                                   | 3.23+git2-1.8.6.jolla                                   | armv7hl
v | native-common               | gstreamer1.0-vaapi                                   | 1.18.5-1.6.1.jolla                                      | 1.18.5-1.6.3.jolla                                      | armv7hl
v | adaptation-community        | hybris-libsensorfw-qt5-hal                           | 0.11.4-1.6.11.jolla                                     | 0.11.4-1.6.12.jolla                                     | armv7hl
v | native-common               | kernel-headers                                       | 5.4.23-1.2.11.jolla                                     | 5.4.23-1.2.12.jolla                                     | armv7hl
v | native-common               | libdrm                                               | 2.4.110+git1-1.7.4.jolla                                | 2.4.110+git1-1.7.5.jolla                                | armv7hl
v | native-common               | libgps                                               | 3.23+git2-1.8.5.jolla                                   | 3.23+git2-1.8.6.jolla                                   | armv7hl
v | adaptation-community        | libhybris                                            | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libEGL                                     | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libGLESv2                                  | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libOpenCL                                  | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libhardware                                | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libnfc                                     | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libsync                                    | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libvibrator                                | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-libwayland-egl                             | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | adaptation-community        | libhybris-tests                                      | 0.0.5.33.tama.1-1.6.8.jolla                             | 0.0.5.33.tama.1-1.6.9.jolla                             | armv7hl
v | native-common               | libva                                                | 2.6.1+master.20200301232537.48b1b74-1.5.35.jolla        | 2.6.1+master.20200301232537.48b1b74-1.5.37.jolla        | armv7hl
v | adaptation-community-common | mce-plugin-libhybris                                 | 1.14.1-1.5.10.jolla                                     | 1.14.5-1.3.1.jolla                                      | armv7hl
v | native-common               | qt5-qtdeclarative-import-multimedia                  | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia                                     | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-gsttools                            | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-audio-pulseaudio             | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-mediaservice-gstcamerabin    | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-mediaservice-gstmediacapture | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-mediaservice-gstmediaplayer  | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qt5-qtmultimedia-plugin-resourcepolicy-resourceqt    | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.35.jolla | 5.6.2+git19+v4l.20201017112737.1.gf3a60a78-1.3.37.jolla | armv7hl
v | native-common               | qtscenegraph-adaptation                              | 0.7.6-1.3.18.jolla                                      | 0.7.6-1.3.20.jolla                                      | armv7hl

You can see that many shared repositories are affected. Could you have packages in these repositories rebuilt to a compatible format, or is there a way to update rpm and related tools to make zstd-compressed packages installable on old systems?

Which port ist that? Sounds like a question for the maintainer, but I’m really uncertain about this since I’ve only had the issue appear with 4.5 and I only deal with the apps I maintain as advised by @olf

@olf, are you aware of an version rpm (pkcon, too?) which might help resolve this?

1 Like

Try that rpm.

1 Like

Thanks. Upgrade complete by upgrading rpm and rpm-libs to 4.16.1.3+git8-1.10.1 first, as well as related packages. In detail:

$ sudo ssu re 4.4.0.58
$ sudo zypper ref
$ sudo zypper up rpm rpm-libs store-client patchmanager deltarpm zypper sailfish-installationhandler
$ sudo ssu re 3.4.0.24
$ sudo zypper ref
$ sudo zypper up
1 Like

You are skipping about a gazillion stop releases… no wonder it didn’t work for you.

3 Likes

But sadly the system became broken after upgrading. The unlock screen cannot kick in after reboot.
I have to restore the system from a backup made with twrp recovery.

But with packages within common repositories built with incompatible conpression, it is even impossible to update packages under the same release.

I don’t follow what you are saying… Do you understand what i mean by “stop release”?
For the rest of us we got the prerequisite package in 4.X and only in 4.X+1 was the new compression actually used.

The Issue I met is not related to “stop release”, but what poetaster assumes occurs on packages within common repositories.

But it is, you’re trying to skip a release that introduces new compression method, skipping a stop release doesn’t work

2 Likes

I am not trying to skip a release that introduces new compression method. I am just trying to update packages within a release before introducing new compression method (3.4.0.24), but the packages going to install are built with newly introduced compression method not supported by old system.

I believe the underlying problem is that repos like these:

nemo:devel:hw:native-common
nemo:devel:hw:sony:scorpion_windy

only have one set of build versions defined, i.e. “latest”, and BUILDFLAG enabled so will always (re)build packages incompatible with anything between the initial release of the port and “latest”.

3 Likes

Porters can use hw-common from the specific Sailfish OS release, they just don’t define hw-common in their ssu config, doing so will ensure that they use hw-common from Sailfish OS as defined in the Jolla ssu config.

For the device repository it is also use the release version. For example as I did here:

This ssu-config can be reused by other porters for this intend:

3 Likes

Please stay on topic and create a new thread for this topic.

@nephros had already answered the question above.