SFOS 4.5.0 switches RPM compression. What does that mean for existing RPMs?

General
18

@nephros, do I understand correctly that you are suggesting to put
%define _binary_payload w2.xzdio
in spec files to set the compression to xz -2 (as it might have been before)?

Notes:

  • xz -2 is slow on decompression, but better than bzip2 -9 (w9.bzdio).
  • The only real alternative is to use gzip by %define _binary_payload w9.gzdio (= gzip -9): Slightly worse compression ratio, similar compression speed, but much faster decompression. OTOH, SailfishOS 2.2.1 provides xz / liblzma 5.0.4 (released 22 June 2012), so I assume it has been in SailfishOS since its inception (first releases 0.99.x.y in November 2013).
  • xz's default compression level 6 is very slow, memory- and CPU-intensive for both compression and decompression, and its levels {7,8,9} extremely so (even more than zstd's “ultra”-levels {20,21,22}). See xz's man-page, sections “Memory usage” and “-0 … -9” for details (!).
  • Do not set %define _source_payload w2.xzdio or something similar (as often suggested), because “SRPM payload compression should stay at gzip (there’s almost no benefit in changing the compression, because SRPM’s contents is compressed already)”. This also implies that any %define _<*>_payload w<?>.gzdio continues to work fine (although only 9 makes sense for substituting <?>).

References:

  • Excellent documentation with background information for Fedora 30→31 switching from xz -2 to zstd -19: Changes/Switch RPMs to zstd compression - Fedora Project Wiki
    • How to test an installed package (requires RPM ≥ 4.14.0, which is in SailfishOS since at least v2.2.1):
      rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
    • IMO zstd -15 would have been slightly more desirable: Much faster compression speed, faster decompression, only slightly worse compression ratio; this can be achieved by setting %define _binary_payload w15.zstdio in a spec file for SailfishOS ≥ 4.4.0 (requires the corresponding SDK release for building).
    • BTW, RHEL 8→9 followed this switch, as SailfishOS 4.4.0→4.5.0 does, Ubuntu 22.04 did and Arch Linux did in January 2020.
    • But zstd / libzstd is not available for Fedora < 28, RHEL < 8.2, *buntu < 18.10 etc.; I assume it has been introduced into SailfishOS only recently (v4.4.0?). BTW, Linux integrates zstd for kernel space users (no, this does not address persons :wink:) since 4.14 (November 2017).
    • Using xz -2 or bzip2 -9 compresses only slightly better than gz -9 at roughly the same compression speed but much slower decompression; but gzip's default 6 is compressing much quicker and usually yields the same compression ratio: Now I remember why I stubbornly kept using tar -cvzf <tarball> and never switched to anything else, also for compatibility’s sake. This is also the reason why Debian 9 / Ubuntu 18.04 started using xz / lzma compression for packages and used gzip prior to that. OpenWRT and other embedded Linux distributions continue to use gzip compression for their packages due to its lightness (WRT CPU, memory and decompression speed).
  • Need new option to control RPM compression level · Issue #1675 · jordansissel/fpm · GitHub
  • 1715799 – Re-enable zstd compression support in RHEL-8 rpm
  • https://github.com/eclipse-archived/packagedrone/pull/133#discussion_r228995829
  • Historic (on RHEL 4), but informative: rpm - rpmbuild change compression format - Stack Overflow

Can someone please try to install an RPM package built with the Sailfish-SDK for 4.5.0 (or the sailfishos-4.5.0 DoD repository at the SailfishOS-OBS) on SailfishOS ≤ 4.3.0, to ensure that nothing else changed. It must emit at least rpmlib(PayloadIsZst) is needed by <package name>-<version>-<release>.<arch>, but any other rpmlib(…) is needed by <…> would be something to also take into account.
Background: With the switch from w9.gzdio to w2.xzdio in RHEL 5→6, also the filedigest was altered from md5 (to sha1, IIRC), see here (it is historic as for RHEL 5→6, the write-up is erratic, the suggestions flawed, it is just to provide a reference and evidence what happened).

2 Likes