Compiling Rust 1.75.0 in Platform SDK

[direc85]

Hello everybody!

Rust 1.75 was merged to master March 15th 2024, so the next Sailfish SDK release should include it! Until then you can compile it yourself (or download it from my repository).

With this post you can build your own Rust 1.75.0 in the Platform SDK and install the packages in Sailfish SDK. You can then compile applications with it with a small workaround, as it you will see near end of this post. I’ll update this post if/when the repos change. Once the next SDK with updated Rust (and its dependencies) is out, I’ll post those instructions in a reply!

Note that all commands (except step 1) is done in the Sailfish Platform SDK prompt - not in tooling, not in any target. This goes for the package installation, too. This is a rare occurrence, but in this case it is necessary. This is because when the compilation reaches the part of python3 x.py build which again executes a highly specific variant of cargo build, it just stops/hangs with no activity whatsoever. If anyone has any ideas on how to fix that, please let us know!

Edit 1: Fix zyper typo, add link to rainemak’s blog post, fix wrong assumtions about proc-macro2 compilation error.
Edit 2: Bump to Rust 1.75, more typo fixes, use my own repository for LLVM 15, condense commands
Edit 3: Fixed and clarified LLVM version numbers
Edit 4: Removed the TMPDIR workaround (“fixed” in the PR), fix a formatting error
Edit 5: Use sailfishos/rust master instead of Rust 1.75 pull request by @rubdos

Step 1: Get Rust sources

Install git-lfs and clone sailfishos/rust:

$ git clone https://github.com/sailfishos/rust-cross.git
$ git clone https://github.com/sailfishos/rust.git
# Pro tip: Install git-lfs now to prevent downloading Rust 1.52 tarball
$ cd rust
$ git checkout 1.75.0+git1 

Next, enter the Sailfish SDK:

$ sfossdk

Step 2: Install LLVM 16 and Clang 16 from my repository. The Sailfish repository is at version 14 at the time of writing, which is what Piggz’s OBS repositories currently offer. Rust 1.75 requires LLVM 15, however.

SDK $ sudo zypper ar --no-gpgcheck https://direc.kapsi.fi/sailfish-repo/i486 direc85
SDK $ sudo zypper ref
SDK $ sudo zypper in --force --from direc85 cmake llvm llvm-devel llvm-libs clang clang-devel clang-libs

Step 3: Install the cross compilation packages

Then a few more repos are needed in the Sailfish SDK prompt to install all the cross packages from. These instructions are adopted from @rainemak’s detailed post:

SDK $ sudo ssu ar jolla-aarch64 https://releases.jolla.com/releases/4.5.0.18/jolla/aarch64
SDK $ sudo ssu ar jolla-armv7hl https://releases.jolla.com/releases/4.5.0.18/jolla/armv7hl
SDK $ sudo zypper ref
SDK $ sudo zypper in cross-aarch64-gcc cross-aarch64-binutils cross-aarch64-as cross-aarch64-glibc cross-aarch64-glibc-devel cross-aarch64-glibc-headers cross-aarch64-kernel-headers
SDK $ sudo zypper in cross-armv7hl-gcc cross-armv7hl-binutils cross-armv7hl-as cross-armv7hl-glibc cross-armv7hl-glibc-devel cross-armv7hl-glibc-headers cross-armv7hl-kernel-headers

Step 4: Install rest of the build time dependencies

SDK $ sudo zypper in ccache cmake gdb libffi-devel make ncurses-devel "pkgconfig(libcurl)" "pkgconfig(liblzma)" "pkgconfig(openssl)" "pkgconfig(libz)" fakeroot

Step 5: Build Rust

This is pretty much just building a dumb package. Let’s build the actual compiler first - this takes 36 minutes on my laptop:

SDK $ cd ~/rust
SDK $ for ARCH in i486 armv7hl aarch64
  do
    rpmbuild --define "_topdir $PWD" --define "_sourcedir $PWD" --define "_target_cpu $ARCH" -bb rust.spec
  done

Note that the i486 compiles “properly” and the aarch64 and armv7hl architectures build stubs.

Step 6: Install the new Rust

SDK $ cd i486
SDK $ sudo zypper in *.rpm

Step 7: Create the Rust lib cross packages

SDK $ cd ~/rust-cross
SDK $ for ARCH in armv7hl aarch64
  do
    rpmbuild --define "_topdir $PWD" --define "_sourcedir $PWD" --define "_target_cpu $ARCH" -bb rust-cross-$ARCH.spec
  done

Step 8: Make the packages available in Sailfish SDK

Note that I’m not adding my repository here, because it would also provide Rust 1.75 packages - which is not what we want in this case.

Exit the Sailfish SDK to your everyday shell. My project folder in SFDK is ~/SFOS, so adjust accordingly. You also need to have a Sailfish application project, I’ll use Whisperfish here (as it’s a Rust application).

$ mkdir -p ~/SFOS/RPMS
$ cd ~/SFOS
$ git clone https://gitlab.com/whisperfish/whisperfish.git
$ cd whisperfish
$ sfdk config --global --push output-prefix ~/SFOS/RPMS

Next, move the packages you compiled to the “local repository”:

$ for ARCH in i486 armv7hl aarch64
  do
    mkdir -p ~/SFOS/RPMS/SailfishOS-4.5.0.18-$ARCH
    mv ~/rust/RPMS/$ARCH/* ~/SFOS/RPMS/SailfishOS-4.5.0.18-$ARCH/
  done

We’ll also need to download the necessary LLVM packages:

$ for ARCH in i486 armv7hl aarch64
  do
    URL=https://direc.kapsi.fi/sailfish-repo/$ARCH/rpm
    for F in clang-libs-16.0.6+git1-0.$ARCH.rpm clang-16.0.6+git1-0.$ARCH.rpm clang-libs-16.0.6+git1-0.$ARCH.rpm
    do
      curl $URL/$F --output $F
    done
  done

Step 9: Install the Rust and LLVM packages to SFDK tooling

Since the tooling can’t access the host file system, you need to put the files to a web server somewhere (either on your LAN or on the Internet), upload the packages there, and download them back. I think this can be achieved on localhost too, but as I have a server available, I just used that. I use my own compilation results below, but you should use your own server here:

Download and install the necessary LLVM and Rust packages to the SFDK tooling:

$ sfdk exec SailfishOS-4.5.0.18 bash
# cd ~
# sfdk tools tooling exec SailfishOS-4.5.0.18 bash
# URL=https://direc.kapsi.fi/sailfish-repo/i486/rpm
# for F in rust-1.75.0+git1-1.i486.rpm cargo-1.75.0+git1-1.i486.rpm rust-std-static-aarch64-unknown-linux-gnu-1.75.0+git1-1.i486.rpm rust-std-static-armv7-unknown-linux-gnueabihf-1.75.0+git1-1.i486.rpm rust-std-static-i686-unknown-linux-gnu-1.75.0+git1-1.i486.rpm llvm-libs-16.0.6+git1-0.i486.rpm clang-libs-16.0.6+git1-0.i486.rpm clang-16.0.6+git1-0.i486.rpm
  do
    curl $URL/$F --output $F
  done
# zypper in *.rpm

Step 10: Compile a package with the new Rust

Finally!

Go to the Sailfish application repo, set the target, and compile away!

Note: If you have compiled with another Rust version before, you must run cargo clean first!

$ cd ~/SFOS/whisperfish
$ for ARCH in i486 armv7hl aarch64
  do
    sfdk config --push target SailfishOS-4.5.0.18-$ARCH
    sfdk build-shell cargo clean
    sfdk build
  done

---

As of February 19 2024, the .spec workaround is not needed any longer, because the issue with temporary files was worked around in the upcoming Rust 1.75 pull request instead. This means the .spec workaround still works, but is not needed nor recommended. This also means that the underlying issue in scratchbox2 is still there - see earlier versions of this post for the details.

---

Thanks to everyone involved in this rather massive Rust updating project: @rubdos, @mal, @rainemak, @vige and everyone else I forgot to mention here! This has been a huge undertaking and would not have happened without the help of many sailors and community members!

[rubdos]

@direc85

(it just wanted to install the version 10 and not version 14, I don’t know why)

--allow-vendor-change, maybe?

@direc85

Then a few more repos are needed in the Sailfish SDK prompt to install all the cross packages from. These instructions are adopted from @rainemak’s detailed post:

Oooooh. Thanks for the summary! This makes a lot of sense now. Permalink to @rainemak’s post.

FYI, you have a typo on the paragraph (zyper instead of zypper on the last zyper in.

I’m using your hints to get Rust 1.75 to compile now. That thing has been released yesterday

[rubdos]

@direc85

I believe the proc-macro2 build command should not have --target i686-unknown-linux-gnu -C linker=i686-unknown-linux-gnu-gcc in it but --target armv7-unknown-linux-gnueabihf -C linker=armv7hl-meego-linux-gnueabi-gcc since if I take the cargo build command above and modify it accordingly, the crate compiles successfully. According to my research, this would be correct in the “normal” cross compilation case, but as we’re effectively running in scratchbox which does all kinda magic here, we need to make the value of SB2_RUST_TARGET_TRIPLE propagate further.

I do not necessarily agree, here! I would expect proc_macro2 to usually be a build-time dependency, which means it gets executed in the tooling, and thus should indeed be an i686 executable/host executable. So I think the build command might even be correct… but your conclusion might still be right, since it’s now trying to cross-compile for i686, while it should probably use a native compilation target, or something along those lines.

[direc85]

@rubdos

--allow-vendor-change , maybe?

I did try that, and --force too, but I just wasn’t able to make Zypper pick packages from that repository. If someone knows how to make that happen, it would make the process a lot less hairy

@rubdos

Permalink to @rainemak’s post.

I forgot the link, thanks for nudging! I’ll put it in, as well as fix the zyper typo.

@rubdos

I would expect proc_macro2 to usually be a build-time dependency, which means it gets executed in the tooling, and thus should indeed be an i686 executable/host executable.

I stand corrected. I was confident that I double checked that Whisperfish CI uses the target architecture on that, but that is not the case (tip: click “Complete Raw” and search for proc-macro2). I’ll edit the post accordingly. So: the issue seems to really be that the expected file is not in /tmp/. I believe mal also hit this same issue, so I’ll direct my efforts into figuring out which part of the compilation is supposed to create the file and why it’s not there.

This also means I don’t have to hack-patch Rust source code any more, at least not in the same sense, as it appears to be correct!

Edit: Yeah, Rust 1.75 is now out! I looked at it briefly, and yeah, it looks to pack quite a punch!

Edit 2: Fixed link URL

[direc85]

Good news! I finally managed to make Rust 1.75 compile Whisperfish for all three architectures! The “fix” sounds quite blunt - just don’t try linking the symbols.o file since it doesn’t exist - but it was effective! See this pull request for details:

Don't export symbols or synthetic object files in Scratchbox

The longer story is that the symbols.o (and a few others) are not created for all targets, so this is not as invasive as one could think. For whatever reason there’s a mismatch in the detection in Sailfish SDK / Scratchbox environments. There may be better ways to fix the issue, but this one seems to do the trick…

The change that broke the behaviour in Scratchbox seems to have been released in Rust 1.62:

Generate synthetic object file to ensure all exported and used symbols participate in the linking

Edit: I update the post and title to build Rust 1.75 instead, but that will have to wait until tomorrow!

[direc85]

Okay, sometimes the solution is the simplest one possible. I reverted the patch above (0009) and put this in harbour-whisperfish.spec instead (in %build):

export TMPDIR=$(realpath "$RPM_BUILD_DIR/../.tmp") 
mkdir -p $TMPDIR

And now the “unpatched” Rust 1.75 is compiling the aarch64 build happily as I speak! (I double checked that by commenting those two lines out, and the compilation fails.)

[tortoisedoc]

Hmmm

`std::env::temp_dir` should return env var XDG_RUNTIME_DIR under Linux if declared

bug in the env / missing env var (TMPDIR or XDG_RUNTIME_DIR missing?)

[direc85]

Rust currently (and by the looks in the future too) only uses $TMPDIR and falls back to /tmp which works fine outside SB2. I think we’ll have to patch Rust to fix that instead, I have an idea on how to do that…

[tortoisedoc]

Hmm hold your horses;
there’s a /tmp in sb2 no?

[direc85]

Yes there is, but as I discussed with @rubdos, it’s possible that the /tmp it’s using is in the tooling and not the target. Or, perhaps the /tmp mount is created when the process starts and destroyed when the process ends. Something like that, maybe, perhaps. Those would pretty much explain why the files get created just fine by the compiler, but don’t exist for the linker any more…

[tortoisedoc]

Hmm yeah I recall similar issues on the x86_64 target. Wouldnt it make more sense to export the XDG_RUNTIME_DIR instead in the target?

[direc85]

Update: Rubdos’ Rust fork/PR no longer requires the .spec TMPFILE workaround, so step 11 is no longer required.

I’d have edited the post but it wasn’t possible any longer. Could someone turn that into a wiki post, please?

[vige]

Done, it’s a wiki now.

[pherjung]

@direc85

git checkout rust-1.75.0

Is it possible that’s in reality the following?

git checkout origin/1.75.0

[pherjung]

@direc85

Step 4: Install rest of the build time dependencies

I get an error with this step:

[…]
'pkgconfig(libz)' not found in package names. Trying capabilities.
No provider of 'pkgconfig(libz)' found.

And later I get these errors:

[pherjung@bernina sailfish-rust]$ for ARCH in i486 armv7hl aarch64                                                                                                          
> do                                                                                                                                                                        
>     rpmbuild --define "_topdir $PWD" --define "_sourcedir $PWD" --define "_target_cpu $ARCH" -bb rust.spec                                                                
> done                                                                                                                                                                      
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.npP3up                                                                                                                        
+ umask 022                                                                                                                                                                 
+ cd /home/pherjung/projets/sfos/librespot/sailfish-rust/BUILD                                                                                                              
+ cd /home/pherjung/projets/sfos/librespot/sailfish-rust/BUILD                                                                                                              
+ rm -rf rust-1.75.0-i686-unknown-linux-gnu                                                                                                                                 
+ /bin/tar -xof /home/pherjung/projets/sfos/librespot/sailfish-rust/rust-1.75.0-i686-unknown-linux-gnu.tar.gz                                                               
/bin/tar: This does not look like a tar archive                                                                                                                             
                                                                                                                                                                            
gzip: stdin: not in gzip format                                                                                                                                             
/bin/tar: Child returned status 1                                                                                                                                           
/bin/tar: Error is not recoverable: exiting now                                                                                                                             
error: Bad exit status from /var/tmp/rpm-tmp.npP3up (%prep)                                                                                                                 
                                                                                                                                                                            
                                                                                                                                                                            
RPM build errors:                                                                                                                                                           
    Bad exit status from /var/tmp/rpm-tmp.npP3up (%prep)                                                                                                                    
warning: Macro expanded in comment on line 535: %version dependency for external deployment but the cross-                                                                  
[…]

Won’t be this night I’ll get librespot compiled

[direc85]

Cloning sailfishos/rust should now work indeed, I’ll have to update the post again!

Not sure what’s up with the missing libz dependency… I’ll have a look tonight.