Automated build system to handle multiple packages

Platform Development
#1

As we have OBS stalling on updates and no aarch64 support on it, I am looking for ways to compile and maintain applications that are so far supported by OBS (maps stack, flatpak, and few others).

In principle, without going into corner cases, it seems that requirements are rather simple:

  • all should be running on a single PC, no server clusters needed
  • allow to organize linked projects in some manner. for example, having a folder with RPM SPEC links to your SPECs in src folders
  • detect build dependencies of the packages
  • build packages respecting the dependencies
  • support for multiple arch’es

It seems to me that combining rpmspec, sfdk, make, symlink processing, and having some simple config file should allow to hack together with relatively small effort.

Few questions:

  • is anyone aware of such tool?
  • Jolla was earlier suggesting that they look for solutions to the same problem, if I understood their responses regarding OBS correctly. Any updates or sources to look at/to use?
3 Likes
Sfdk/mb2 handling of shared output-dir
#2

Sources of inspiration:

and

https://pagure.io/koji/

1 Like
#3

Thanks!

It looks like Mock will generate chroots, something that is probably not needed and, I suspect, will interfere with sfdk.

As for Koji, it looks like it is some kind of larger framework with build machines involved.

#4

Indeed we are looking for possible solutions. And it seems we have come to pretty much same conclusion about the requirements than what you have listed above. We have some ideas about possible solutions, but that’s about it. I guess I could say “all ideas are welcome”.

5 Likes
#5

Thank you for inside-info. Pity that it hasn’t moved much yet nor community OBS has not picked up aarch64.

Unless something comes out, I would probably hack something together using the tools above and python-rpm-spec to generate Makefile. If you have some tool missing in the list that maybe useful, please let me know.

1 Like
#6

I wrote a small tool for building such projects: https://github.com/rinigus/tbuilder

This tool allows you to build multiple RPM SPECs while parsing and finding the dependencies through rpmspec. Unfortunately, it has limitations - rpmspec doesn’t determine all provided symbols (pkg-config(libname) for example) and dependency resolution powers are significantly impacted. There are manual workarounds for now, but I don’t think it is a long term solution to have them.

The above limitation is filed as https://github.com/rinigus/tbuilder/issues/1 . Limitation discussed on sfos-porters channel and picked up by Thaodan as well.

Bugs experienced:

  • Sfdk does not handle rpmbuild options properly

  • several packages refused to build in AppSDK, but were fine when using OBS and PlatformSDK. They seem to be hitting sb2 limitations and among them are:

    • collectd and flatpak/ostree. These projects are using bison and while running, bison is starts m4 as a process. That m4 was started out of sb2 env and was missing in AppSDK. “Cure” was to install bison into the AppSDK main environment as in docker exec sailfish-os-build-engine zypper --non-interactive in bison

    • flatpak/libappstream-glib - complains about missing lib while it is installed and required in the target. I suspect that somehow it is managing to escape sb2. Edit: seems like issue is limited to x86 target, aarch64 (me) and armv7hl (elros1) managed to build just fine. elros1 was also able to build x86 target without issues. Why it failed for me, I don’t know.

As PlatformSDK seems to be much more stable for some reason, would be great to have an option to communicate with it via sfdk or similar tool. Even if it requires starting some daemon in it manually.

Edit: added note regarding libappstream-glib and used arch; + reproducibility

2 Likes
#7

This is a known bug which should be fixed in the next SDK release.

1 Like
#8

The build system has been updated. Now it is able to resolve build requirements and build the collection of RPMs in required order. For that, it is querying target packages via zypper and is able to distinguish requirements provided by the target and local packages. Knowing that, it is building the packages step-by-step as build requirements are resolved by local packages.

So far, I managed to build maps apps, flatpak support, and presage-based keyboard predictions using this approach. This has exposed several corner cases that are fixed now.

Will continue with the port for Sony Tama devices and see what is needed in addition to support ports.

All in all, it looks like it will be able to fulfill my needs and should be helpful for others not able to access OBS with required targets (aarch64 and maybe some other conditions).

Links:

3 Likes
#9

That looks awesome! I do have one improvement idea though: Perhaps you could use createrepo_c, rpm and rpmspec from build engine instead, in order to minimize outside dependencies?

2 Likes
#10

Good idea - I will look into it

#11

This is to announce that the automated builder system is ready for testing by others.

I have implemented the suggestion of @vige regarding use of createrepo_c, rpm, rpmspec from build engine. Actually, it was required to handle some packages in the device port due to the way SPEC files are written and use installed packages to derive required versions (kernel and pulseaudio).

Using TBuilder, I have managed to build automatically

  • Sony Tama device AOSP10/aarch64 based port packages
  • maps stack (Pure Maps and OSM Scout Server)
  • Flatpak support
  • SystemDataScope and collectd

For use with the ports, see example project at https://github.com/sailfishos-sony-tama/tbuilder-project

Edit, 6 days later: Marking this as a solution as it does what I expected from such system. Recent changes include keeping only latest built RPMs and incrementing release suffix.

5 Likes