Mb2 shared output directory: allow to disable zypper distribution update

[rinigus]

Indeed, contamination between the builds can be prevented with snapshots. And it is a great feature.

However, the issue experienced here is within a build of a single package. So, in my case, while building droid-hal-version-h8324:

• zypper dup pulls in droid-config-XXXX and, as a result, droid-system-apollo and droid-system-apollo-h8314. The latter was by chance and, in theory, same could be provided by droid-system-apollo-h8314 (pay attention to the number behind h!)

• zypper install of dependencies on the line after that in mb2 script finds out that actually droid-system-apollo-h8324 is needed and that conflicts with installed h8314 system-apollo package.

This would all be prevented if zypper dup would be absent. Hence my request.

[martyone]

@rinigus

pulled into the target one line below.

Ah now I see! The devil is in the details

Before I dig deeper, let me ask, are you sure the way things are packaged here is optimal? Couldn’t this be possibly fixed by different packaging strategy?

Imagine the output directory was a regular repository. It is absolutely valid to do a zypper dup prior to zypper in from a regular repository. So it makes me think there is likely a more correct solution. I am not saying that it is necessarily an issue with your packaging! This is just the easiest idea to start with

[rinigus]

The packaging follows the same approach as in https://build.merproject.org/project/show/nemo:devel:hw:sony:tama . It allows to support all 6 devices from Sony Tama and, in this respect, is “optimal”. As a bonus, it exposes issues in the build systems when I work on automating the builds using SDK.

I agree that it is valid to make zypper dup from regular repository. However, it is also valid to ask not to do that, which is my request.

Also regular repository can have packages providing the same symbols and in conflict with each other. As long as they are not pulled in as build requirements at the same time, it is fine. In these cases, zypper dup will be of more harm than use.

I could even suggest that use of zypper dup in this context is unexpected and should not be enabled by default. This is as the requirements should be set by zypper in as it is for regular builds.

[martyone]

What I am trying to avoid is:

“This is FooBar. You can build it simply with mb2/sfdk, just do not ever forget to pass these magic options…”

I.e., avoid surprises where possible. That’s why I want to be sure that adding the option is the right solution to the right problem.

I could even suggest that use of zypper dup in this context is unexpected and should not be enabled by default. This is as the requirements should be set by zypper in as it is for regular builds.

This might be actually the right direction. Maybe the zypper dup could be completely removed. But I am afraid it wasn’t added there just for fun. It’s likely there to deal with package versions moving back and forth, where the sole zypper in would fail.

Maybe the (equivalent of) zypper dup could be done some more clever way, so that it does pick randomly where multiple options exist. OBS also deals with similar issues https://en.opensuse.org/openSUSE:Build_Service_prjconf#Prefer. Ideas?

[rinigus]

Yes, it makes sense not to overload with options.

Re Prefer: In context of automated builds using tbuilder, the solution that works for range of projects is rather radical:

1. use clean SDK target and make/force reset snapshot
2. install packages requested by user into that snapshot. those are installed even if not in BuildRequires
3. build a single package. This includes installation of BuildRequires
4. go to 1

Here, step 2 corresponds to OBS Prefer. It is not exactly the same, but seems to be sufficient for the cases I have tested so far. I haven’t implemented “-package”, let’s see if it is needed.

[martyone]

Could you please try whether doing zypper in also prior to zypper dup (ignoring possible failure) prevents the issue?

diff --git a/sdk-setup/src/mb2 b/sdk-setup/src/mb2
index dd793b2..f3ba2ee 100755
--- a/sdk-setup/src/mb2
+++ b/sdk-setup/src/mb2
@@ -1032,6 +1032,7 @@ maybe_verify_target_dependencies() {
             filter_repo "$exclude_source_name" "$OPT_OUTPUTDIR" "$filtered_outputdir"
             local args=(--"$OPT_SEARCH_OUTPUTDIR_VERBOSITY" --non-interactive \
                 --plus-repo "$filtered_outputdir" --no-gpg-checks)
+            sb2 -t "$OPT_TARGET" -m sdk-install -R zypper "${args[@]}" in "${deps[@]}"
             sb2 -t "$OPT_TARGET" -m sdk-install -R zypper "${args[@]}" dup --from '~plus-repo-1'
             sb2 -t "$OPT_TARGET" -m sdk-install -R zypper "${args[@]}" in "${deps[@]}"
         else

[rinigus]

I will be able to check in few hours.

I think it will work for my case. But then there is a question regarding on why not to allow disabling it? Downgrade is the only reason I can see in using dup with the downgrade of some package installed by default in the base target, but that should be rare. Proper use of snapshots should take care of these downgrades in most of the practical cases.

[martyone]

@rinigus

But then there is a question regarding on why not to allow disabling it?

There is still a possibility to find a solution that works without tweaking.

[rinigus]

Nice goal. Just from my current troubleshooting, I must say that having zypper dup executed during a build was unexpected. Which made debugging much harder than it would be otherwise. In this respect, I would have dropped dup and let developers run it if needed for their projects.

PS: but it is OK to have different opinions, I will test that solution as well and will report back

[martyone]

The dup happens when the --search-output-dir option is active. The option exists to save developers from the need to maintain their build targets manually - all build time dependencies, including those built locally, are installed automatically. For that to work reliably (=no manual intervention is needed) the dup is needed.

I found a possible source of your issues. https://github.com/rinigus/tbuilder/blob/master/tbuilder#L129 By passing an argument to the --snapshot option you forces it to reuse single snapshot for various packages. I didn’t check further what the scope of self.project is, but with the argument passed to --snapshot it definitely does not use a dedicated snapshot for each build.

Please try again with original mb2 and --snapshot without argument.

In this case it will use single snapshot for single build directory. It will use a pool of snapshots and recycle them but never use a dirty snapshot for different build directories. (Pool size is currently hardcoded as SNAPSHOT_POOL_SIZE in mb2 sources.)

[martyone]

To avoid confusion: When a build directory is removed and created again (or just cleared the .mb2 subdirectory), the effect is the same as using a different build directory. So it is OK to use the same path for different packages. It is only required to clear the build directory before proceeding with different package.

[rinigus]

@martyone, thanks for looking into the code! It is a bit hidden, but the snapshot is reset before building each package at https://github.com/rinigus/tbuilder/blob/master/tbuilder#L78

tbuilder creates Makefile to keep track between dependencies. After building one package, a non-zero exit is imposed also for successful build to recheck availability of packages, reset the snapshot and start again.

So, while the same name snapshot is used for the project, it should be clean one without any extras installed.

[rinigus]

Should have included into original reply, sorry for splitting into two messages:

Named snapshot allows me to handle package preferences, similar to OBS. Thus, after force reset of the snapshot, the preferred packages are installed (if any) and the build of the next package is started.

See example config at https://github.com/sailfishos-sony-tama/tbuilder-project/blob/hybris-10/config.yaml . These packages are under install keyword. (kernel modules packages will be removed from that list in the next update)

[martyone]

@rinigus

but the snapshot is reset before building each package

My understanding is you do this just to fight the issue with dup and you want to get rid of this.

I.e. that the issue with dup happens unless you reset the snapshot before building each package.

This is not surprising. When you compare it to OBS, it starts every build with a clean target. And this is what you achieve either with temporary snapshot (--snapshot without argument) or with named snapshot (--snapshot with argument) reset prior to each build as you do.

If you need some fixed deps installed in the snapshot, you can

1. create a named snapshot by using sdk-manage directly and
2. run mb2 passing the named snapshot as --target and --snapshot without argument in addition to that

$ sdk-assistant list
sf
├─sfarm
│ └─sfarm.mb2.YRN  (snapshot)
└─sfx86
  ├─sfx86.mb2.EKH  (snapshot)
  ├─sfx86.mb2.U48  (snapshot)
  └─sfx86.mb2.V24  (snapshot)
$ sdk-manage target snapshot sfx86 sfx86.project
$ sdk-assistant package-install sfx86.project packageA packageB ...
...
$ sdk-assistant list
sf
├─sfarm
│ └─sfarm.mb2.YRN  (snapshot)
└─sfx86
  ├─sfx86.project  (snapshot)
  ├─sfx86.mb2.EKH  (snapshot)
  ├─sfx86.mb2.U48  (snapshot)
  └─sfx86.mb2.V24  (snapshot)
$ mb2 -t sfx86.project --snapshot build
Taking snapshot 'sfx86.project.mb2.W8F' of 'sfx86.project' target...
Using /srv/mer/toolings/sf//opt/cross/bin/i486-meego-linux-gnu-gcc to detect target architecture:
Finished writing sb2.gcc.config
gcc configured.
...
$ sdk-assistant list
sf
├─sfarm
│ └─sfarm.mb2.YRN  (snapshot)
└─sfx86
  ├─sfx86.project  (snapshot)
  │ └─sfx86.project.mb2.W8F  (snapshot)
  ├─sfx86.mb2.EKH  (snapshot)
  ├─sfx86.mb2.U48  (snapshot)
  └─sfx86.mb2.V24  (snapshot)

[rinigus]

Let’s start with the test result:

Having sb2 -t "$OPT_TARGET" -m sdk-install -R zypper "${args[@]}" in "${deps[@]}" before zypper dup and later followed with zypper in will solve my issue. In my case, the last zypper in would be no-op.

Re “dup is considered harmful in my case”:

As it is in released mb2 (before your patch or a patch allowing to disable zypper dup), resetting snapshot is insufficient in my case. If you look into details of Mb2 shared output directory: allow to disable zypper distribution update, zypper dup can install into the clean snapshot the package that will conflict with the buildrequirement package. That what happened in my case.

As with OBS, TBuilder starts with a clean snapshot and it looks to be an only sane way to ensure consistency of the environment. So, regardless to how zypper dup is resolved, I will keep it that way.

Re snapshots.snapshots and packages:

Very clever. In this case, it would be possible to use, but not from the start. Some of the preferred packages are built within that project (libhybris for example) and they fail to install at the start of the build. This failure is reported by tbuilder but considered OK and the build continues. Later, after libhybris is available, it is installed to replace Mesa libs, similar to OBS.

So, I could use your suggestion if I track the status of installation. In addition, I would have to track all packages that were installed during these operations (libhybris and all dependencies). The snapshot with these installed packages would have to be reset as soon as one of them is rebuilt for one reason or another. Which would make it non-trivial to implement, unfortunately.

[martyone]

@rinigus

As it is in released mb2 (before your patch or a patch allowing to disable zypper dup), resetting snapshot is insufficient in my case. […] zypper dup can install into the clean snapshot the package that will conflict with the buildrequirement package.

I still fail to clearly see how it happens that zypper dup on a clean target pulls in the unwanted package. Because something needs to require a package in order to be pulled in by zypper dup. And if we are talking about clear target…

Before you wrote

zypper dup pulls in droid-config-XXXX and, as a result, droid-system-apollo and droid-system-apollo-h8314

Which of the packages from config.yaml pulls in droid-config-XXXX as a dependency? Or which way it is required if not this way?

[rinigus]

Which of the packages from config.yaml pulls in droid-config-XXXX as a dependency? Or which way it is required if not this way?

droid-config-XXXX provide the same symbols as ofono-configs-mer and usb-moded-defaults. In fact, they obsolete these packages. As a result, configs are pulled in during zypper dup. As, in turn, they require droid-system-, the whole sequence of events starts.

PS: Just checked the logs, droid-config packages are not pulled by any of the forced-installed packages requested in the project

[martyone]

Great, your last message made it clear I think. Based on your previous test with the in-dup-in variant, simply swapping it should be sufficient and it should work equally well in general - test please https://github.com/sailfishos/sdk-setup/pull/236.

[rinigus]

I checked it on compilation of droid-hal-version- for 3 devices and it worked for all of them. So, in my case, this change is sufficient.

There is still some uncertainty in pulling some other packages (one other config was pulled during dup for all devices - droid-config-h8216-policy-settings), but that seemed not to interfere with the build. This aspect is somewhat concerning, but it is hopefully OK for most cases.

[martyone]

This is now fixed in Sailfish SDK 3.6, which was released to Early Access users just a while ago.

In addition to that, sfdk now allows to alter some aspects of its operation, including build target initialization, with hooks. See HOOKS in sfdk --help-all.