Migrating Sailfish OS git repos to Github

[dcaliste]

Great, thanks. That is indeed what I was looking for. It will be helpful.

[oroesler]

@ballock

The major reason for not migrating the issues and PRs is that these differ between GitLab and GitHub and migrating them will require more effort, while the benefit of that is questionable.

I just want to notice that actually both Github and gitlab’s Python ABI are pretty straightforward, basic issue transfer actually isn’t hard. It’s you decision where to spend your time, but even besides reimplementing this I highly doubt there is no foss tool for Gitlab to GH migration and counterwise.

About the benefecial part: Important things will get a new issue and MR/PR, that’s probably true. But what will eventually get lost is important side-info from Issues and MR, resulting in Bugfixes and Improvements introducing new bugs.

Just my 2 cents, fine with as it is.

[KuroNeko]

Most of the bugs were reported into the jolla bug tracker. Gitlab was never used for issues, apart from some exceptions. But even then, most of that had a jolla bug tracker mirror, afaik. So at least according to the policy, there should have been nothing of value in the issues. Not sure, what the reality is though.

[poetaster]

Thanks! I was looking around too… with less concise results. And was wondering why this isn’t a menu item when your logged in

[ballock]

Hi, everyone,

It has been over 2 months since the migration started, and I saw a number of PRs being migrated to GitHub. Thanks to everyone who took the time to reopen the PRs, I believe this also helps the teams to focus their efforts on PRs that are still cared about.

After September, 14th we plan to shut down git.sailfishos.org. This will make it a full 3 month period from the time when the repositories were migrated.

If you still have any MRs in GitLab that you want to retain, please repoen them in GitHub before the shutdown. I hope this is enough time for everyone who needs access to the old repos.

Also, git.sailfishos.org has a number of personal projects. I am guessing that these were mostly used to create a code branch for an MR, and may no longer be needed after the MRs are migrated to GitHub. However, if any of you hold any code of value in the personal projects in git.sailfishos.org, please migrate them to your git server of choice before GitLab is shut down.

To make it easier for people to find the new repositories, we will create redirects from the git.sailfishos.org public repositories to the GitHub ones, and a redirect from git.sailfishos.org to github.com/sailfishos. This should also keep the historical references alive.

[olf]

Foreword

I appreciate that the migration from git.sailfishos.org to github.com/sailfishos was handled better WRT announcements for specific actions and timelines provided in this thread, after the original announcement lacked a clear scope and any timelines.

Specific issue

After September, 14th we plan to shut down git.sailfishos.org . This will make it a full 3 month period from the time when the repositories were migrated.

Why the haste for this shutdown?
git.sailfishos.org has been put into read-only mode (“archived”) three months ago: While this (in conjunction with the announcements in this thread) should have been sufficient time for copying content away, it does not and cannot resolve the need for references, especially to content, which was not migrated (i.e., everything, except for source code).
_{Side note: As Jolla uses the Free Software variant of Gitlab for their instance, there are no license fees, only the infrastructure cost of running a very lightly used server.}

Example for this issue

Due to the migration away from git.sailfishos.org, I had to re-issue two bug reports here at FSO (because the issue tracker at github.com/sailfishos is switched off for most, although not all sub-projects, see e.g. the sailfish-browser): 1 and 2
But they both reference the original context at git.sailfishos.org (comprised of issue reports there and related merge request comments there) to be fully comprehensible.

Suggestion how to handle this better

As Jolla’s understandable intention is to switch their Gitlab instance off sooner or later, postpone this step at least until the turn of the year (i.e., by 2,5 months or more).
This may provide sufficient time for open issues (e.g. the two mentioned above) to be read, understood and addressed by Jolla and / or others at their new location.

Additionally, the implementation of the redirects might be vastly improved:

To make it easier for people to find the new repositories, we will create redirects from the git.sailfishos.org public repositories to the GitHub ones, and a redirect from git.sailfishos.org to github.com/sailfishos . This should also keep the historical references alive.

This action / implementation will break all references (i.e., links), except those directed at the root of a specific project.
Plus it is tedious and error prone to create 702 redirects (the current number of sub-projects at GitHub), even when performed by a script. But these are the limitations of simple HTTP-redirects.
A proper implementation uses server-side scripting (e.g. Apaches’ or nginx’s mod_rewrite or other means), see URL redirection - Wikipedia.
It would be great if Jolla implements such a server-side “link rewriter script” for all calls of “https://github.com/sailfishos”-links: It simply has to transform a small part of a called link (in sed syntax: s/git.sailfishos.org\/mer-core/github.com\/sailfishos/).
Consequently this works for all links to source code (i.e., everything that was migrated), by performing e.g.,
https://git.sailfishos.org/mer-core/udisks2/blob/master/rpm/0002-Drop-smartata-dependencies.patch
→
https://github.com/sailfishos/udisks2/blob/master/rpm/0002-Drop-smartata-dependencies.patch

[lbt]

The main answer here is simply lack of time. The server cost is negligilble in the scheme of things.

I installed and maintained gitlab voluntarily as part of all the rest of the Mer project infra almost single handed - and it wasn’t easy. Gitlab had (has) a very high frequency update cycle and the amount of work per update was multiple hours. Then they would do security updates on top of that. Eventually I fell behind and Jolla also decided to move to github. At this point the service was badly outdated and we (well, mainly I) pushed to put our (limited) community manpower into the OBS revival.

As for the haste; there was a suggestion to leave the service running in RO mode but again I have been pushing to close it down ASAP - running a very outdated service with known security holes is a dis-service to the community IMHO and although we considered upgrading I advised that it would be better not to use any resource doing so - so you can blame me there.

I didn’t look at how the redirects are being done but I admit that I assumed that the gitlab vs github url layout was not similar enough to support reliable rewritten urls especially when there will be issues and diff urls etc - I’ll suggest we spend a little time looking at that as I agree it would be useful in the short term (actually @ballock just mentioned this post in our meeting as I’m typing and he is planning on doing some rewrite rules).

I hope this explains some of the reasoning and background.

[olf]

@lbt, first of all: Thank you very much for your prompt and exhaustive reply!

@lbt

The main answer here is simply lack of time. The server cost is negligilble in the scheme of things.

That makes sense / is what I assumed.

[…] and we (well, mainly I) pushed to put our (limited) community manpower into the OBS revival.

Also a “thank you” for this decision: IMO it was the right one.

As for the haste; […]

While your arguments are understandable, please let it run a little longer in read-only mode, as other sailors (as you reported) and I (above) are suggesting.

I didn’t look at how the redirects are being done but I admit that I assumed that the gitlab vs github url layout was not similar enough to support reliable rewritten urls

As denoted above: They are for all source code, including branches!

especially when there will be issues

You did not migrate issues.
Hence links will either point to nowhere or a different issue.

and diff urls etc -

Those may be hard.
IMO: never mind, source links are crucial!

I’ll suggest we spend a little time looking at that as I agree it would be useful in the short term (actually @ballock just mentioned this post in our meeting as I’m typing and he is planning on doing some rewrite rules).

I hope this explains some of the reasoning and background.

Thanks a lot, also to @ballock for planning to use rewrite rules instead of redirects.

[lbt]

@olf

While your arguments are understandable, please let it run a little longer in read-only mode, as other sailors (as you reported) and I (above) are suggesting.

With any rewrite/redirect in place the service would be essentially inaccessible after the cutoff date anyway so there’s no real point keeping it running.

In any case my plan was to shut the service down but not decommision it entirely for some time (months).

Then if some issue came to light were a community member had forgotten/lost something it would be trivial to restart the service (for a short period) to provide access to recover data (via oldgit.sfos.org or something)

I hope that makes sense.

[olf]

@lbt

With any rewrite/redirect in place the service would be essentially inaccessible after the cutoff date anyway so there’s no real point keeping it running.

I clearly denoted a reason to keep git.sailfishos.org running for an additional while in the section “Example for this issue” in my original post above: To keep references to issues and merge requests working a little longer. For enabling Jolla employees and others to comprehend the context of issues migrated here (to FSO).

That sure implies to wait with activating the rewrite rules, or to explicitly exclude the URLs of issues and merge requests from being rewritten by the rewrite rules; as they were not migrated, that also makes sense in the long run (even after switching off git.sailfishos.org).
Hence the latter (“rewrite rules, which exclude issues & MRs”) would be the best solution, if that is sufficiently easy to implement (should be, IMO).

[ballock]

I have published the redirect service and you can check what it does and what it does not. It’s git-redir.sailfishos.org - you can test what redirect you get by replacing your git.sailfishos.org host in your browser’s URL field. It should redirect all mer-core repositories (and some other special ones, too), handles paths beneath the repo, so commit and tree references should work, handles query parameters, so cloning from the repositories should work.

Since issues and merge requests don’t map to GitHub, they redirect to the root of this repository in GitHub. This is the right thing to do since the GitLab instance is going to be shut down and it will be impossible to fetch content from it in any way but manually.

The redirect service took me some effort to create, it’s not trivial, but it should do the job. Feel free to test it and report if you find any issues with it.

[olf]

@ballock, thank you for implementing this diligently.
But you call for testing a black box, which makes little sense IMO: Can one review somewhere what you implemented (i.e., as a “white box”)?

BTW,

@ballock

Since issues and merge requests don’t map to GitHub, they redirect to the root of this repository in GitHub. This is the right thing to do since the GitLab instance is going to be shut down and it will be impossible to fetch content from it in any way but manually.

Oh how I love these implicit "No"s from sailors by completely ignoring input or questions from community members.
IMHO an explicit “No, we have decided otherwise and will not do this” is

• clearer and unambiguous
• more polite

[poetaster]

So, as Olf suggests, you’re going to ignore his request to keep the original URLs available and /dev/null them to the root of a github repo?

When?

[poetaster]

Btw. Olf, since you’re being a cipher, please tell me how may I pm you if I wish to discuss coordinating this kind of public discourse?

[schmittlauch]

Another question about Issues, as most discussion had been about migrating existing ones or not:

It looks like most packages on GitHub do not have issues enabled, where is the best place to report base system bugs, issues, or remarks now? The internal bug tracker will stay closed, and merproject’s bugzilla is dead as well.

I am asking that question because I am wondering about where to move my only open git.sailfishos.org issue about sailifishos/policycoreutils. As it has neither been closed for irrelevance nor received any comment, I still feel the urge to keep that remark about an earlier potential mistake of mine open.

[olf]

@schmittlauch, you are late to the party!

According to the schedule set by Jolla, you should save your open issue somewhere ASAP.

For some sub-projects at https://github.com/sailfishos/, the GitHub issue tracker is switched on.
Checked for GitHub - sailfishos/policycoreutils : Bad luck!
Side note: I am trying to convince Jolla to open the GitHub issue tracker for all sub-projects (some reasons denoted there) at https://github.com/sailfishos/.

So your only possibility to keep that issue publicly visible at an appropriate place is the “Bug Reports”-category here at FSO: Bug Reports - Sailfish OS Forum
Side note 2: The main reason why I am asking Jolla to keep git.sailfishos.org running a little longer and to not rewrite URLs to merge requests and issues there to /dev/null, nirvana or anywhere else now, is that I only copied & pasted an abbreviated / consolidated version of my open issues ([1] and [2]) to FSO (but I may have missed to copy something) and omitted context information discussed in merge requests comments there.
But as that request seems to be ignored, do copy your issue and its context entirely!

Happy copying & pasting!

[olf]

@poetaster, this forum (FSO) allows for PMs, so does TMO.
Plus I provide a way to chat with me, if you click on “Contact info” on my TMO user-page.

P.S.: What is “since you’re being a cipher” supposed to mean?
(No, I am not RSA, ECC, AES or ChaCha! )

[poetaster]

Sorry, but when I click on your user name I don’t see any of the usual links to profile/pm, etc. Maybe a forum bug? What I get is:

# olf

**[missing {{voteCount}} value]** Upvotes

This user's public profile is hidden.

On other users i see a message button. So that’s what I meant by cipher.

[olf]

@poetaster

Maybe a forum bug?

Likely, because I have received PMs from other users here at FSO (the last one on 9 July 2021), and have not altered the settings recently.
You may use TMO for PMs instead, that works.

AFAIKS, there are two settings WRT PMs:

1. User Profile → Notifications → Messages → Allow other users to send me personal messages
   This is checked (i.e, on), which is the default.
2. User Profile → Notifications → Users →
   • Ignored
     List is empty
   • Muted
     List is empty
   • Only allow specific users to send me personal messages
     This is unchecked (i.e, off), which is the default.

AFAICS, there is nothing I can do to make PMs working.
Hence you might want to file a bug report.

[deloptes]

I also do not understand - the 3 months match the vacation period and is not enough time to deal with it.
At least for me I had to finish a project for leaving, start another project, move to another country and have vacation in between. So there was no time to look into this item on my todo list.
But anyway … I have the code on my server and can start over with github.
Again I wonder why Jolla does not put the same energy in fixing bugs.