Migrating Sailfish OS git repos to Github

Announcements
#41

Foreword

I appreciate that the migration from git.sailfishos.org to github.com/sailfishos was handled better WRT announcements for specific actions and timelines provided in this thread, after the original announcement lacked a clear scope and any timelines.

Specific issue

After September, 14th we plan to shut down git.sailfishos.org . This will make it a full 3 month period from the time when the repositories were migrated.

Why the haste for this shutdown?
git.sailfishos.org has been put into read-only mode (“archived”) three months ago: While this (in conjunction with the announcements in this thread) should have been sufficient time for copying content away, it does not and cannot resolve the need for references, especially to content, which was not migrated (i.e., everything, except for source code).
Side note: As Jolla uses the Free Software variant of Gitlab for their instance, there are no license fees, only the infrastructure cost of running a very lightly used server.

Example for this issue

Due to the migration away from git.sailfishos.org, I had to re-issue two bug reports here at FSO (because the issue tracker at github.com/sailfishos is switched off for most, although not all sub-projects, see e.g. the sailfish-browser): 1 and 2
But they both reference the original context at git.sailfishos.org (comprised of issue reports there and related merge request comments there) to be fully comprehensible.

Suggestion how to handle this better

As Jolla’s understandable intention is to switch their Gitlab instance off sooner or later, postpone this step at least until the turn of the year (i.e., by 2,5 months or more).
This may provide sufficient time for open issues (e.g. the two mentioned above) to be read, understood and addressed by Jolla and / or others at their new location.

Additionally, the implementation of the redirects might be vastly improved:

To make it easier for people to find the new repositories, we will create redirects from the git.sailfishos.org public repositories to the GitHub ones, and a redirect from git.sailfishos.org to github.com/sailfishos . This should also keep the historical references alive.

This action / implementation will break all references (i.e., links), except those directed at the root of a specific project.
Plus it is tedious and error prone to create 702 redirects (the current number of sub-projects at GitHub), even when performed by a script. But these are the limitations of simple HTTP-redirects.
A proper implementation uses server-side scripting (e.g. Apaches’ or nginx’s mod_rewrite or other means), see URL redirection - Wikipedia.
It would be great if Jolla implements such a server-side “link rewriter script” for all calls of “https://github.com/sailfishos”-links: It simply has to transform a small part of a called link (in sed syntax: s/git.sailfishos.org\/mer-core/github.com\/sailfishos/).
Consequently this works for all links to source code (i.e., everything that was migrated), by performing e.g.,
https://git.sailfishos.org/mer-core/udisks2/blob/master/rpm/0002-Drop-smartata-dependencies.patch
→
https://github.com/sailfishos/udisks2/blob/master/rpm/0002-Drop-smartata-dependencies.patch

5 Likes
#42

The main answer here is simply lack of time. The server cost is negligilble in the scheme of things.

I installed and maintained gitlab voluntarily as part of all the rest of the Mer project infra almost single handed - and it wasn’t easy. Gitlab had (has) a very high frequency update cycle and the amount of work per update was multiple hours. Then they would do security updates on top of that. Eventually I fell behind and Jolla also decided to move to github. At this point the service was badly outdated and we (well, mainly I) pushed to put our (limited) community manpower into the OBS revival.

As for the haste; there was a suggestion to leave the service running in RO mode but again I have been pushing to close it down ASAP - running a very outdated service with known security holes is a dis-service to the community IMHO and although we considered upgrading I advised that it would be better not to use any resource doing so - so you can blame me there.

I didn’t look at how the redirects are being done but I admit that I assumed that the gitlab vs github url layout was not similar enough to support reliable rewritten urls especially when there will be issues and diff urls etc - I’ll suggest we spend a little time looking at that as I agree it would be useful in the short term (actually @ballock just mentioned this post in our meeting as I’m typing and he is planning on doing some rewrite rules).

I hope this explains some of the reasoning and background.

10 Likes
#43

@lbt, first of all: Thank you very much for your prompt and exhaustive reply!

That makes sense / is what I assumed.

Also a “thank you” for this decision: IMO it was the right one.

While your arguments are understandable, please let it run a little longer in read-only mode, as other sailors (as you reported) and I (above) are suggesting.

As denoted above: They are for all source code, including branches!

You did not migrate issues.
Hence links will either point to nowhere or a different issue.

Those may be hard.
IMO: never mind, source links are crucial!

Thanks a lot, also to @ballock for planning to use rewrite rules instead of redirects.

1 Like
#44

With any rewrite/redirect in place the service would be essentially inaccessible after the cutoff date anyway so there’s no real point keeping it running.

In any case my plan was to shut the service down but not decommision it entirely for some time (months).

Then if some issue came to light were a community member had forgotten/lost something it would be trivial to restart the service (for a short period) to provide access to recover data (via oldgit.sfos.org or something)

I hope that makes sense.

3 Likes
#45

I clearly denoted a reason to keep git.sailfishos.org running for an additional while in the section “Example for this issue” in my original post above: To keep references to issues and merge requests working a little longer. For enabling Jolla employees and others to comprehend the context of issues migrated here (to FSO).

That sure implies to wait with activating the rewrite rules, or to explicitly exclude the URLs of issues and merge requests from being rewritten by the rewrite rules; as they were not migrated, that also makes sense in the long run (even after switching off git.sailfishos.org).
Hence the latter (“rewrite rules, which exclude issues & MRs”) would be the best solution, if that is sufficiently easy to implement (should be, IMO).

#46

I have published the redirect service and you can check what it does and what it does not. It’s git-redir.sailfishos.org - you can test what redirect you get by replacing your git.sailfishos.org host in your browser’s URL field. It should redirect all mer-core repositories (and some other special ones, too), handles paths beneath the repo, so commit and tree references should work, handles query parameters, so cloning from the repositories should work.

Since issues and merge requests don’t map to GitHub, they redirect to the root of this repository in GitHub. This is the right thing to do since the GitLab instance is going to be shut down and it will be impossible to fetch content from it in any way but manually.

The redirect service took me some effort to create, it’s not trivial, but it should do the job. Feel free to test it and report if you find any issues with it.

2 Likes
#47

@ballock, thank you for implementing this diligently.
But you call for testing a black box, which makes little sense IMO: Can one review somewhere what you implemented (i.e., as a “white box”)?

BTW,

Oh how I love these implicit "No"s from sailors by completely ignoring input or questions from community members. :roll_eyes:
IMHO an explicit “No, we have decided otherwise and will not do this” is

  • clearer and unambiguous
  • more polite
#48

So, as Olf suggests, you’re going to ignore his request to keep the original URLs available and /dev/null them to the root of a github repo?

When?

#49

Btw. Olf, since you’re being a cipher, please tell me how may I pm you if I wish to discuss coordinating this kind of public discourse?

#50

Another question about Issues, as most discussion had been about migrating existing ones or not:

It looks like most packages on GitHub do not have issues enabled, where is the best place to report base system bugs, issues, or remarks now? The internal bug tracker will stay closed, and merproject’s bugzilla is dead as well.

I am asking that question because I am wondering about where to move my only open git.sailfishos.org issue about sailifishos/policycoreutils. As it has neither been closed for irrelevance nor received any comment, I still feel the urge to keep that remark about an earlier potential mistake of mine open.

#51

@schmittlauch, you are late to the party! :wink:

According to the schedule set by Jolla, you should save your open issue somewhere ASAP.

For some sub-projects at https://github.com/sailfishos/, the GitHub issue tracker is switched on.
Checked for GitHub - sailfishos/policycoreutils : Bad luck!
Side note: I am trying to convince Jolla to open the GitHub issue tracker for all sub-projects (some reasons denoted there) at https://github.com/sailfishos/.

So your only possibility to keep that issue publicly visible at an appropriate place is the “Bug Reports”-category here at FSO: Bug Reports - Sailfish OS Forum
Side note 2: The main reason why I am asking Jolla to keep git.sailfishos.org running a little longer and to not rewrite URLs to merge requests and issues there to /dev/null, nirvana or anywhere else now, is that I only copied & pasted an abbreviated / consolidated version of my open issues ([1] and [2]) to FSO (but I may have missed to copy something) and omitted context information discussed in merge requests comments there.
But as that request seems to be ignored, do copy your issue and its context entirely!

Happy copying & pasting!

#52

@poetaster, this forum (FSO) allows for PMs, so does TMO.
Plus I provide a way to chat with me, if you click on “Contact info” on my TMO user-page.

P.S.: What is “since you’re being a cipher” supposed to mean?
(No, I am not RSA, ECC, AES or ChaCha! :roll_eyes:)

#53

Sorry, but when I click on your user name I don’t see any of the usual links to profile/pm, etc. Maybe a forum bug? What I get is:

# olf

**[missing {{voteCount}} value]** Upvotes

This user's public profile is hidden.

On other users i see a message button. So that’s what I meant by cipher.

#54

Likely, because I have received PMs from other users here at FSO (the last one on 9 July 2021), and have not altered the settings recently.
You may use TMO for PMs instead, that works.

AFAIKS, there are two settings WRT PMs:

  1. User Profile → Notifications → Messages → Allow other users to send me personal messages
    This is checked (i.e, on), which is the default.
  2. User Profile → Notifications → Users →
    • Ignored
      List is empty
    • Muted
      List is empty
    • Only allow specific users to send me personal messages
      This is unchecked (i.e, off), which is the default.

AFAICS, there is nothing I can do to make PMs working.
Hence you might want to file a bug report.

Bug Report ‒ Forum User Info broken → Unable to send PM to affected users
#55

I also do not understand - the 3 months match the vacation period and is not enough time to deal with it.
At least for me I had to finish a project for leaving, start another project, move to another country and have vacation in between. So there was no time to look into this item on my todo list.
But anyway … I have the code on my server and can start over with github.
Again I wonder why Jolla does not put the same energy in fixing bugs.

1 Like
#56

I am sorry to hear not all of you are pleased with the GitHub migration.

I will make it clear: we will not migrate issues nor PRs from the GitLab instance. There will be no permanent links to any archived version of prior issues nor PRs.
The reason is that it takes too much engineer time to migrate them.

The deadline was set to “after September 14th”, and now I can give an exact date. I will replace the GitLab URLs with the redirector service on 7th of October, during business hours.

2 Likes
#57

@ballock Can someone give instructions/procedures how to proceed with PRs that are left over after 7th of October?

#58

@deloptes Sure. Instructions below. I don’t know my audience, so I’ll try to be as concise as possible, while keeping the details that will let the less experienced git/GitHub people dig in.

  1. Go to the new repository URL on GitHub. If you don’t know the URL, you can use your browser to get to the previous URL, it can also be found in your local git checkout. The redirect service should transfer you to the GitHub URL.
  2. Fork the repo to your home in GitHub. You need a GitHub account for that, and the “Fork” icon is on right-hand top.
  3. Add the GitHub-forked repo as a “git remote” to your local git repository, where you were working on the PR previously. The URL is under the green “Code” button.
  4. git push your changes to the git remote that you created.
  5. Create a pull request from the created GitHub repository, from the pushed branch. The “New pull request” button is under the “Pull requests” tab.
  6. Make sure the base repository, its base, and your repository’s branch is correct.
  7. Enter the related PR details, description, etc.
  8. Submit.
4 Likes
#59

@ballock perfectly clear - great and many thanks!

#60

Sad to see, that this was too quick for your fellow sailors!
Unfortunately this is what I expected to happen, hence advised against it, but even clearly pointing out the consequences for Jolla did not make a difference. :frowning: