Migrating Sailfish OS git repos to Github

[blizzz]

Not a fan of either Microsoft Github or Google Gitlab. There is little point in hosting a VCS yourself though.

Generally, I am concerned about Github being the central, mighty place for software. Apart of e.g. Iran which is cut off (on a different note, I order my favorite coffee via direct email, because it’s not available on the website for Paypal).

For the network effects Github is of course the best place to find contributors. That’s also why Nextcloud ended up there.

You see Microsoft’s EEE strategy in action. Just have a look at Github’s attempt to replace the CLI git utilities with their own flavour.

It’s just sad and disillusioning.

[olf]

Dear @veskuh,

one issue I see, that the Issues and Merge Requests usually will not be migrated.

Hence it would make sense to either take care open issues and MRs at git.sailfishos.org (some of which are open for long) before migrating these source repos or to also migrate them to Github (which will need some scripts to perform this; AFAIK such scripts do not publicly exist, yet).

For an example, see https://git.sailfishos.org/mer-core/udisks2

P.S.: Before starting the next hosting migration, it IMO does make sense to finish the last one, first: I have not seen any action (but maybe I missed something) towards migrating the content at together.jolla.com (“TJC”) to forum.sailfishos.org (“FSO”), as announced / promised by Jolla multiple times.
Can you please point out the status of that migration.

[rinigus]

I don’t think that they can realistically fix all before migration. So, probably issues will have to be reviewed and migrate accordingly. “Spring cleaning” style.

Otherwise, Github/Gitlab seems to rather divisive issue, way more heated than I expected. In the end of the day, it is good to consolidate on single platform and I see benefit in offloading this to some service provider. So, all in all, whatever destination you choose, looking forward to it.

[veskuh]

@olf

one issue I see, that the Issues and Merge Requests usually will not be migrated.

Hence it would make sense to either take care open issues and MRs at git.sailfishos.org (some of which are open for long) before migrating these source repos or to also migrate them to Github (which will need some scripts to perform this; AFAIK such scripts do not publicly exist, yet).

For an example, see https://git.sailfishos.org/mer-core/udisks2

Yes, thanks for pointing this out. This should have been mentioned in my original message. We do not plan to migrate open pull requests and would prefer to handle those before migration as much as possible. This is real issue and something where need to put more effort.

P.S.: Before starting the next hosting migration, it IMO does make sense to finish the last one, first: I have not seen any action (but maybe I missed something) towards migrating the content at together.jolla.com (“TJC”) to forum.sailfishos.org (“FSO”), as announced / promised by Jolla multiple times.
Can you please point out the status of that migration.

This would need a proper update on the related thread. To put it short: this has been on hold as it was more complicated than originally thought.

[olf]

@veskuh

This would need a proper update on the related thread. To put it short: this has been on hold as it was more complicated than originally thought.

@veskuh, seriously a big “thank you” for this statement (although unfortunate)!
It is the first statement WRT this topic from an Jolla employee for almost a year (despite that being asked so many times).
And it was honest and authentic.

Thus please trigger your colleagues (i.e., Jolla’s sailors) to explain the situation a bit better and Jolla’s ideas how to proceed.
Thanks!

[pcfe]

(just forwar to the relevant Product Manager)
Can Jolla please clarify which of the following you are planning to migrate (I’d like to know, I’m realistic that you will leave some history behind, but it would be good to know how much exactly)

• Issues, closed
• Issue, open
• tags
• releases
• current active branches
• commits in those branches
• (PRs is clear now and understandable)

I ask because well,… for one Jolla does not have the best track record when it comes to moving data from a deprecated system (forum, wiki, etc) to a new one.
Secondly, in other companies, I have seen a few version control system migrations, those ranged all the way from a super nice migration of all open and closed issues and all active branches and all commits in those branches all the way down to the horrifying only a shallow clone of depth 1 of the main branch being pushed to the new git server, nothing else

Also, please remember that if repo maintainers were to not plan to maintain issues and/or wikis, etc on one or more repos, to switch those off on a per repo base. Primarily so that it’s clear which your single source of truth for bugs for each repo is. Nothing is more frustrating to me than to notice I filed a bug in system A only to be told sorry, wrong bugtracker, please use X to file bugs against this repo.

pcfe

[attah]

This is not a version control system change. It is still git, so commits and tags will follow, probably most branches too.
I’m guessing the rest will be left behind, and honestly that doesn’t seem so bad to me.
The relevant MRs/PRs can be reopened, and not that i think releases are used, but after a few months historical releases are quite useless. There appears to be a grand total of 21 issues across all projects in mer-core, 17 of which are open. I don’t think they were meant to be used, as there is some sort of internal Bugzilla or similar, and we users are relegated to the forum.

[ballock]

Hi, pcfe.

I own the hands performing the migration. What attah guessed is right.

My plan is to migrate only git repositories themselves, without any custom GitLab/GitHub added functionality. Thus what is going to be migrated are:

• all git branches
• git commits in all git branches
• git tags

As for:

• issues - not migrated. Sorry.
• relases - I don’t think we ever used those.
• PRs - not migrated. Sorry.

The major reason for not migrating the issues and PRs is that these differ between GitLab and GitHub and migrating them will require more effort, while the benefit of that is questionable.

As for PRs, these will be used in GitHub, so anyone who had a change proposal and submitted it as a PR to GitLab is encouraged to recreate the PR in GitHub. I know I could spend a (non-trivial) amount of time migrating the PRs, however, most PRs that I checked were either old-age, incomplete, or otherwise not worth it. I am pretty sure the PRs that are of worth will either be recreated by the people who submitted them originally, or by the team maintaining the repository.

We are planning to keep the current GitLab instance running in read-only mode for some time after the GitHub repositories are taken into use to allow interested parties to move anything from a current PR to the new repositories.

As for the bugtracker, I am sorry about the situation, but I am unable to do anything about it. The Jolla employees are using an internal bugtracker, and any issue/bug/PR reported elsewhere needs to be mapped to an internal bug anyway, thus it does not matter much if the bug is reported in any Bugzilla, GitLab or GitHub issues, on the Forum, on IRC, or a community beer event. The internal bugs created will not disappear, and will hopefully be resolved by the team.

I will check with the teams to see if they are planning to use the issues or wikis in GitHub, and will lock them otherwise. Thank you for the note about that.

[veskuh]

We’ve made progress in migrating. We are soon ready to move to take github into use and then make projects at git.sailfishos.org read-only

We have tried to be active in reviewing, commenting and merging open pull requests for the past weeks. However, we can’t get the queue completely empty as there are always new ones incoming, and for many MRs review cycle takes long time. To get forward we need to set a timeline for the transition even though there are going to be open MRs. After the transition merge requests are no longer handled in git.sailfishos.org, but instead we encourage contributors open new ones in GitHub.

We target to take GitHub in use and make git.sailfishos.org projects read-only during the week starting 14.6. We will continue to review, comment and merge contributions at git.sailfishos.org until then.

Big thank you for the contributors for the patience and working with us to make this change and to make the software better.

BR,
Vesku

[dcaliste]

Does anyone know how do you list opened MRs for the SailfishOS organisation in Github ?

[poetaster]

I was always under the impression it’s a pull request that the owner of the repo then moves to merge but there is no ‘merge request’ per se. I couldn’t find such a thing in the docs. But maybe your referring to something like Rebase and merge your pull request commits

Thanks Guys! I’m happy, but I know not everyone will be ;<

[dcaliste]

Ok, sorry wrong wording in the context of Github. I’m looking for a place that lists all opened PRs for a given organisation, here SailfishOS. That is a convenient way to baby-sit opened PRs, see what is still pending and need attention.

When you work with the full software stack, you end up with multiple PRs opened in many projects. For instance, to deal with calendar entries, it happens that I have PRs in kf5-calendarcore, mKCal, nemo-qml-plugin-calendar and buteo-sync-plugin-caldav at the same time. Having the possibility to list my opened PRs makes it easier to see how things are evolving.

But more generaly, that’s interesting also to see PRs opened by others. You can learn about new pieces of the code, or comment there if it happens that this is related to something you know or work on, but you’re not actively watching the repo. One example : the latest PR by @mal in cmake repo. I’m not watching this repo and I don’t plan to, but it happened that he submitted a PR that actually solve a problem we faced with @PureTryOut in the mkcal repository. I know this because I’ve seen it in the activity list in Gitlab before the migration. Now I would have missed it.

[Keto]

Would this help

https://github.com/pulls?q=is%3Aopen+is%3Apr+user%3Asailfishos

(Note, you need to be logged in on github to access that page)

[dcaliste]

Great, thanks. That is indeed what I was looking for. It will be helpful.

[oroesler]

@ballock

The major reason for not migrating the issues and PRs is that these differ between GitLab and GitHub and migrating them will require more effort, while the benefit of that is questionable.

I just want to notice that actually both Github and gitlab’s Python ABI are pretty straightforward, basic issue transfer actually isn’t hard. It’s you decision where to spend your time, but even besides reimplementing this I highly doubt there is no foss tool for Gitlab to GH migration and counterwise.

About the benefecial part: Important things will get a new issue and MR/PR, that’s probably true. But what will eventually get lost is important side-info from Issues and MR, resulting in Bugfixes and Improvements introducing new bugs.

Just my 2 cents, fine with as it is.

[KuroNeko]

Most of the bugs were reported into the jolla bug tracker. Gitlab was never used for issues, apart from some exceptions. But even then, most of that had a jolla bug tracker mirror, afaik. So at least according to the policy, there should have been nothing of value in the issues. Not sure, what the reality is though.

[poetaster]

Thanks! I was looking around too… with less concise results. And was wondering why this isn’t a menu item when your logged in

[ballock]

Hi, everyone,

It has been over 2 months since the migration started, and I saw a number of PRs being migrated to GitHub. Thanks to everyone who took the time to reopen the PRs, I believe this also helps the teams to focus their efforts on PRs that are still cared about.

After September, 14th we plan to shut down git.sailfishos.org. This will make it a full 3 month period from the time when the repositories were migrated.

If you still have any MRs in GitLab that you want to retain, please repoen them in GitHub before the shutdown. I hope this is enough time for everyone who needs access to the old repos.

Also, git.sailfishos.org has a number of personal projects. I am guessing that these were mostly used to create a code branch for an MR, and may no longer be needed after the MRs are migrated to GitHub. However, if any of you hold any code of value in the personal projects in git.sailfishos.org, please migrate them to your git server of choice before GitLab is shut down.

To make it easier for people to find the new repositories, we will create redirects from the git.sailfishos.org public repositories to the GitHub ones, and a redirect from git.sailfishos.org to github.com/sailfishos. This should also keep the historical references alive.

[olf]

Foreword

I appreciate that the migration from git.sailfishos.org to github.com/sailfishos was handled better WRT announcements for specific actions and timelines provided in this thread, after the original announcement lacked a clear scope and any timelines.

Specific issue

After September, 14th we plan to shut down git.sailfishos.org . This will make it a full 3 month period from the time when the repositories were migrated.

Why the haste for this shutdown?
git.sailfishos.org has been put into read-only mode (“archived”) three months ago: While this (in conjunction with the announcements in this thread) should have been sufficient time for copying content away, it does not and cannot resolve the need for references, especially to content, which was not migrated (i.e., everything, except for source code).
_{Side note: As Jolla uses the Free Software variant of Gitlab for their instance, there are no license fees, only the infrastructure cost of running a very lightly used server.}

Example for this issue

Due to the migration away from git.sailfishos.org, I had to re-issue two bug reports here at FSO (because the issue tracker at github.com/sailfishos is switched off for most, although not all sub-projects, see e.g. the sailfish-browser): 1 and 2
But they both reference the original context at git.sailfishos.org (comprised of issue reports there and related merge request comments there) to be fully comprehensible.

Suggestion how to handle this better

As Jolla’s understandable intention is to switch their Gitlab instance off sooner or later, postpone this step at least until the turn of the year (i.e., by 2,5 months or more).
This may provide sufficient time for open issues (e.g. the two mentioned above) to be read, understood and addressed by Jolla and / or others at their new location.

Additionally, the implementation of the redirects might be vastly improved:

To make it easier for people to find the new repositories, we will create redirects from the git.sailfishos.org public repositories to the GitHub ones, and a redirect from git.sailfishos.org to github.com/sailfishos . This should also keep the historical references alive.

This action / implementation will break all references (i.e., links), except those directed at the root of a specific project.
Plus it is tedious and error prone to create 702 redirects (the current number of sub-projects at GitHub), even when performed by a script. But these are the limitations of simple HTTP-redirects.
A proper implementation uses server-side scripting (e.g. Apaches’ or nginx’s mod_rewrite or other means), see URL redirection - Wikipedia.
It would be great if Jolla implements such a server-side “link rewriter script” for all calls of “https://github.com/sailfishos”-links: It simply has to transform a small part of a called link (in sed syntax: s/git.sailfishos.org\/mer-core/github.com\/sailfishos/).
Consequently this works for all links to source code (i.e., everything that was migrated), by performing e.g.,
https://git.sailfishos.org/mer-core/udisks2/blob/master/rpm/0002-Drop-smartata-dependencies.patch
→
https://github.com/sailfishos/udisks2/blob/master/rpm/0002-Drop-smartata-dependencies.patch

[lbt]

The main answer here is simply lack of time. The server cost is negligilble in the scheme of things.

I installed and maintained gitlab voluntarily as part of all the rest of the Mer project infra almost single handed - and it wasn’t easy. Gitlab had (has) a very high frequency update cycle and the amount of work per update was multiple hours. Then they would do security updates on top of that. Eventually I fell behind and Jolla also decided to move to github. At this point the service was badly outdated and we (well, mainly I) pushed to put our (limited) community manpower into the OBS revival.

As for the haste; there was a suggestion to leave the service running in RO mode but again I have been pushing to close it down ASAP - running a very outdated service with known security holes is a dis-service to the community IMHO and although we considered upgrading I advised that it would be better not to use any resource doing so - so you can blame me there.

I didn’t look at how the redirects are being done but I admit that I assumed that the gitlab vs github url layout was not similar enough to support reliable rewritten urls especially when there will be issues and diff urls etc - I’ll suggest we spend a little time looking at that as I agree it would be useful in the short term (actually @ballock just mentioned this post in our meeting as I’m typing and he is planning on doing some rewrite rules).

I hope this explains some of the reasoning and background.