Offline custom maps + GPX Viewer

Shocker · 23 August 2022 19:00

Hello, after 10 years with Nokia 808 I finally migrated to Xperia III with Sailfish OS. Everything went smoothly and now I am happy user of new phone with interesting OS. On old phone I used a lot Sport Tracker application (mainly for bike trips) with Polar HRM. I have a lot saved routes (only locally - I didn’t upload results). Now I tried Kuri and it works almost like Sport Tracker, but I have some questions related to maps and GPX support.

On old phone I used offline maps prepared by Mobile Atlas Creator (MOBAC) application. The big advantage of this app is that I can download (as simple image tiles) from custom source. I used as source mainly this map: osmapa.pl (it is free fork of OSM) - because it has a good coverage of walking trails / bike tracks. Second good map which I used is mapy.cz. It is possible with ANY native app available for Sailfish to download and use offline custom maps? I know that there is good application: OSM Scout Server (used with Kuri) - but I don’t know how can I replace maps downloaded by this application with my custom map. I also know that for mapy.cz it is dedicated Android app - but it is not a perfect solution to run Kuri and Mapy.cz application simultaneously…
Second thing is to migrate old saved activities from Sports Tracker to my new phone (locally). Unfortunately Kuri doesn’t recognize GPX files exported from Sports Tracker (which is strange, I thought that GPX files are unified - it is, after all, simple XML - GPX versions of Kuri/Sports Tracker are the same version=“1.1”). I found also Mæp application - which can open these GPX files, but it shows only basic info (no graphs, no HR pulse, etc.). Do you know if there is ANY native Sailfish app which can open GPX files with all saved details? I found one fantastic application which can be used also with custom maps: GPXSee - it is QT app, and somebody long time ago ported it for Maemo, but I cannot find for Sailfish OS… There is of course Andorid version of this app, but it not works, crashes at start. Rest popular Android applications - doesn’t work - because “you don’t have Google Services”). So I stuck and wrote this post

If you have any ideas how to deal in Sailfish OS with custom maps and GPX files I will be very grateful.

Regards,
R.

defactofactotum · 23 August 2022 19:20

Xperia III?
I’ve tried Kuri in the past but it doesn’t seem available for my 64bit device and overall I found OSM Scout to be the best for tracking. You can import and export gpx and download offline maps at country level.

karry · 23 August 2022 19:41

You may try OSM Scout It is standalone app, don’t mix with OSM Scout server that provides maps to other apps over local http.

leon · 23 August 2022 23:16

Its possible to load GPX data into Pure Maps!

rinigus · 24 August 2022 06:23

Everything is possible, just depends on whether you are prepared to work for it .

One simple option would be to download tiles into some folder and serve them using python -m http.server 8000. Then you would need a client app that would be able to display it. Pure Maps can easily be extended by custom map providers, including a local ones. See docs and config files in that directory for example. OpenSlopeMap is one of the examples of raster tiles. To import it to Kuri, you would probably need to check its source and replace URL that it is using to load maps. It may require writing some simple Mapbox map style to go with it and serve that from the folder with tiles. While sounds complicated, it is not. See https://github.com/rinigus/pure-maps/blob/master/poor/map.py#L139 for style generation example. Mapbox docs are also great source for info: Add a raster tile source | Mapbox GL JS | Mapbox and Sources | Style Specification | Mapbox GL JS | Mapbox

Note that I haven’t added ability to serve some extra map tiles using OSM Scout Server. In principle, it would require additional code but, if there is a lot of interest, we can work on it. Ideally, interested users would have to open an issue and try to implement it. I’ll be happy to give pointers on where the changes are needed.

If you want to compromise on being offline and if osmapa.pl / mapy.cz have their tiles available, it is trivial to add those providers directly to Pure Maps. That would involve some caching for occasional offline use, but you would require online access mostly during hiking.

Why Kuri doesn’t load your GPX files I don’t know. Is there some kind of error during import? My experience with Kuri is limited and I don’t know how it loads the data.

Side note: Pure Maps, while can load GPX files for routing, is not designed for tracking and handling GPX tracks. I believe that Pure Maps has already a lot of functionality and a dedicated sports / tracking app makes way more sense than to try to squeeze it in there. In this respect, Kuri make more sense to use for it. OSM Scout by @karry maybe good as well for it, I just don’t know how well it is supported.

Shocker · 25 August 2022 18:43

Thank you all for replies. In meantime I did more tests with tracking apps (Kuri / Laufhelden / OSM Scout).

Lets start from GPX files:

Indeed OSM Scout can import GPX files from my old Nokia Sports Tracker (and from Kuri/Laufhelden also, so I suppose from most of the tracking apps) without any errors, that is good, but… the bad thing is that during import application removes heart rate statistics (pulse bpm)…
I don’t know why, but all files (imported and created directly by OSM Scout) doesn’t show the route on map - when I click “show” in track details - map center on the area of saved track, but shows nothing (no color of the track - even if I try change this color) - it is a bug or I missed something? @karry - this is a question for you when you have a moment if you can check this “bug” and check this point about Heart Rate import from GPX files…
Import GPX to Kuri/Laufhelden also solved… I checked this thread, and funny thing that one additional line in GPX broke the import in the app, after deleting it, import was successful (import to Kuri/Laufhelden = copy/paste GPX file to application directory, there is no import option in application like in OSM Scout).

Now about the maps… This topic is harder than I thought…
Lets forget for a while about offline maps (which I asked in first post).

If I understood correctly the easiest way to add additional custom (simple image tiles) maps is add them to Pure Maps. Unfortunately as mentioned Pure Maps application is not dedicated to hiking/cycling - so it will not help me - because I will not use this app as main tracking app. It is good app for example for car navigation.
So my another question is there easy way to add online maps (tiles server name) to any application that shows map in Sailfish OS? For example in MOBAC or GPXSee (which I mentioned in first post) I had to only create custom XML file and put it in the application directory, example from MOBAC:

<?xml version="1.0" encoding="UTF-8"?>
<customMapSource>
   <name>Mapa.cz</name>
   <minZoom>0</minZoom>
   <maxZoom>19</maxZoom>
   <tileType>png</tileType>
   <url>http://{$serverpart}.mapserver.mapy.cz/wturist-m//{$z}-{$x}-{$y}</url>
   <serverParts>m1 m2 m3</serverParts>
</customMapSource>

or in GPXSee (I created this based on another official examples):

<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://www.gpxsee.org/map/1.4">
  <name>Mapy.cz</name>
  <url>http://m1.mapserver.mapy.cz/wturist-m/$z-$x-$y</url>
  <copyright>© Seznam.cz</copyright>
</map>

Point about adding extra maps to OSM Scout Server it is also interesting @rinigus. I can help of course with this topic if it is possible, because I saw your answer on github - you wrote: “I don’t plan to include tile caching into OSM Scout Server as it is written to work with the data that it has (search and routing) which is impossible when using just tiles.” - and these XML files are only tiles (?)…

BTW. OSM Scout has built in Open Cycle Map (offline), and I can use this as substitute of for example Mapy.cz. If the application has also Heart Rate Monitor option implemented like Kuri/Laufhelden - then I could use only OSM Scout

Regards,
R.

karry · 25 August 2022 21:09

Hi, regarding OSM Scout

Can you link here schema of gpx extension for pulse monitor please? I will create ticket on Github and implement it in future possibly.

Gpx tracks are rendered by offline map renderer. It needs to be initialized. Did you have downloaded at least some offline database? Did you have offline rendering enabled in map settings?

Of course, I had this usecase in mind. I just don’t include multiple providers by default, because its licenses don’t allow it usually. But you can add any (raster) tile server manually to /usr/share/harbour-osmscout/resources/online-tile-providers.json and restart OSM Scout then. Just add another json object to the list:

  {
    "id": "mapycz",
    "name": "Mapy.cz",
    "servers": [
      "http://m1.mapserver.mapy.cz/wturist-m/%1-%2-%3",
      "http://m2.mapserver.mapy.cz/wturist-m/%1-%2-%3",
      "http://m3.mapserver.mapy.cz/wturist-m/%1-%2-%3"
    ],
    "maximumZoomLevel": 19, 
    "copyright": "© Seznam.cz, OpenStreetMap contributors and others"
  }

rinigus · 26 August 2022 17:57

If OSM Scout looks good functionality wise, then go with it. For Kuri, you would have to start adjusting its code in settings and maybe something else. I am not familiar with the internals of Kuri - just assuming it is not too complicated to point towards some other online server.

Re OSM Scout Server and caching: This type of request comes once in a while. Its surely not a priority, as I have hands full with many other aspects of the server.

Note that the vector tile import will be changed soon and maybe there will be more data shown or available in tiles. Let’s see whether someone will want to work on map styles targeting hiking requirements better.

Shocker · 26 August 2022 18:14

Thank you very much @karry!

You were right, I downloaded offline maps, and now everything works as expected (route is visible on the map).
The great thing (thank you for hint about file where I can add maps) is that I can see this route on added maps (mapy.cz and osmapa.pl) - this is great, and I can use your app without additional android applications!

So the last thing is this pulse in GPX (imported), because I think that adding pulse monitor via bluetooth device to your app is not in your “to do list”

What I see in the created files:

GPX from Kuri/Laufhelden uses “TrackPointExtension” if I see correctly:

<gpx xmlns="http://www.topografix.com/GPX/1/1" xsi:schemaLocation="http://www.topografix.com/GPX/1/1 http://www.topografix.com/GPX/1/1/gpx.xsd http://www.garmin.com/xmlschemas/GpxExtensions/v3 http://www.garmin.com/xmlschemas/GpxExtensionsv3.xsd http://www.garmin.com/xmlschemas/TrackPointExtension/v1 http://www.garmin.com/xmlschemas/TrackPointExtensionv1.xsd" version="1.1" Creator="Laufhelden" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:gpxtpx="http://www.garmin.com/xmlschemas/TrackPointExtension/v1">
            ...
            <trkpt lat="50.4237992" lon="19.27443156">
                <time>2022-08-24T07:16:54Z</time>
                <ele>360.9833984375</ele>
                <extensions>
                    <dir>282.5</dir>
                    <g_spd>3.50999910126326</g_spd>
                    <h_acc>3.79009246826172</h_acc>
                    <v_acc>2.5</v_acc>
                    <gpxtpx:TrackPointExtension>
                        <gpxtpx:hr>129</gpxtpx:hr>
                    </gpxtpx:TrackPointExtension>
                </extensions>
            </trkpt>
            ...

GPX from Sports Tracker (I see that also TrackPointExtension is used, but the XML is in general more “dirty”):

<gpx xmlns="http://www.topografix.com/GPX/1/1" version="1.1" creator="Sports Tracker" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:schemaLocation="http://www.topografix.com/GPX/1/1  http://www.topografix.com/GPX/1/1/gpx.xsd http://www.garmin.com/xmlschemas/GpxExtensions/v3 http://www.garmin.com/xmlschemas/GpxExtensionsv3.xsd http://www.garmin.com/xmlschemas/TrackPointExtension/v1  http://www.garmin.com/xmlschemas/TrackPointExtensionv1.xsd"  xmlns:gpxtpx="http://www.garmin.com/xmlschemas/TrackPointExtension/v1"  xmlns:gpxx="http://www.garmin.com/xmlschemas/GpxExtensions/v3">
...
<trkpt lat="50.070905" lon="18.420838">
<ele>339.0</ele>
<desc>Speed 13.2 km/h Distance 0.01 km</desc>
<time>2022-07-09T09:41:23.35</time>
<name>2</name>
<sym>Dot</sym>
<extensions><gpxtpx:TrackPointExtension><gpxtpx:hr>122</gpxtpx:hr></gpxtpx:TrackPointExtension></extensions>
</trkpt>
...

If you need more data (or help) let me know.

@rinigus thank you for answer and for all your suggestions, for now your applications (offline maps + Pure Maps) are very helpful when I want to do quick search something on the map (without Internet access)

Regards,
R.

elBoberido · 4 February 2023 18:02

@Shocker Kuri maintainer here

I did some refactoring the last weeks and also changed the native file format to reduce the file size. I’m currently writing an importer for GPX and if you have a file from SportsTracker you can share I could take care that those imports also work. What would be great is a file with only a few track points and ideally with a pause and if Sports Tracker supports laps, then that would also help a lot.

Ahtisilli · 4 February 2023 19:43

Thanks for an excellent app!

I started using Kuri when I got GPS working reasonably well on my XA2. I also started using the Strava integration which fed the data to Finnish Kilometrikisa, a biking campaign from May to September.

I noticed that the file sizes were bigger than MeeRun’s but using GPX has a major advantage: the workout phases (bike ride to floorball practice + bike ride back home = daily bike ride) are easy to join in a text editor so there is no need to keep Kuri paused for hours, and actually Kuri crashes a lot more during pauses (not Kuri’s fault but Android apps cause lipstick crashes).

So if possible I’d like to be able to choose GPX as the file format for easy editing. One space saving solution would be to save the summary in a database, screenshot somewhere and zip the GPX data. Or for a custom binary format MeeRun-level crash recovery and GPX import and export would be OK (preferably with the Pause tags; MeeRun loses them in GPX export and the tag for import isn’t documented).

Improvements that would be nice:

monthly summaries like in MeeRun, and search or jump to date, collapse months to monthly summaries…
split Kuri into a virtually crash-proof tracking daemon and the UI, similar to Phonehook

elBoberido · 5 February 2023 13:09

@Ahtisilli I guess this is my xkcd:1172

Editing GPX files manually sounds horrible. Would you change your workflow when I add an option to merge an activity to the preceding one?

Crash recovery is one of the reasons for the new data format although it will most probably not be in the next major release. It makes things just easier by not having to work with xml. The new format is basically a compressed TOML file. There will of course be an option to export all activities to GPX at once.

Memory consumption should also be much better with the next release which should prevent the OOM killer to kick in.

Monthly summary like in MeeRun is also on my to-do list but will most likely come with a point release after the next major one. For the next release I just want to finish GPX import/export, nicer diagram view and activity merges.

Ahtisilli · 6 February 2023 09:35

It’s either search + replace (I lost a workout, copied a new or existing workout, change date and approximate time) or cut - copy - paste (combining parts of different workouts to recreate a lost one or remove stuff when I forgot to pause). The simple stuff can be done on the phone with nano.

For significant GPS errors (straight line or spaghetti) I use RouteConverter - routeconverter.com - on desktop Linux (Debian testing with LXQT). I add just the minimum amount of trackpoints to follow the route, ignore timestamps and add elevation to fit between the real data points.

That’s very tempting. Can you add adding pauses to finished workouts and removing the data in them? I once got a pretty weird workout: 10 km cycling in Tampere, a 160 km train ride to Helsinki and 25 km cycling in and around Helsinki…

Nice!

Great!

[quote]Monthly summary like in MeeRun is also on my to-do list but will most likely come with a point release after the next major one. For the next release I just want to finish GPX import/export, nicer diagram view and activity merges.
[/quote]

Yup, the engine room is a priority, and Kuri is already good enough for me.

Tagging pauses seems to be a problem between different apps and servuces. [/trkpt][/trkseg][trkseg][trkpt] seems like the best idea but in MeeRun export/import changes pauses to laps and the clock keeps running.

Letting the user choose automatic exported file naming would be nice - I would start with Sport-YYYYMMDD-weekday_abbreviation_in_Finnish-no_spaces…

I thought a bit about an archive mode - how to keep the workout summaries and map pictures but delete (or store offline) the raw GPS data. When the monthly summaries are available, there could be an icon to write-protect the workout summaries for the entire month. On the other hand, GPS data compresses about 95 per cent so disk space might not become a problem.

elBoberido · 8 February 2023 19:04

That’s very tempting. Can you add adding pauses to finished workouts and removing the data in them? I once got a pretty weird workout: 10 km cycling in Tampere, a 160 km train ride to Helsinki and 25 km cycling in and around Helsinki…

I think I can cover the easy case with the next release, i.e. merge to the preceding one. Adding pauses should also not be too difficult with the new design but would take some time, especially for the UI. I think for these cases you would need to export to gpx, edit and then import into the app.

Tagging pauses seems to be a problem between different apps and servuces. [/trkpt][/trkseg][trkseg][trkpt] seems like the best idea but in MeeRun export/import changes pauses to laps and the clock keeps running.

I have to investigate a bit more but currently I’m thinking of using the heuristic you mentioned for export with the same track point at the end of one lap and the beginning of the next one. For import I’m planing to treat pauses of 1 second as lap. At least for MeeRun gpx since it seems to do it this way.

Letting the user choose automatic exported file naming would be nice - I would start with Sport-YYYYMMDD-weekday_abbreviation_in_Finnish-no_spaces…

Currently not on my to-do list but if it does not take too much time I will implement it.

I thought a bit about an archive mode - how to keep the workout summaries and map pictures but delete (or store offline) the raw GPS data. When the monthly summaries are available, there could be an icon to write-protect the workout summaries for the entire month. On the other hand, GPS data compresses about 95 per cent so disk space might not become a problem.

The summary is already separated from the track data and one can delete the track data without problems. Of course the map and diagram will not work any more but the summary is still there.

elBoberido · 8 April 2023 14:31

There is now a v0.4.90 release on Kuri Testing | OpenRepos.net — Community Repository System. Except the missing local export to GPX this will eventually become v0.5. The export to GPX will be added soon. Please test. I refactored quite a lot but overall code quality should be better than before. Activity recording/loading is affected the most and the data is now stored in ‘.local/share/org.kuri/kuri’. This should make it possible to enable sandboxing in the future.

You can always go back to v0.4 and redo the migration of the activities when switching to v0.5.

elBoberido · 26 July 2023 18:25

I uploaded v0.4.91 which will eventually become v0.5.0 to Kuri Testing | OpenRepos.net — Community Repository System. There is now an import/export to GPX functionality.

If there are no big issues, I’ll release v0.5.0 next week.