[SIP-130] Migrating from Mapbox to MapLibre

[Aalbedon]

[SIP-130] Proposal for migrating from Mapbox to MapLibre

Motivation

With the Mapbox GL JS library becoming closed source and the existing map plugins available in Superset being based on the legacy data API, migrating to the MapLibre project (an open source Mapbox fork) should make the transition simpler while providing the opportunity to bring the map plugins into the fold of the new data framework.

Proposed Change

Create new map plugins using deck.gl (or perhaps kepler.gl?) and MapLibre GL JS (https://github.com/maplibre/maplibre-gl-js).
Although backporting code (that is not covered by the BSD 3-Clause license) from Mapbox GL JS to MapLibre GL JS is not permitted, the expectation is that the gap in functions/features required by Superset to implement MapLibre is not large (see API documentation: https://maplibre.org/maplibre-gl-js/docs/).

The new plugins would be implemented using TypeScript and the new data API framework.
Details on the specifics may have to be provided on a per plugin basis.

The MapLibre Style Specification (https://github.com/maplibre/maplibre-style-spec) may offer useful utilities for managing a MapLibre implementation.

Examples of what is possible with MapLibre GL JS are available here: https://maplibre.org/maplibre-gl-js/docs/examples/

New or Changed Public Interfaces

Plugins using Mapbox will need to be either modified, or (preferably) replaced. These will be discussed in subsequent SIPs.

Add new plugins/charts:

• MapLibre
• deckgl

Would deprecate:

• legacy-plugin-chart-map-box (https://github.com/apache/superset/tree/master/superset-frontend/plugins/legacy-plugin-chart-map-box)
• legacy-preset-chart-deckgl (https://github.com/apache/superset/tree/master/superset-frontend/plugins/legacy-preset-chart-deckgl)

New dependencies

MapLibre GL JS is licensed under the 3-Clause BSD license.

Migration Plan and Compatibility

It is unknown whether any database migrations are necessary.
However, by deprecating the Mapbox-based plugins, existing charts based on those would have to be re-implemented by users or preferably supported by a migration.

Rejected Alternatives

Various past discussions covered alternatives such as kepler.gl (https://github.com/keplergl/kepler.gl), Leaflet (https://github.com/Leaflet/Leaflet), and a deck.gl + Leaflet implementation (https://github.com/zakjan/deck.gl-leaflet).

Although not all of these were firmly rejected, MapLibre potentially offers a less cumbersome migration.

[birkskyum]

Disclaimer: i'm a board member in the maplibre non-profit org

Just a note on the alternatives - deck.gl works great with maplibre-gl, to the point that latest version of kepler.gl actually has migrated entirely from mapbox-gl to maplibre-gl, for the same licensing reasons as this issue point to.

It will be helpful when implementing a typescript superset map plugin for maplibre, that maplibre-gl, in contrast to mapbox-gl, is written entirely in typescript. I believe maplibre-gl currently has all the functionality required by superset, and it's for the most part a drop-in replacement - to that end, I'd be happy to help answer any question around maplibre-gl usage and technicalities around this migration.

Some other advantages that would apply here are:

advantages over mapbox-gl v1 or a fork of it

• The maintenance burden is moved to the MapLibre org. There is paid stable maintenance on the project, and also financial investments in new feature development, so PR's /issues won't be left for dead. All this is covered by the awesome MapLibre Sponsors who all benefit from the existence of MapLibre because it allow the cost to be shared instead of covering full maintenance/development of separate forks, and it keeps an industry wide compatibility.
• Lots of performance work has been done - notably microsoft has been helping out by profiling bing maps usage extensively on the millions of users there.
• maplibre has since v3 supported webgl2 which almost all users will be using at this point, and in the increasingly rare case it's not supported there is still a webgl1 fallback in place.
• there is a growing community around maplibre-gl, we've e.g. seen a 100% increase in downloads (now ~400k/weekly) just over the past half year.
• many companies are helping out with new features related to their respective domain of expertise.
• Continous support for emergent standards, like the new datasets from the Overture Maps Foundation as shown in this example.

here are contribution graphs, with the area approx. marked after the fork, to visually represent the scale of shared continuous investments which aren't present in mapbox-gl v1:

maplibre-gl ( 2000+ PR's with bug-fixes and improvements)

advantages over mapbox-gl v2+

• maplibre-gl and mapbox-gl v1 are rendering libraries where mapbox-gl v2 is a rendering service. Both can be paired with data/styles from local files or services, but the difference is that by staying a renderer library, maplibre itself is free to use, and will run without an api key. This has multiple advantages:
  • allows rendering offline, on-prem
  • gives better startup performance, because the code doesn't "calling home" to check the api key billing status
  • It's more convenient for new users, not to have to signup for mapbox and get an api key.
• The maplibre license permits removing the logo from lower left, in case superset users rather see more of their plots.
• In case you bundle the renderer with superset, that something you can certainly do with maplibre-gl since it's just a library.

[rusackas]

I would love to see this proposal / effort move forward. @Aalbedon do you want to move this forward with a [DISCUSS] thread on the Dev mailing list? Let me know if you need any assistance with that effort.

[rusackas]

Checking in to see if you plan to implement this and/or carry the process through to completion, @Aalbedon. If not, this will eventually be closed due to inactivity.

[birkskyum]

@rusackas , before closing it for inactivity, give me a ping. I don't mind championing the necessary work.

[rusackas]

It won't be closed for quite a while... I just want to make sure it's able to move forward. There are really two parts to push on:

1. Making sure the process is followed (i.e. running this SIP through the Discuss and Vote threads on the mailing list)
2. Implementing the feature.

I can certainly assist with Item 1, but would only want to do so if someone's going to follow through here. If you're game, to tackle implementation @birkskyum, I'll happily start on the more bureaucratic part ;)

[birkskyum]

That sounds great - I'm up for taking on the implementation part.

[rusackas]

@birkskyum do you think we should limit the scope of this SIP to the Mabbox -> MapLibre migration? Then after that, we can address a DeckGL plugin migration separately? The DeckGL one seems like a more difficult task, but if you're up for it, we can leave them combined :) People will probably want to know more details about how that effort will be tackled.

I'll open the [DISCUSS] thread now, and we can start fielding such questions :)

[rusackas]

I should also say that unless we have feature parity (I'm not sure about that, based on the initial proposal), this would be considered a "breaking change" and would have to be held until a major version of Superset. If it's a smooth transition, I don't think it'd need to wait, but I think users would want some form a migration script in either case. @michael-s-molina has been making several of these and may be able to help point you in the right direction for that stage of the effort.

[birkskyum]

I think it's a good idea to split up these tasks where possible.

[rusackas]

Ok... then I think in effect, this SIP should be considered a proposal to merely to replace Mapbox with MapLibre, on the basis of all the pricing/licensing/support/modernization virtues listed above.

We can open a separate SIP for more details about each of the plugins (any revisions to control panels, migrating to typescript, talk of drilling, etc etc etc) after this general product migration is settled.

I've slightly updated the original description to address this point.

[rusackas]

@birkskyum do you have any idea when you might be able to jump in for the implementation part? I can put this SIP up for a VOTE now if you think it's ready.

[birkskyum]

I can start already next week, and i think the SIP looks ready.

[JavierMorales2]

When could the implementation for the migration from Mapbox to MapLibre be ready?

[birkskyum]

@JavierMorales2 , I haven't deep dived the superset code yet, and I don't know how much different the new Data API framework is, but my guess, based on migrating plotly.js (python graphic tool) is that an initial implementation of a new plugin likely will take 2-3 weeks before it's functional, and that the following QA and release planning and preparation (docs, migration guides etc.) can require additional 1-3 months.

[rusackas]

Vote approved! Let's do the thing!!! :D @birkskyum let us know if you need any help along the way. I'll close the issue as it's approved, but continue to track it through the rest of the SIP kanban board.

[birkskyum]

@rusackas

I'm not sure about how to use the Kanban board - should I write here, or make new tickets in this repo with a [130] tag, or what's the preferred flow so I align?

First steps
I've had a good look at how to cut this work into smaller steps which could land independently, to approach the migration in a careful manner. What caught my eye is that react-map-gl, which is used in the legacy deckgl/mapbox plugins is v6.x (which is mapbox-only). Interesting, react-map-gl v7 (it was basically a rewrite) has support for both mapbox and maplibre, so upgrading that first would be a great preparation step before moving to maplibre - and it would potentially allow for publishing new releases of the legacy deckgl and mapbox plugins (same mapbox version) to secure that progress.

Unfortunately, I'm currently a bit stalled keeping my dev-environment up - this docker-compose issue (i think it's just that) seems to be affecting me:

• zstd not found by superset_node during docker compose up dev mode #30206

[victorfonseca]

@rusackas

I'm not sure about how to use the Kanban board - should I write here, or make new tickets in this repo with a [130] tag, or what's the preferred flow so I align?

First steps I've had a good look at how to cut this work into smaller steps which could land independently, to approach the migration in a careful manner. What caught my eye is that react-map-gl, which is used in the legacy deckgl/mapbox plugins is v6.x (which is mapbox-only). Interesting, react-map-gl v7 (it was basically a rewrite) has support for both mapbox and maplibre, so upgrading that first would be a great preparation step before moving to maplibre - and it would potentially allow for publishing new releases of the legacy deckgl and mapbox plugins (same mapbox version) to secure that progress.

Unfortunately, I'm currently a bit stalled keeping my dev-environment up - this docker-compose issue (i think it's just that) seems to be affecting me:

• zstd not found by superset_node during docker compose up dev mode #30206

Anyone on this?

[rusackas]

@birkskyum sorry for the radio silence. Ping me/us on slack if you ever get stalled here.

I don't think you need to add/track tickets for the work... feel free to open incremental PRs as you see fit.

as for the dev environment, I think you simply need to install zstd on your machine and then run Superset.

I also linked an PR where someone is adding an OSM layer to DeckDL... maybe we need to make sure things are a bit "agnostic" wherever possible, enabling choice/config to pick between tile layers (OSM, MapLibre, whatever). The more "pluggable" things are, the better! Then we might even be able to leave MapBox as a configurable/legacy option for those who want it.