Explorers: move to ETL as export steps

[lucasrodes]

Our landscape of explorers is very diverse. This is an attempt to study bringing all our public explorers (45) into ETL as export steps, to leverage our latest tooling.

An idea for this is to start backporting some explorers to ETL:

• 🎉 convert explorers from TSV to YML #4080

Explorer types

There are three general categories:

1. Some explorers rely on CSVs
2. Some others are built using charts from our site.
3. Some others are based on references to indicators.

However, within each of these, there are some features (one explorer may be a mix of these):

1. CSV-based

• Is it in ETL?
• Is it from GitHub? owid/owid-datasets, owid/importers, or another?
• Is there processing in owid-content?

3. Indicator-based

• Uses variable IDs
• Uses catalog paths
• Is it from ETL (export://) or manually created?

I've tried classifying all our explorers in this spreadsheet. I've added the number of explorer views and number of user views, which should help us prioritise any work here.

[pabloarosado]

Thank you @lucasrodes for this exploration (on explorers!). Given that it's unclear how we'll tackle the transition, maybe we can close this issue (but we keep it as reference). And during the next shaping session we can decide how to move forward.

[lucasrodes]

Hi @pabloarosado I rather keep this one open.

I'm currently working on TSV-to-JSON, which I think falls under the umbrella of this issue.

[pabloarosado]

All right @lucasrodes, thanks for rephrasing the description. Feel free to close it when you are done with the TSV-to-JSON, or let me know if you need any help.

[Marigold]

@lucasrodes FYI this code by Lars could be useful (we talked about it yesterday). It parses TSV files and saves them as JSON to MySQL. It could also be useless though ,and we might want to use an entirely different structure, so don't feel any pressure.