Organic Maps
Weblate for localizing app UI string resources #4515
Replies: 11 comments 38 replies
-
Can you import new translated strings as part of making beta release. And export new strings to translate at this step? At least that is how StreetComplete is doing this. See streetcomplete/StreetComplete@a5c93a9 that updates app metada, iD preset data, translations... BTW, StreetComplete is using POEditor which has nice GUI for translators, personally I dislike Transifex one and do not remember using weblate. But POEditor developers are not responsive at all and for example getting translator credits relies on a really hacky tool ( streetcomplete/StreetComplete#629 ). https://wiki.openstreetmap.org/wiki/Translation#Editors_and_other_contributing_tools - Every Door and OsmAnd are using weblate. So I bet that Weblate works. Maybe you can ask them how well ot worked? |
Beta Was this translation helpful? Give feedback.
-
The problem is that current translation tooling overwrites all android/apple files from
Thanks for more examples. I was too lazy to list all 2000+ users of Weblate.
I don't have motivation to use proprietary tools when good open-source alternatives are available. |
Beta Was this translation helpful? Give feedback.
-
Why it is a problem? https://github.com/organicmaps/organicmaps/blob/master/tools/unix/generate_localizations.sh has no clear docs but it seems to pull data from from strings files and generate android/iphone/jquery localization files. So weblate could import (new strings to translate) and export (all translations) to strings files. And android/iphone/jquery localization files be generated from these. Or have I misunderstood how things are going? |
Beta Was this translation helpful? Give feedback.
-
|
Weblate doesn't support the strings files format used by OM. |
Beta Was this translation helpful? Give feedback.
-
|
What is the incompatibility? Why it cannot be converted? Is it about some special characters within strings that have special meaning or is it about general file structure? Or is single message decomposed into a complex array of different related strings? |
Beta Was this translation helpful? Give feedback.
-
|
The biggest incompatibility is likely that our strings file contains all the languages in the single file, while all Weblate-supported formats are one lang per file. |
Beta Was this translation helpful? Give feedback.
-
|
Both translation methods are suitable for me. If it's possible to edit translations in both platforms at the same time, Weblate will be helpful for translators not used to GitHub. And probably, more translators will contribute Organic Maps. It is more comfortable for me to translate via GitHub. But I also contribute to many projects in Weblate, Crowdin and Transifex. So I can also work with this localization platforms. And I'd like to check new translations from Weblate for Turkish, since I know many mistakes in OsmAnd localization (it's also available in Weblate). If it can mention or request review from me, I can correct the translation easily. |
Beta Was this translation helpful? Give feedback.
-
|
thank you @rtsisyk . Yes, Weblate is the best way for good and consistent translations. Since Weblate 4.15, you can also manage teams directly in Weblate. Translators, reviewers and coordinators. This way you can also eliminate bad or pointless translations. Michal and Orangesunny are always willing to help if there are still technical problems. |
Beta Was this translation helpful? Give feedback.
-
I'm not familiar with Weblate yet but have 2 questions as a translator (intending to start with Hindi translations for the app UX) :
Appreciate any help / pointers here on how to contribute in this space. |
Beta Was this translation helpful? Give feedback.
-
No, please continue in strings.txt. Do you need any help?
"Add translation" button on Weblate will work. |
Beta Was this translation helpful? Give feedback.
-
Thanks for asking. Will reach out as needed. Strings.txt format looks simple enough. |
Beta Was this translation helpful? Give feedback.
-
|
I just used Weblate (for the first time) for adding Hindi translations for app-description on F-droid and app-update description in Google Play Store. Experience was pretty smooth, but hope I did not make changes in the wrong places. For example, I looked up the strings, added Hindi translations (and fixed a few), clicked the "Save and continue" button - and moved along: Two questions / comments:
cc: @rtsisyk |
Beta Was this translation helpful? Give feedback.
-
Yes, just add translations. GitHub PRs are generated every 24h. Your changes have been merged already (see 9bc1281). |
Beta Was this translation helpful? Give feedback.
-
|
We should move to Weblate for sure. But doing it for iOS/Android UI only and leaving all strings used by core (types translations, sounds, search categories) behind... Maybe we should research in more detail an idea to move from |
Beta Was this translation helpful? Give feedback.
-
|
You could have some linter that would verify that translations match as needed, and it could be e.g. integrated into the CI to block such pull requests from being merged. |
Beta Was this translation helpful? Give feedback.
-
|
categories.txt and types_strings.txt are out of scope of this proposal. |
Beta Was this translation helpful? Give feedback.
-
ATM |
Beta Was this translation helpful? Give feedback.
-
Sorry, I meant categories.txt. They have strange format which is not even covered by current tooling. |
Beta Was this translation helpful? Give feedback.
-
|
As @pastk mentioned above, at the moment, without refactoring, some strings/changes from categories.txt should match types_strings.txt exactly, otherwise, the search won't work. |
Beta Was this translation helpful? Give feedback.
-
TBH lately its many more Weblate commits in the git log :) |
Beta Was this translation helpful? Give feedback.
-
|
Website/F-Droid/AppStore commits are configured to be grouped by author:
There are plenty of configuration options on when and how to push. |
Beta Was this translation helpful? Give feedback.
-
|
Adding Weblate support would be way easier if we refactor categories.txt first, as mentioned here: #2780 (comment) The idea is simple: categories.txt should not have any duplicate strings matching types_strings.txt, they all should be loaded from the types_strings.txt file. All translations should be done by someone who understands the context and should be reviewed, to avoid breaking the code or breaking the UX. That's why keeping the workflow with the developer/Github Translator team reviews is important. |
Beta Was this translation helpful? Give feedback.
-
|
related discussions about |
Beta Was this translation helpful? Give feedback.
-
|
I'm working on localization for sounds.txt as part of my TTS street names PR, however I want to be clear that I 100% support moving to industry standard i18n formats and existing tooling to encourage contributions. My locale file viewer is 90% intended to help visualize contextual examples of formatter strings like "Take exit $1 onto $2 towards $3" that get really complex and confusing really fast, and I don't plan to make it a fully functional Weblate replacement. It's also trivial to swap out the backend for JSON/XML language files; I wrote it initially for those files and only swapped it to sounds.txt to accomodate the status quo. @biodranik had some good feedback in chat about things we want to ensure pass QA before they make their way into
He also wants to:
Finally: I'm giving a green light to try the weblate approach with two important conditions:
P.S. Don't forget to review and merge existing localization PRs to avoid wasting our contributor's time. P.P.S. Please don’t get me wrong. Until now I spent a lot of time on reviewing/polishing/merging translations from contributors. Any solution that simplifies that process would be awesome. While any other solution that increases my time spent on translations would be an additional burden that I want to avoid. |
Beta Was this translation helpful? Give feedback.
-
|
I've reviewed the PoC: https://hosted.weblate.org/projects/organicmaps/android/ |
Beta Was this translation helpful? Give feedback.
-
|
iOS doesn't look processed: 
Compare with the Android: 
|
Beta Was this translation helpful? Give feedback.
-
|
Very promising! How does it work with same strings used both in Android and iOS? Are they linked, or both projects has to be translated? (I just did a small change in iOS to test as well) |
Beta Was this translation helpful? Give feedback.
-
|
Also, how it will be with same strings as EN (ex.: drive-through, same in many languages), same strings related to PT/PT-BR and ES/ES-MX for example? The usual way in these services is to duplicate everything, so the translation is not "empty". |
Beta Was this translation helpful? Give feedback.
-
|
EN is used everywhere by default if other translations are missing. This is a preferred, easier to support behavior. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Everyone, this work looks really interesting. Quick intro. I have previous experience as a developer of localisation tools, managing translations for many languages in South Africa, looking at i18n issues across Africa and other parts of the world. I'm contracted by NLNet to assist projects that they fund in any localisation related issues. I've had a few chats with @rtsisyk and have experimented with him looking at using some of the native file formats directly. Moving to Weblate and simplifying the l10n process seems like a win to me, there's a bit of process involved but ultimately I think both translators and developers win. Interestingly, a lot of the concerns raised in https://github.com/orgs/organicmaps/discussions/4515#discussioncomment-6858466 would be caught (not always) earlier by a tool like Weblate instead of wasting developers time. |
Beta Was this translation helpful? Give feedback.
-
|
Hello Dwayne, welcome! I was surprised to discover that you are one of the two original authors of https://github.com/translate/translate and a contributor of South African languages to OpenOffice.org/LibreOffice, Mozilla, KDE and other cool projects. I have no doubt that you will bring decades of your valuable experience to our project. |
Beta Was this translation helpful? Give feedback.
-
|
Let's form a working group to explore possibilities. As far as I remember, @zyphlar previously shown interest in improving translation tools. @matheusgomesms, @nautilusx, @pastk what is about you? We can discuss here, on GitHub, or join a chat room. I can see 59 people in |
Beta Was this translation helpful? Give feedback.
-
|
Count on me. Not a dev, but translator of many apps. |
Beta Was this translation helpful? Give feedback.
-
|
@dwaynebailey welcome onboard! There are many issues with Weblate we discovered during our experiments, hope you've read about them here and in other related discussions on Github and in our Telegram/Matrix. From our experience, the highest quality translations come from active Github contributors/active OM users (you can ask to be included in our translations team for specific language by the way). Allowing anyone to translate without context is great, but it led to some bad results with our website and app store translations using Weblate, so here we need a quality review. The main benefit of GitHub is a quality review by many pairs of eyes which helps to prevent typos, mistakes, bad words, etc. It would be a pity to lose this benefit. The main benefit of Weblate and some other translation tools is the ability to know which translations are outdated and need some attention. Several independent translation areas can be improved:
More details:
To sum up: we're building an alternative to Google and Apple Maps for a billion users. If a new tool can provide a better user and contributor experience, then it is preferred over using not-so-good existing tools. |
Beta Was this translation helpful? Give feedback.
-
|
Btw, adding a plugin to Weblate that can read our twine strings and generate them back from Weblate would likely solve most of all mentioned issues/cases. |
Beta Was this translation helpful? Give feedback.
-
|
It would be fair to mention that the existing strings.txt-based system creates extra hassle for developers. I can see "It should be in strings.txt" comment in the every PR from newcomerts that changes strings. Recent examples:
This list is endless... It is pretty obvious that Android developers are required to learn this maps.me's homebrew translation system instead of using standard Android tooling. The theory that maps.me system is more convenient for developer than standard Android tools doesn't hold water. |
Beta Was this translation helpful? Give feedback.
-
|
Note that we are developing software that also works on iOS (and other OSes), and reuses a lot of common strings. The current approach works efficiently for those who know how to use it. Placing a clearer link to documentation may be helpful. Another alternative could be to fix the proposed translation properly instead of discussing it. |
Beta Was this translation helpful? Give feedback.
-
The problem is that it severely restricts translations to those who can deal with pull requests, coding and github. See #10109 |
Beta Was this translation helpful? Give feedback.
-
I would like to restart discussion about adopting Weblate for localizing app UI string resources.
I have a feeling that this project has very significant gap in localization of app UI string resources (Android/iOS). See funny examples what we have currently in #4499. People asks about better translation tooling too often:
The question is: What is preventing us from switching localization of Android && iOS application strings to Weblate?
👍 if you just agree that Weblate is a viable tool for translating Android and iOS applications.
👎 if you don't agree, and please provide elaborate in comments.
Disclaimer: nobody promises direct democracy here, but it would be nice to hear from @organicmaps/translations and @organicmaps/contributors at least. Weblate adopters (and haters, if any), please weigh in! Add your constructive feedback.
Boring text below.
Table of Contents
Existing Solution
Quoting a relevant section of TRANSLATION.md:
The existing solution is based around the idea of having the single file that can fit all translations to all languages for all parts of the app. This file is used as a source of truth to generate Android and iOS localization, as well as internal stuff for C++ core. Instead of editing Android's
.xmlresources and iOS's.stringsfiles, developers and translators can add localizations in the single place and generate corresponding files by using a tool.The tool is called twine. This is a third-party solution created in the early days of mobile app development. Organic Maps uses a forked version of this tool because the original tool doesn't handle plurals and other corners cases properly. A special
./tools/unix/generate_localization.shshell scripts exists to cover all nuances and produce Android and iOS resources out of strings.txt.The idea looks solid on the paper, but has some downsides on practice. The devil is in detail, you know. Let's see how it works from different angles.
From translator's prospective
Workflow
strings.txt is huge (1M), therefor GitHub usually doesn't display the content of this file in Web interface. It is not possible to edit this file on GitHub and contribute. The only one way is to clone the entire repository (10G+ and growing, cloning takes hours), edit the file locally, commit changes and create a GitHub Pull Requests. As a translator, you need to learn how to use
git, how to clonegitrepositories, how to makegitcommits and how to create GitHub PRs. Not to mention, that file is huge and not every text editor can handle it properly.vimandvscodeprobably will fly, but other tools may suffer. Anyway, as a translator you need to clone repo, edit files, commit changes to GitHub.Pros
Cons
git clone,git add,git commit, etc... not everyone in the world is a software developer!gitrepo (10G+ and growing) takes time.... a lot of time.From developer's prospective
Workflow
Using Android as an example, since I am more familiar with this topic:
android:textattributes of the layout to see how it looks like.res/values/strings.xml.strings.txt(JetBrains is super slow).res/values/strings.xmlintostrings.txtwithout messing.tools/unix/generate_localizations.shto overwriteres/values/strings.xml.tools/unix/translate_categories.shwhich is actually generates Google Translate for strings.txt as you can guess from the name.strings.txtediting Arabic and Hebrew RTL to fix missing placeholders for formatting.Pros
Cons
res/values/strings.xmland every iOS developers knows how to update.strings, but none of them know how to work with magnificentstrings.txtapproach.From maintainer's prospective
As a maintainer, your most difficult challenge that you need to deal with endless translation changes coupled together with changes of the code. One string in the code produces at least 100 strings in translations in the existing system. You have a big hammer to punish people if they don't use separate commits for "[strings] Regenerated" to avoid endless conflicts. Probably a one third of commits in the repo are endless "Regenerated".
Problems
I have been told by authors that this system is perfect and mankind will never get anything better in this century. Probably. There is just one small problem: translations are not contributed by translators.
Case 1
One example. On 2022-07-09 @arnaudvergnet added a bunch of new strings to the project (thanks, Arnaud!). Arnaud was very persistent, made 105 iterations and finally figured out how to use
tools/unix/translate_categories.shtogether withtools/unix/generate_localizations.shto generate content forstrings.txt:Today is 2023-02-17. After 7+ months, we can see the following:
OK, 38 languages were generated by Arnaud from Google Translated and only 4 were reviewed later after 7+ months. Maybe this string is boring and nobody cared about. However, Website on Weblate has been fully translated to 15 languages and 4 languages are not finished yet. We have people who can translate at least 20 languages. They just don'tt bother to translate app UI string resources by using tools and approaches provided.
The string above is not alone. I remember that I added "Import bookmarks" at the beginning of the project two years ago, and it stayed completely untranslated for a YEAR. I took a brief look at
strings.txtand realized that with small exceptions, strings were not revised since MAPS.ME times.Case 2
One more example. @jaivsh volunteered to contribute Hindi translations. I sent him a link to instructions how to translate website via weblate (https://github.com/organicmaps/organicmaps.github.io/blob/master/TRANSLATIONS.md) and instructions on how to translate app UI string resources (https://github.com/organicmaps/organicmaps/blob/master/docs/TRANSLATIONS.md) via
strings.txt. Believe you or not, but we got fully translated organicmaps.app website in Hindi on the next day! Maybe somebody needs to review translations, but I super impressed. This is the real power of open source community.Guess what happened with strings.txt? That PR (#4266) is still going back and forth in the last month. People extracted translations from
strings.txtto Google Spreadsheet to collaborate! It is still not finished, despite having significantly fewer words than organicmaps.app website.Real troubles
There are more funny issues. Arabic, Farsi and Hebrew translations in strings.txt are just utter crap:
trash.txt:
Do you see "OpenStreetMap keyword" in text? I don't. Let's try to translate back to English:
I am ashamed to see that we have such garbage in this app.
Author's Considerations
My subjective feeling that existing solution is OK-ish for developers. Devs are already familiar with
gitand editing 1M file is not a big deal for them. Yes, you need to learn some new crap instead of doing what you usually do during Android and iOS development. Yes, 10+ steps instead 3. Annoying, yeah. But tolerable. Not to mention that developers don't want to care about translations and want to write code, but this system forces them to spend time in the areas there they are not really competent.For translators, the existing system obviously doesn't fly. This is just a fact, based on the number of translations contributed in strings.txt and website. Maybe it was good before better tools like Transifex and Weblate entered the market, but numbers speak louder than any words. We should address this issue to get better translations. Including languages like Arabic, Farsi, Hebrew and dozens of Indian languages that were never covered by our predecessors.
Alternatives considered/discarded
Keep both systems in parallel
The workflow is not clear. Some people will edit
strings.txt, some people will contribute on Weblate and then both parties will collide during the run of./tools/unix/generate_localization.sh. The tool will just overwrite everything, nullifying any changes made outsidestrings.txt. Not good... Weblate itself can tolerate edits of Android/iOS formats via GitHub with some limitations, but it has no idea that .xml/.strings are regenerated from third source.Twine has a rudimentary support for re-generating
strings.txtback from generated translation files. I have tried that. It can't get stable round-trip without constantly changing both source and target strings. I generate Android strings.xml from strings.txt and when try to updatestrings.txtback from Android. The resultingstrings.txtis different. Formatting is different. The order of strings is different. Believe you or not, values were different.I spent some time on this issue and realized that significant investments (months) are needed at least to fix the round trip. It will be a new tool which looks like Twine. It is also worth mention that MAPS.ME and now this project use a forked version of Twine with some additional... hacks. For example, original Twine doesn't support plurals properly. Does it make sense to invest time into died tool to support a weird use-case that will not work properly in any case? There should be the only one source of truth, not two.
Add support for
strings.txtformat in WeblateThis approach is a bit challenging. I spent some time on this idea and discussed with the upstream. The most, if not say all, translation formats are designed to have one file per language. Weblate is also designed to have one file per language. Yes, Weblate does support about 40 formats, but it looks like that all of them use one file per language, not one huge file that stores all languages. Maybe authors of 40 formats just need to educate themselves in Twine.
It is difficult to give an estimate of complexity for this case because it is not just one more format in addition to the existing 40, it is some significant changes on the internal Weblate architecture. My rough estimation here that at least 1-2 man-months of Python engineering time are required if upstream would blindly accept all changes, which is unlikely. Communication will easily extend it to 6+ months.
Use another intermediate format
Yes, it is possible to use some well-known translation format support by Weblate to generate
strings.txtto generate Android string resources and Apple iOS strings. An intermediate format to generate an intermediate format. Intermediates formats all the way down. Sounds like overengineering.Write our own translation tool
Yeah, this is undoubtedly what we are missing in this project. Let's write our best translation tool. Transifex and Weblate are in danger!
Train translators
Yes, absolutely! We definitely can do the better job here. For example, run programming courses and train translators how to commit changes of the 1M file into
gitrepository by using mobile phone.Do nothing
This is the current "default route". I am not satisfied with the current quality of translations and this is a reason why this post is written.
Proposed Solution
The proposed solution is pretty straightforward:
rm -rf data/strings/strings.txtandtools/unix/generate_localizationas well astools/twine/**actually, the existing system covers multiple areas, including app UI string resources, sound resources, map types strings and others. This proposal is about app UI string resources ONLY. Other strings can be kept as is for now, and we will deal with them later. For example, sounds strings are already in JSON translation format that is supported by Weblate and there are no actions needed other than adding that directory to Weblate. Please focus on the primary problem, which is Android and iOS UI strings.My estimation is a couple of weeks. The proof of concept (PoC) was made in the past. All problems are already known. Again, only Android and iOS are in-scope. Other things can continue to use old tooling for some time, this is not a big deal for now.
Why Weblate
Weblate is the strongest candidate in this area. Weblate is used by such projects as Fedora, FreeBSD, Libreoffice, Tor, Godot Engine, Odoo, F-Droid, OsmAnd and others, I am just too lazy to continue qua^WDuckDuckGoing.
According to https://github.com/WeblateOrg/weblate, Weblate is used by over 2500 libre projects and companies in more than 165 countries. Weblate team is super responsive and very helpful. I filed several tickets in the past and most of them were fixed on the same day. Trust me, you will not get the same level of support from enterprise bro even for the vast amount of money. Nobody fixes problems such quickly as Weblate! Even Organic Maps. Period. Kudos to Michal and entire Weblate team.
We already ran very successfully Weblate migration project (https://github.com/orgs/organicmaps/discussions/4076) for organicmaps.app website. Something like 15 languages are already translated and 4 or 5 are in progress. See https://hosted.weblate.org/projects/organicmaps/website/
Numerous people, including active members of @organicmaps/translations teams, have expressed their interest in this topic in the past:
👍 if you just agree that Weblate is a viable tool for translating Android and iOS applications.
👎 if you don't agree, and please provide elaborate in comments.
Disclaimer: nobody promises direct democracy here, but it would be nice to hear from @organicmaps/translations and @organicmaps/contributors at least. Weblate adopters (and haters, if any), please weigh in! Add your constructive feedback.
Beta Was this translation helpful? Give feedback.
All reactions