tx client fails on push multiple files using -a option

[Mi605]

Bug-Report:

Scope:

• TX Client, version=1.6.5

Focus:

• tx push -t -a

Issue:

• Instead of pushing all languages as requested by the option -a, the transifex client pushes again and again the same few languages until it has pushed the count of languages which should be pushed. Hence at least half of the translations up to letter z are not transferred to transifex and need to be pushed one by one manually.

Example of a push run:

$ tx push -t -a
# Getting info about resources

xxxxxxxxx-testing.test-files - Done
[##############################] (1 / 1)

# Pushing translations

test-files [af] - Done
test-files [af] - Done
test-files [af] - Done
test-files [af] - Done
test-files [am] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [af] - Done
test-files [am] - Done
test-files [ar] - Done
test-files [be] - Done
test-files [bg] - Done
test-files [bn] - Done
test-files [bs] - Done
test-files [ca] - Done
test-files [ceb] - Done
test-files [co] - Done
test-files [cs] - Done
test-files [cy] - Done
test-files [da] - Done
test-files [de] - Done
test-files [el] - Done
test-files [eo] - Done
test-files [es] - Done
test-files [et] - Done
test-files [eu] - Done
test-files [fa] - Done
test-files [fi] - Done
test-files [fil] - Done
test-files [fr] - Done
test-files [fr_BE] - Done
test-files [fy] - Done
test-files [ga] - Done
test-files [gd] - Done
test-files [gl] - Done
test-files [gu] - Done
test-files [ha] - Done
test-files [haw] - Done
test-files [he] - Done
test-files [hi] - Done
test-files [hr] - Done
test-files [ht] - Done
test-files [hu] - Done
test-files [hy] - Done
test-files [id] - Done
test-files [is] - Done
test-files [it] - Done
test-files [ja] - Done
test-files [jv] - Done
test-files [ka] - Done
test-files [kk] - Done
test-files [km] - Done
test-files [kn] - Done
test-files [ko] - Done
test-files [ku] - Done
test-files [ky] - Done
test-files [lb] - Done
test-files [lo] - Done
[##############################] (105 / 105)
$

[Mi605]

Meanwhile I believe this issue is connected with #174 . Actually tx client probably doesn't retransfer all the resources again and again as the console output suggests. When using the | cat workaround to avoid the massive console spam it turns out there actually was a typo in one of the translation files causing the upload to stop after lo language. This piece of information was simply disguised by some thousands of "done" messages, causing console buffer overrun. Nevertheless It shows 105 /105 in the last line, suggesting all languages have been uploaded properly, while not only the one with the typo was not transferred, but also all the ones in alphabet after this. I only accidentally noticed that all languages after lo were not updated when looking into the resource in transifex web UI some days later.

Additional info: The typo causing the behaviour described was a set of duplicate ASCII double quoting in one of the entries within a .strings type resource:

"key" = ""string …"";

instead of

"key" = "string …";

[Mi605]

Additional finding: On this error tx client outputs exit code 1 (one) properly, so I could have recognised immediately not all files have been uploaded, even when error message lines were disguised in console, simply by checking echo $? immediately after execution was finished.

[kbairak]

Ok, the code for this is a bit complicated so I'm not sure I will be able to come up with a satisfactory solution. One can be found, I'm sure, but it would be a big investment. Here are my findings.

1. Sometimes, uploads are going to fail and it is not going to be the CLI's fault. It can simply be that a local language file has a syntax error (as in your case)

2. The default behavior in such cases is for the CLI to abort

3. Because in Go, you can't terminate a thread (goroutine) that has started from "the outside", aborting means that all tasks that have started will get to finish, but new tasks will get skipped, ie finish before they do actual work

4. There is a --skip flag built for this purpose. If it is set, the CLI will try to push all language files (no abort)

5. Because of the real-time output, figuring out what has happened is not as straightforward as it would seem:

   • One option is to pipe the output to grep (this will disable the real-time output like when doing |cat) and counting occurrences of "Done"

     ➜ tx push -taf | grep Done | wc -l
     5  # A syntax error stopped the process
     ➜ tx push -taf --skip | grep Done | wc -l
     19  # A syntax error caused one file to fail but the rest were pushed

   • The other is to inspect the output by hand

6. The internals of Go and the way we implemented the "worker pool" concept, make it hard to have the final appearance of the progress bar reflect the number of tasks that finished successfully. So, for now, we will defer to the options outlined in 5 in order to get a picture of what went or didn't go well with the tasks

[kbairak]

Hey @Mi605 ,

So, it turns out that I was able to implement this (see the attached PR). If the rest of my team agrees that it is a valuable addition and it passes code review, this will be added in the next release.