Skip to content

Commit

Permalink
Merge pull request #30 from Aeliot-Tm/translation_api
Browse files Browse the repository at this point in the history
Implement usage of Google Cloud Translation API
  • Loading branch information
Aeliot-Tm authored May 14, 2021
2 parents 00512f9 + 8d1135d commit 1127cd8
Show file tree
Hide file tree
Showing 28 changed files with 809 additions and 79 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
CHANGELOG
=========

2.4.0
-----
* Implement usage of Google Translate API to translate missed keys.

2.3.2
-----
* Fix getting of project directory path for Symfony >=5.0
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ Full information about files transformation see [there](docs/lint/lint_yaml_comm
```
Full information about updating of YAML files see [there](docs/transform_yaml_files.md).

### Machine Translation via Vendor's API

Full information about machine translation see [there](docs/machine_translation.md).

**NOTE:** There used standard `\Symfony\Component\Yaml\Yaml` class for dumping, so it inserts single-word values without escaping.

---
Expand Down
11 changes: 7 additions & 4 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,16 @@
- [x] [Implement base testing of translations](docs/lint/lint_yaml_command.md):
- all files presented
- all variable filled for each language
- files have duplicated keys.
- [ ] Implement extended testing of translations:
- empty values detector
- files have duplicated keys
- all keys of translations match pattern.
- [ ] Implement extended testing of translations:
- empty values detector.
- [x] Make compatible with Symfony versions since 3.4 till 5.2.
- [ ] Extend support of translation files formats.
- [ ] Implement auto-translation via Google/Yandex API and so on.
- Implement auto-translation via vendors API:
- [x] Google Translate
- [ ] Yandex Translate
- [ ] and so on.
- [ ] Implement command "Make my project perfect :)".


Expand Down
1 change: 1 addition & 0 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
"require": {
"php": "^7.4",
"ext-json": "*",
"google/cloud-translate": "^1.10",
"symfony/config": ">=3.4",
"symfony/console": ">=3.4",
"symfony/dependency-injection": ">=3.4",
Expand Down
21 changes: 17 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,17 @@ First aff all, add root node `aeliot_trans_maintain:` to your config files (see

### Basic configuration:

There is displayed default values of any configuration.

```yaml
aeliot_trans_maintain:
insert_missed_keys: 'no' # Switch on/off decorator for the standard translator and define mode of inserting missed keys
translation_api:
google:
key: ~ # your key to the Google Cloud Translate API
limit: 500000 # limit of symbols per month. Can be null. Limit ignored if value is empty (0 or null)
yaml:
indent: 4 # Size of indents in YAML files
insert_missed_keys: no # Switch on/off decorator for the standard translator and define mode of inserting missed keys
```
#### Accepted keys for insert_missed_keys:
Expand All @@ -22,19 +28,26 @@ It is recommended to use values: "no" or "end".
### Usage of Environment variable:
Example (you can get more information in the [document](https://symfony.com/doc/current/configuration/env_var_processors.html)):
Example:
```yaml
# Add parameter TRANS_MAINTAIN_INSERT_MISSED_KEYS=end into .env.local to switch on translator decorator and clear cache
parameters:
env(TRANS_MAINTAIN_INSERT_MISSED_KEYS): no
env(TRANS_MAINTAIN_INSERT_MISSED_KEYS): 'no'
env(GOOGLE_TRANSLATE_API_KEY): ~

aeliot_trans_maintain:
insert_missed_keys: "%env(TRANS_MAINTAIN_INSERT_MISSED_KEYS)%"
insert_missed_keys: '%env(TRANS_MAINTAIN_INSERT_MISSED_KEYS)%'
translation_api:
google:
key: '%env(GOOGLE_TRANSLATE_API_KEY)%'
```
After that you can easily switch on/off translator decorator and inserting of missed translation keys by adding/changing of parameter
`TRANS_MAINTAIN_INSERT_MISSED_KEYS=end` in .env.local file in the project folder.

You can get more information in the [official document](https://symfony.com/doc/current/configuration/env_var_processors.html).


---
*[Read Me](../README.md)*
23 changes: 23 additions & 0 deletions docs/machine_translation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
Machine Translation via Vendor's API
===================================

There is implemented usage of Google Claud Translation.
Add credentials to the [configuration](configuration.md) for the usage of machine translation.
After that, you can execute command:
```shell
php bin/console aeliot_trans_maintain:yaml:translate --domain=messages --source_locale=en --target_locale=de
```
or short mode
```shell
php bin/console a:y:translate -d messages -s en -t de
```

Here:
- **domain** - one o several domains for translation. There may be several options.
- **source_locale** - locale there existing translations will be taken. Should be single.
- **target_locale** - locale with missing translations, and new ones will be inserted there. There may be several options.

**NOTE**! Exception will be thrown in target locale has no missed translations if compare it with source location.

---
*[Read Me](../README.md)*
46 changes: 10 additions & 36 deletions src/Command/ExportMissedTranslationsCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@

namespace Aeliot\Bundle\TransMaintain\Command;

use Aeliot\Bundle\TransMaintain\Service\Yaml\FilesMapProvider;
use Aeliot\Bundle\TransMaintain\Service\Yaml\KeysParser;
use Aeliot\Bundle\TransMaintain\Service\Yaml\MissedValuesFinder;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
Expand All @@ -14,55 +13,30 @@

final class ExportMissedTranslationsCommand extends Command
{
private FilesMapProvider $filesMapProvider;
private KeysParser $keysParser;
private MissedValuesFinder $missedValuesFinder;

public function __construct(FilesMapProvider $filesMapProvider, KeysParser $keysParser)
public function __construct(MissedValuesFinder $missedValuesFinder)
{
parent::__construct('aeliot_trans_maintain:yaml:export_missed_translations');

$this->filesMapProvider = $filesMapProvider;
$this->keysParser = $keysParser;
$this->missedValuesFinder = $missedValuesFinder;
}

protected function configure(): void
{
$this->setDescription('Command for the sorting of yaml files');
$this->setDescription('Export missed translations in YAML files');
$this->addArgument('domain', InputArgument::REQUIRED, 'Domain name');
$this->addArgument('locale_from', InputArgument::REQUIRED, 'Locale code from data will be taken');
$this->addArgument('locale_for', InputArgument::OPTIONAL, 'Locale code exported for. Used for filterring if passed');
$this->addArgument('source_locale', InputArgument::REQUIRED, 'Locale code from data will be taken');
$this->addArgument('target_locale', InputArgument::OPTIONAL, 'Locale code exported for. Used for filterring if passed');
}

protected function execute(InputInterface $input, OutputInterface $output): int
{
$domain = $input->getArgument('domain');
$localeFrom = $input->getArgument('locale_from');
$domainsFiles = $this->filesMapProvider->getFilesMap();

if (!isset($domainsFiles[$domain][$localeFrom])) {
throw new \InvalidArgumentException(
\sprintf('Invalid domain "%s" or locale "%s" posted', $domain, $localeFrom)
);
}

$parsedKeys = $this->keysParser->getParsedKeys($domainsFiles[$domain]);
$omittedKeys = $this->keysParser->getOmittedKeys($parsedKeys);
$allOmittedKeys = $this->keysParser->mergeKeys($omittedKeys);

$values = array_intersect_key(
$this->keysParser->parseFiles($domainsFiles[$domain][$localeFrom]),
array_flip($allOmittedKeys)
);
if ($input->hasArgument('locale_for')) {
if (!$filterKeys = $omittedKeys[$localeFor = $input->getArgument('locale_for')] ?? null) {
throw new \InvalidArgumentException(\sprintf('There is no omitted keys for locale "%s"', $localeFor));
}

$values = array_intersect_key($values, array_flip($filterKeys));
}

ksort($values);
$sourceLocale = $input->getArgument('source_locale');
$targetLocale = $input->hasArgument('target_locale') ? $input->getArgument('target_locale') : null;

$values = $this->missedValuesFinder->findMissedTranslations($domain, $sourceLocale, $targetLocale);
$output->writeln(Yaml::dump($values));

return 0;
Expand Down
2 changes: 1 addition & 1 deletion src/Command/LintYamlCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ public function __construct(LinterRegistry $linterRegistry)

protected function configure(): void
{
$this->setDescription('Command for the sorting of yaml files');
$this->setDescription('Check YAML files');
$this->addArgument('linter', InputArgument::IS_ARRAY, 'List of linters', [LinterRegistry::PRESET_BASE]);
$this->addOption('domain', 'd', InputOption::VALUE_REQUIRED | InputOption::VALUE_IS_ARRAY, 'Filter domains');
$this->addOption('locale', 'l', InputOption::VALUE_REQUIRED | InputOption::VALUE_IS_ARRAY, 'Filter locales');
Expand Down
151 changes: 151 additions & 0 deletions src/Command/TranslateMissedKeysCommand.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
<?php

declare(strict_types=1);

namespace Aeliot\Bundle\TransMaintain\Command;

use Aeliot\Bundle\TransMaintain\Exception\ApiLimitOutOfBoundsException;
use Aeliot\Bundle\TransMaintain\Service\ApiTranslator\Translator;
use Aeliot\Bundle\TransMaintain\Service\Yaml\BranchInjector;
use Aeliot\Bundle\TransMaintain\Service\Yaml\FileManipulator;
use Aeliot\Bundle\TransMaintain\Service\Yaml\FilesFinder;
use Aeliot\Bundle\TransMaintain\Service\Yaml\MissedValuesFinder;
use Aeliot\Bundle\TransMaintain\Service\Yaml\TransformationConveyor;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;

final class TranslateMissedKeysCommand extends Command
{
private BranchInjector $branchInjector;
private FilesFinder $filesFinder;
private FileManipulator $fileManipulator;
private MissedValuesFinder $missedValuesFinder;
private Translator $translator;
private TransformationConveyor $transformationConveyor;

public function __construct(
BranchInjector $branchInjector,
FilesFinder $filesFinder,
FileManipulator $fileManipulator,
MissedValuesFinder $missedValuesFinder,
TransformationConveyor $transformationConveyor,
Translator $translator
) {
parent::__construct('aeliot_trans_maintain:yaml:translate');

$this->branchInjector = $branchInjector;
$this->filesFinder = $filesFinder;
$this->fileManipulator = $fileManipulator;
$this->missedValuesFinder = $missedValuesFinder;
$this->transformationConveyor = $transformationConveyor;
$this->translator = $translator;
}

protected function configure(): void
{
$this->setDescription('Command for translation missed data');

$this->addOption('domain', 'd', InputOption::VALUE_REQUIRED | InputOption::VALUE_IS_ARRAY, 'Domain name');
$this->addOption(
'source_locale',
's',
InputOption::VALUE_REQUIRED,
'Locale code where data will be taken',
'en'
);
$this->addOption(
'target_locale',
't',
InputOption::VALUE_REQUIRED | InputOption::VALUE_IS_ARRAY,
'Locale code where are missed data'
);
}

protected function execute(InputInterface $input, OutputInterface $output): int
{
$sourceLocale = $input->getOption('source_locale');
$targetLocales = $this->getTargetLocales($input);

foreach ($this->getDomains($input) as $domain) {
foreach ($targetLocales as $targetLocale) {
$values = $this->missedValuesFinder->findMissedTranslations($domain, $sourceLocale, $targetLocale);
if (!$values) {
continue;
}

[$values, $isLimitReached] = $this->translateBatch($values, $targetLocale, $sourceLocale);

if ($values) {
$this->save($domain, $targetLocale, $values);
}

if ($isLimitReached) {
$output->writeln('[ERROR] API Limit reached');
}
}
}

return 0;
}

/**
* @return string[]
*/
private function getDomains(InputInterface $input): array
{
$requestedDomains = $input->getOption('domain') ?: [];
$existingDomains = $this->filesFinder->getDomains();

return $requestedDomains ? array_intersect($existingDomains, $requestedDomains) : $existingDomains;
}

/**
* @return string[]
*/
private function getTargetLocales(InputInterface $input): array
{
$requestedLocales = $input->getOption('target_locale') ?: [];
$existingLocales = $this->filesFinder->getLocales();

return $requestedLocales ? array_intersect($existingLocales, $requestedLocales) : $existingLocales;
}

private function merge(array $yaml, array $values): array
{
foreach ($values as $key => $value) {
if (!$this->branchInjector->inject($yaml, $key, $value)) {
throw new \DomainException(\sprintf('Cannot inject key %s', $key));
}
}

return $yaml;
}

private function save(string $domain, string $targetLocale, array $values): void
{
$path = $this->filesFinder->locateFile($domain, $targetLocale);
if ($this->fileManipulator->exists($path)) {
$values = $this->merge($this->fileManipulator->parse($path), $values);
}
$values = $this->transformationConveyor->transform($values);
$this->fileManipulator->dump($path, $values);
}

private function translateBatch(array $values, string $targetLocale, ?string $sourceLocale = null): array
{
$isLimitReached = false;
$translatedValues = [];
foreach ($values as $key => $value) {
try {
$translatedValues[$key] = $this->translator->translate($value, $targetLocale, $sourceLocale);
} catch (ApiLimitOutOfBoundsException $exception) {
$isLimitReached = true;
break;
}
}

return [$translatedValues, $isLimitReached];
}
}
Loading

0 comments on commit 1127cd8

Please sign in to comment.