Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-implement data generation for trainers #355

Merged
merged 3 commits into from
Feb 1, 2025

Conversation

lhearachel
Copy link
Collaborator

This PR replaces the single-file-processing Python scripts for trdata and trpoke with a batch processor in C++ that uses the ROM's struct-definitions for packing. Like with the species re-implementation, the list of known files to be packed is passed to the batch-processor via Meson's environment feature: https://mesonbuild.com/Reference-manual_returned_env.html#environment-env

The approach here hinges on breaking NAIX-generation away from NARC-construction. Because Meson is built on the premise static build-configuration, we can assert that a known set of files must exist at build-time, and that if this set were to change, then the configuration itself must change. This is sensible from a build-standpoint, as well, because other aspects of the game do not directly hinge on the exact contents of an archive; accessing a trainer from trdata.narc only requires that you know its numeric identifier.

It follows, then, that changes to existing trainer data files does not affect other modules (field scripts, ROM code, etc.), but breaking the contract of what data files are built into the trainer archives does affect those modules.

Potential Concern: Environment Variable Size

Some light reorganization of the structure tree has been done in anticipation of this concern (and for neater reorganization once more trainer-related archives are unpacked, e.g. sprites), but it is still worth bringing up.

To the best of the research that I have done, the maximum limit of an environment variable is:

The environment variable constructed is a list of file-stems separated by semi-colons. As a short measurement, I wrote some Python code to determine how big the environment variable currently is with the vanilla repository:

Python 3.12.4 (main, Jul  2 2024, 08:08:25) [GCC 14.1.1 20240522] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pathlib
>>> print(sum(map(lambda f: len(f.stem) + 1, pathlib.Path('.').glob('*.json'))))
17496

However, a caveat to this is that there are a number of trainer data files that are, to the best of our knowledge thus-far, wholly unused in the game. It is more likely, then, that a user would re-use one of these files than make a wholly new one; even then, we are still only cresting half of the theoretical limit mentioned above for Windows.

Benchmarks

Benchmarks below are taken from local builds on my system:

  • OS: Arch Linux x86_64
  • Kernel: 6.12.10-arch1-1
  • Shell: fish 3.7.1
  • CPU: AMD Ryzen 7 3700X (16 cores) @ 3.6GHz
  • RAM: 2x16GB

From Clean

Builds from clean see a 12-second time reduction on my machine (Arch Linux

# Before

> time make
...
________________________________________________________
Executed in  111.30 secs    fish           external
   usr time  481.49 secs  515.00 micros  481.49 secs
   sys time  336.57 secs  116.00 micros  336.57 secs


# After

> time make
...
________________________________________________________
Executed in   98.99 secs    fish           external
   usr time  383.42 secs  656.00 micros  383.42 secs
   sys time  314.64 secs    0.00 micros  314.64 secs

Incremental

> time make data
ninja -C build data
ninja: Entering directory `build'
ninja: no work to do.

________________________________________________________
Executed in  143.65 millis    fish           external
   usr time   87.62 millis    0.00 micros   87.62 millis
   sys time   55.73 millis  705.00 micros   55.02 millis

> touch res/trainers/data/ace_trainer_abel.json


# Before

> time make data
ninja -C build data
ninja: Entering directory `build'
[100% 1129/1129] Generating res/field/scripts/scr_seq.narc with a custom command

________________________________________________________
Executed in    2.67 secs    fish           external
   usr time   21.91 secs  597.00 micros   21.91 secs
   sys time   11.42 secs  146.00 micros   11.42 secs


# After

> time make data
ninja -C build data
ninja: Entering directory `build'
[100% 1/1] Generating res/trainers/datagen_trainer_out with a custom command (wrapped by meson to set env)

________________________________________________________
Executed in  210.03 millis    fish           external
   usr time  134.91 millis    0.00 micros  134.91 millis
   sys time   74.70 millis  793.00 micros   73.91 millis

@lhearachel lhearachel merged commit 5f63fd2 into pret:main Feb 1, 2025
2 checks passed
github-actions bot pushed a commit to lhearachel/pokeplatinum that referenced this pull request Feb 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant