Refactor the core build pipeline

[lhearachel]

Overview

This contains some key edits. Namely:

1. All script-utilized constants have been migrated from constgen generation to metang generation.
2. Now that all of our constants use C-preprocessor semantics, we can generate depfiles from the script files. This gives Ninja a more granular view of what individual scripts depend on what.
3. The core build pipeline now operates in two stages: data and rom. The data stage is effectively a phony-target that encapsulates all packaged filesystem archives and generated code. The rom stage builds the game's code and packages it into the final ROM image. This pipeline split is due to what appears to be a bug in Ninja's detection for generated-headers when they are build as one of multiple outputs of their generating build target.
4. Our postconf script modifies the dependency-class of generated constants used by all script files to be order-only. This means that Ninja only cares that the constants are generated at all before proceeding to process any script files; the depfiles generated by gcc tell Ninja which headers these files actually care about.
5. Packing pl_msg.narc no longer generates an index-header. Instead, we now make use of a static ordering that a user will have to edit whenever they want to edit a new file, so that underlying accessors to the NARC know that they must be re-compiled. This, functionally, is similar to needing to edit the files listing for the pre-refactor build procedure, but it means that changing text archives by themselves will not necessitate rebuilding many targets which do not reference the edited banks.

Test Cases

To verify my changes, I touched a single data file of each of the major core component types and verified the dependency chain determined by Ninja. All of the below logs begin from a state where make yields a no-op build.

Pokemon Data Files

The recompiled files for overlays 5 and 113 are related to move tutors and footprint data, respectively.

> touch res/pokemon/zapdos/data.json
> make
meson configure build -Dgdb_debugging=false
ninja -C build data
ninja: Entering directory `build'
[  5% 1/19] Generating res/pokemon/shinzukan.narc with a custom command (wrapped by meson to set env)
[ 10% 2/19] Generating res/pokemon/pl_pokezukan.narc with a custom command (wrapped by meson to set env)
[ 15% 3/19] Generating res/pokemon/datagen_species_out with a custom command (wrapped by meson to set env)
[ 21% 4/19] Generating res/pokemon/zukan_data.narc with a custom command (wrapped by meson to set env)
[ 26% 5/19] Generating res/pokemon/zukan_data_gira.narc with a custom command (wrapped by meson to set env)
[ 31% 6/19] Generating res/pokemon/gen_species_headers with a custom command (wrapped by meson to set env)
[ 36% 7/19] Generating res/text/pokedex_text_banks with a custom command (wrapped by meson to set env)
[ 42% 8/19] Generating 'res/text/pl_msg.narc.p/species_names'
[ 47% 9/19] Generating 'res/text/pl_msg.narc.p/species_name_number_3'
[ 52% 10/19] Generating 'res/text/pl_msg.narc.p/species_weight_gira'
[ 57% 11/19] Generating 'res/text/pl_msg.narc.p/species_name_number_2'
[ 63% 12/19] Generating 'res/text/pl_msg.narc.p/species_weight'
[ 68% 13/19] Generating 'res/text/pl_msg.narc.p/species_name_number_1'
[ 73% 14/19] Generating 'res/text/pl_msg.narc.p/species_names_with_articles'
[ 78% 15/19] Generating 'res/text/pl_msg.narc.p/species_category'
[ 84% 16/19] Generating 'res/text/pl_msg.narc.p/species_height'
[ 89% 17/19] Generating 'res/text/pl_msg.narc.p/species_height_gira'
[ 94% 18/19] Generating 'res/text/pl_msg.narc.p/species_dex_entry'
[100% 19/19] Generating res/text/pl_msg.narc with a custom command
ninja -C build pokeplatinum.us.nds
ninja: Entering directory `build'
[ 25% 1/4] Compiling C object main.nef.p/src_overlay005_ov5_021F77A8.c.o
[ 50% 2/4] Compiling C object main.nef.p/src_overlay113_ov113_02260620.c.o
[ 75% 3/4] Linking target main.nef
[100% 4/4] Generating pokeplatinum.us.nds with a custom command
meson test -C build
ninja: Entering directory `/home/rachel/code/git/pokeplatinum/build'
ninja: no work to do.
1/3 SBIN Checksums       OK              0.02s
2/3 Filesystem Checksums OK              0.07s
3/3 ROM Checksum         OK              0.08s

Ok:                 3
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /home/rachel/code/git/pokeplatinum/build/meson-logs/testlog.txt

Message Banks

Source code files which make use of the encoded text-bank must always be recompiled.

> touch res/text/options_menu.gmm
> make
meson configure build -Dgdb_debugging=false
ninja -C build data
ninja: Entering directory `build'
[ 50% 1/2] Generating 'res/text/pl_msg.narc.p/options_menu'
[100% 2/2] Generating res/text/pl_msg.narc with a custom command
ninja -C build pokeplatinum.us.nds
ninja: Entering directory `build'
[ 33% 1/3] Compiling C object main.nef.p/src_applications_options_menu.c.o
[ 66% 2/3] Linking target main.nef
[100% 3/3] Generating pokeplatinum.us.nds with a custom command
meson test -C build
ninja: Entering directory `/home/rachel/code/git/pokeplatinum/build'
ninja: no work to do.
1/3 SBIN Checksums       OK              0.02s
2/3 Filesystem Checksums OK              0.07s
3/3 ROM Checksum         OK              0.08s

Ok:                 3
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /home/rachel/code/git/pokeplatinum/build/meson-logs/testlog.txt

Generated Constants

This realm still has opportunity for improvement (and will continue to improve as I continue to make changes to the build-scripts). For brevity, the full log is truncated.

> touch generated/abilities.txt
> make
meson configure build -Dgdb_debugging=false
ninja -C build data
ninja: Entering directory `build'
[  0% 1/2232] Generating generated/abilities.h with a custom command
[  0% 2/2232] Generating generated/abilities.py with a custom command
[  0% 3/2232] Generating res/field/encounters/encdata_ex.narc with a custom command (wrapped by meson to set env)
[  0% 4/2232] Generating res/field/encounters/pl_enc_data.narc with a custom command
[  0% 5/2232] Generating res/field/events/zone_event.narc with a custom command
[  0% 6/2232] Generating res/trainers/trdata.narc with a custom command
[  0% 7/2232] Generating res/trainers/trpoke.narc with a custom command
[  0% 8/2232] Generating res/pokemon/zukan_data_gira.narc with a custom command (wrapped by meson to set env)
[  0% 9/2232] Generating res/pokemon/zukan_data.narc with a custom command (wrapped by meson to set env)
[  0% 10/2232] Generating res/pokemon/gen_species_headers with a custom command (wrapped by meson to set env)ss
...
[ 99% 398/400] Compiling C object main.nef.p/src_unk_0200762C.c.o
[ 99% 399/400] Linking target main.nef
[100% 400/400] Generating pokeplatinum.us.nds with a custom command
meson test -C build
ninja: Entering directory `/home/rachel/code/git/pokeplatinum/build'
ninja: no work to do.
1/3 SBIN Checksums       OK              0.02s
2/3 Filesystem Checksums OK              0.07s
3/3 ROM Checksum         OK              0.09s

Ok:                 3
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /home/rachel/code/git/pokeplatinum/build/meson-logs/testlog.txt

Scripts

This was the original bogeyman, and it is finally dead. There is further improvement to be made here for battle subscripts especially, since we have duplicate ordering in generated/battle_subscripts.txt and res/battle/subscripts/sub_seq.order.

> touch res/battle/scripts/subscripts/subscript_pursuit.s
> make
meson configure build -Dgdb_debugging=false
ninja -C build data
ninja: Entering directory `build'
[ 50% 1/2] Generating 'res/battle/scripts/sub_seq.narc.p/subscript_pursuit'
[100% 2/2] Generating res/battle/scripts/sub_seq.narc with a custom command
ninja -C build pokeplatinum.us.nds
ninja: Entering directory `build'
[ 20% 1/5] Compiling C object main.nef.p/src_battle_battle_lib.c.o
[ 40% 2/5] Compiling C object main.nef.p/src_battle_battle_script.c.o
[ 60% 3/5] Compiling C object main.nef.p/src_battle_battle_controller.c.o
[ 80% 4/5] Linking target main.nef
[100% 5/5] Generating pokeplatinum.us.nds with a custom command
meson test -C build
ninja: Entering directory `/home/rachel/code/git/pokeplatinum/build'
ninja: no work to do.
1/3 SBIN Checksums       OK              0.02s
2/3 Filesystem Checksums OK              0.07s
3/3 ROM Checksum         OK              0.08s

Ok:                 3
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /home/rachel/code/git/pokeplatinum/build/meson-logs/testlog.txt

Adding a New Text Bank

I made the following change to generated/text_banks.txt:

diff --git a/generated/text_banks.txt b/generated/text_banks.txt
index 40de2b248..1a829a74a 100644
--- a/generated/text_banks.txt
+++ b/generated/text_banks.txt
@@ -722,3 +722,4 @@ TEXT_BANK_UNK_0720
 TEXT_BANK_UNK_0721
 TEXT_BANK_UNK_0722
 TEXT_BANK_UNK_0723
+TEXT_BANK_UNK_0724

Then attempted to build, which -- as expected -- triggered a regeneration of the build files and failed (because res/text/unk_0724.gmm does not exist):

> make
meson configure build -Dgdb_debugging=false
ninja -C build data
ninja: Entering directory `build'
[  0% 0/1] Regenerating build files.

...

../res/text/meson.build:29:27: ERROR: File unk_0724.gmm does not exist.

A full log can be found at /home/rachel/code/git/pokeplatinum/build/meson-logs/meson-log.txt
FAILED: build.ninja
/usr/bin/meson --internal regenerate /home/rachel/code/git/pokeplatinum .
ninja: error: rebuilding 'build.ninja': subcommand failed
make: *** [Makefile:79: data] Error 1

Renaming an Existing Text Bank

> mv res/text/acuity_cavern.gmm res/text/acuity_cavern_fake.gmm
> make
meson configure build -Dgdb_debugging=false
ninja -C build data
ninja: Entering directory `build'
ninja: error: '../res/text/acuity_cavern.gmm', needed by 'res/text/pl_msg.narc.p/acuity_cavern', missing and no known rule to make it
make: *** [Makefile:79: data] Error 1