Machine readable format

[burner1024]

I'm thinking that it could be beneficial to have IESDP data in machine readable format. Currently there are multiple tools all parsing and reading IE files on their own: DLTCEP, NearInfinity, iesh, probably others. Inevitably, differencies and inconsistencies arise. But if file formats were described in a structured way, we could have a single, truly definitive source of information, and pull updates from it (semi)automatically. (Which is my ulterior motive for BGforge MLS.)

As an example, take a look at sfall docs: functions listed in a yaml file, with some python scripting it converted to markdown, and markdown is published with jekyll, resulting in a nice site.

If you look at opcode description, it's already basically yaml, with html additions. Could probably converted into true yaml semi-automatically.

To be clear, this is not about IESDP looks, just internal data representation: binary file formats, script actions/triggers, effects.

Tagging @Argent77, @4Luke4, @AvengerTeamBG, @FredrikLindgren, @ALIENQuake.

[lynxlynxlynx]

Something like this might have been useful 15 years ago, but now that almost everything is already written, I don't see anyone gaining anything by rewriting all the parsers, writing generators and rewriting the way the data is stored.

On the IESDP side of things the last part may still happen though, since it could then convey the data better (eg. #6 #19 #26).

[burner1024]

almost everything is already written

MLS isn't, hence this issue.
Obviously, I can't speak for devs of other tools, but MLS would benefit.

@lynxlynxlynx let me put it like this: if a such pull request was coming your way, would you be opposed to it?

[lynxlynxlynx]

Depends on what everything you'd stick in it. Eg. if you want to make the file formats machine readable, that's great and wouldn't need to affect deployment, since jekyll has the needed support: https://jekyllrb.com/docs/datafiles/

[burner1024]

Do you want to define a format for, say, opcodes? Or I should do that myself?

[lynxlynxlynx]

I don't know what you mean, since the opcodes are already easily machine readable. Everything is parametrized and the description is plain html most of the time (a few liquid tags here and there).

[burner1024]

Well, html with liquid parts is not exactly machine readable.
Still being parametrized is exactly why it's easier to start with them.

To make an example, maybe something like this
_data/opcodes.yml

- n: 0
  name: AC vs. Damage Type Modifier
  type: stat
  param1: AC Modifier
  param2: Type
  bg1: 1
  bg2: 1
  bgee: 1
  iwd1: 1
  iwd2: 0
  pst: 1
  doc: |
    Applies the modifier value specified by the 'AC Modifier' field to the category specified
    by the 'Type' field.

    Known values for 'Type' are:
      - 0   All
      - 1   Crushing
      - 2   Missile
      - 4   Piercing
      - 8   Slashing
      - 16  Base AC setting (sets the targets AC to the value specified by the 'AC Modifier' field.

    If the targets AC is already 'AC Modifier' or below, this effect will do nothing).

    Each modifier type to AC from this opcode is capped to the range [-20, 20]. Each AC type total is capped to the range [-32768,32767].

  notes:
    - |
      IWD1 and PST use a slightly different version. The "Base AC" sets to **field - 1** instead.
    
      IWD2 uses different parameters altogether.

[ALIENQuake]

The "Known values" can also be one of yaml key with values, right?

[lynxlynxlynx]

The notes are sometimes interspersed in the description, not always at the end. And there can be several of different severities, so the proposed format is not good enough. I also see no way to avoid needing to clean up the descriptions on the user side of things, eg. to get rid of broken links.

And I'm definitely opposed to cramming them all into one file. That's a simple step users can do if they need it.

[burner1024]

The "Known values" can also be one of yaml key with values, right?

Right, but I'm not sure if will be possible to parametrize them and keep current html layout unchanged. And I don't know how I'd use this data yet. If anyone needs it, they are welcome to chime in.

The notes are sometimes interspersed in the description, not always at the end. And there can be several of different severities, so the proposed format is not good enough. I also see no way to avoid needing to clean up the descriptions on the user side of things, eg. to get rid of broken links.

Severity is easy to deal with, adding a separate stanza (warning, important) should be enough.
Being interspersed is harder. Not sure if there's a way to parametrize this properly. I'll search around, if not, I guess keeping the doc monolithical will have to do.

Could you clarify the bit about broken links? Which ones do you mean, why are they broken?

And I'm definitely opposed to cramming them all into one file. That's a simple step users can do if they need it.

Jekyll can read data from dirs, something like _data/opcodes/0.yml. Though it does start to sound like simply moving/renaming opcodes dir, but I'm trying to work out something that'll work for actions and stuff with minimal changes later.

[lynxlynxlynx]

Links: opcodes link to each other via anchors, so you'd end up with broken links unless you replicated the naming and therefore layout (all in one file).

Actions and triggers won't be much better; except for less data, the same problems apply.

[burner1024]

Depending on how this works out, links might be kept and lead back to IESDP:

Maybe it's better to start with something else than opcodes, indeed. Here, I took a shot at actions. Sample data is stored in the data files.
I wanted to avoid duplicating info (action numbers in both filenames and files themselves, etc), but liquid is not flexible enough, so the overall system is almost the same as opcodes, just doesn't require setting game to 0 to filter it out (can be just skipped).
You can launch it locally, check out BG1 and IWD2 action pages.

[lynxlynxlynx]

Looks good. From what I can tell, action descriptions only use colors (can be replaced) and links (markdownify takes care of that?) on top of what's shown above.

[burner1024]

Loaded BG2 actions.
There's minor difference in styling, mostly because markdownify wraps everything into paragraphs. I made adjustments, but didn't go out of my way to make it exactly the same before getting feedback.
(One thing I did fix intentionally is ugly codeblocks)

If all's good, next step would be to load actions from other files, run comm/md5 to find and delete the identical ones, then some diffs to find and combine those that only differ in wording, then move all the rest as variants.

[lynxlynxlynx]

Sounds good, but let's resolve #85 first, so it's clearer how some of the more interesting actions turn up (since you skipped a few).

[burner1024]

I didn't skip anything on purpose, could you point out some?

[lynxlynxlynx]

Most for #85, but also 349 that doesn't have any colouring. Do something like find | wc -l in the dir and you'll see it doesn't match the max action id + 1.

[burner1024]

These were missed due to a mistake in the script, now included. Count won't match anyway, since there are gaps (NID*).
349 is a special one, it has some formatting messed up, but I think it can be corrected latter, after adding variants, along with other manual cleanups here and there.
The table in 349 is just styled as code. I think it's not worth to add a separate kludge just for it, considering that it looks fine, but if you think it's important to have the same background, I guess it can be done.
Pushed updated data.

[lynxlynxlynx]

NIDSpecial1 and co are not a gap, just useless to the modder.

[burner1024]

Well, they are counted as one "action" as far as Jekyll is concerned. Otherwise, they'd produce a full list of "not working" actions, so I thought to keep them combined.
If everything's looks so far, please let me know, I'll proceed with variants.

[lynxlynxlynx]

It doesn't matter for the output, but for people like you that want it as data, it makes no sense to jumble them together.

[burner1024]

I'm not sure what use they are, only completeness. Certainly no point in adding them to completion, everything not working will be skipped.
So do you want to separate them?

[lynxlynxlynx]

Yes, just for consistency.

[burner1024]

All right. I will do that at manual stage. Anything else?

[lynxlynxlynx]

Nothing comes to mind, except that let's do plain bg2 first and iron any problems out before continuing. Also, another PR is open that touches the ee action list and it would just cause conflicts if it wasn't merged first.

[burner1024]

You mean merge upstream? If just BG2 has manual updates applied and merged, that'll make it a little harder to search for differences later. Also, there won't be links to variants, since that data doesn't exist yet.

One more thing I'd like to point out, currently action aliases are added to the same file, how's that?

[lynxlynxlynx]

Ok, then wait a bit, since @4Luke4 is almost done.

Aliases I don't like that way, since I think eg. the RES variants are present in some games without the default version, so it would complicate the layout logic to iterate properly. Just keep it KISS and create a separate file.

[burner1024]

I would like to avoid duplicating descriptions in data and displaying duplicates too, but not sure how to do that while allowing for variants. I'll think about that meanwhile.

Edit: ah, current version doesn't have variant links too, so my second point about merging is moot.

[lynxlynxlynx]

Ideally the descriptions wouldn't be duplicates anyway, since they should explain the parameters used.

[burner1024]

There's many Dialogue/Dialog synonyms, though.

[lynxlynxlynx]

Is it rendered somewhere?

[burner1024]

No, but I can set it up, will share a link.

[burner1024]

here

[lynxlynxlynx]

looks fine 👍

[burner1024]

Added itm v1, eff v1 and v2. Feature blocks seem to be identical except some spelling, so I just symilinked itm to spl one.
I think this is good enough for a start.

Next I plan to try an import, see if any issues come up, maybe some adjustments will be needed. Then send a pull, after it's merged I'll be able to make a real import and publish a release. And then make an announcement on the forum, see if anyone else wants to jump in.

Much of the text is still duplicated between formats (target type, timing, resistance, etc) and probably could be externalized into includes too, but that would make importing somewhat harder. If you ever get to that, please notify.

[lynxlynxlynx]

itmv1 lost the "Melee animation" link. Feature block looks worse due to removal of spacing around =. And timing mode info is better in itm (and target type). It might be better to link it the other way around, but it's now simpler just to add the missing info.

eff look fine.

[burner1024]

Addressed all, also applied the same style to Spell type.

[lynxlynxlynx]

Target type and timing mode still lost some info.

[burner1024]

Ah, sorry, missed that.
But it looks like like target type 7 clause in ITM doesn't apply to SPL? Then they have to be kept separate. (Or change clause text to something like "In ITM: ranged ability type only, otherwise no target").
As for timing mode, why it isn't in sync? Is it just that spl page wasn't updated when itm was, or it actually works differently?

[lynxlynxlynx]

type: sure, be conservative
timing: they do behave the same.

[burner1024]

Cool, added the missing lines.

[burner1024]

Looks good:

If there isn't anything else, I can send a PR.

[lynxlynxlynx]

Nice, feel free to go ahead. :)

[burner1024]

Simple script to convert offsets table into yaml: offsets_to_yaml.py.zip

File naming example: _data/file_formats/itm_v1/extended_header.yml (filename is singular).

Format reference:

- desc: |             # required - markdown
    Attack type
    - 0 = None
    - 1 = Melee
  type: char         # required.
  length: 1          # optional, if not specified, size inferred from type. Known types: char, byte, word, dword, resref, strref
  offset: 0x1        # optional, if specified, current offset is checked against this value, if not equal, an error is raised
  mult: 3            # optional, allows to do stuff like "2*3 (word)"
  unknown: 1         # optional, applies "unknown" style span
  unused: 1          # optional, appends " (unused)" to description and applies "unknown" style span

[lynxlynxlynx]

While researching for the recent opcode regression, I came across a new liquid link tag. Unfortunately it doesn't look like it supports creating relative paths:
https://jekyllrb.com/docs/liquid/tags/#links

[burner1024]

Something I came by recently: https://kaitai.io/.
A similar yaml format declaration, which then compiles into bindings for C++, Python, Java, etc. Could be actually used to unify databases of formats in GemRB, iesh, NearInfinity, etc.
Just food for thought, not suggesting anything yet.

[lynxlynxlynx]

Looks familiar, but I don't think it's feasible. There are so many exceptions, dependencies between fields and special treatment needed besides the fact that we'd need per-game-version copies of many of the formats.

[burner1024]

Is there any system to parameter naming in actions and triggers?
It seems to be modeled after C syntax? I guess "*" should mean a pointer. But why is appended to every parameter then? Like HP(O:Object*,I:Hit Points*).

There are also multiple discrepancies like

PartyHasItem(S:Item*)
# although
HasItem(S:ResRef*,O:Object*)

where the same resref paramerters are referred to differently.

And

ForceSpellRES(S:RES*,O:Target*)

where it's under another alias.

The result is, the doc seems like a C-engine internal reference. Yes, resref is actually a string, and you can compile a baf with pretty much any string. But from scripter's point of view, resref is a distinct type, and the expected API would look more like this

PartyHasItem(ResRef:Item)
HasItem(ResRef:Item, O:Object)
ForceSpellRES(ResRef:Spell, O:Target)

Does that make sense? Would it be prudent to try and make it more consistent, as well as more scripter/modder-oriented?

[Argent77]

The parameter prefix (I, O, P, S) seems to be hardcoded and determines the parameter type. The actual parameter name is mostly* arbitrarily chosen. I'm not so sure about the trailing asterisk, except that it separates the parameter name from an optional IDS file reference. Some definitions in TRIGGER.IDS don't even use the asterisk at all (e.g. XP parameter in the 0x40C5 XPLT(O:Object*,I:XP) definition) - although that could have just been an oversight.

The actual number (and order) of parameters is also relevant, as all triggers and actions must conform to the BCS format specification.

*) From what I can gather from these notes the engine hardcoded at least some parameter names (e.g. name* and area* for global-related actions and triggers).

[lynxlynxlynx]

I don't think it makes sense to try to clean this up: since all the other tools use the provided strings, it could appear that the updated signatures are for distinct things instead.

[burner1024]

all the other tools use the provided strings

Sorry, use how? Any examples?

[lynxlynxlynx]

I doubt anyone parses the names, I meant display (e.g. tooltip in NI). But like @Argent77 already said, the prefix part is crucial for the engines to interpret them correctly anyway, so at best the names could be changed.

[burner1024]

I don't think there are tooltips in NI? Do you mean warnings? And the reason NI is able to show them is because it's actually correctly distinguishes strings from resrefs, and further segregates different resources types.
But I guess I can make it work in a fork.

[lynxlynxlynx]

When you hover over an action, trigger or object, it shows the signature in a tooltip. It has nothing to do with types.

[burner1024]

It doesn't for me. Is there a setting or something?

[lynxlynxlynx]

Maybe, do you have syntax highlighting on? Or maybe your version is too old. But it is true it doesn't work for objects.

[burner1024]

NVM, I can see it now. First appearance has some delay, following ones show up instantly.