Serialization - 1000 cuts

[jlpedrosa]

Hi @vaerh

I got myself into another pitfal because how we serialize the objects and the default values. In this case, the defaults was overriding a behavior when some field is unset. (BGP timers).

In other cases we found defaults that change over time. We tried the "AlwaysPresentUserNotProvided", but that also comes with it's own problems.

I think we need a mechanism to have more fine grained control how the fields are serialised when the user provides it, or how to behave when the router provided it and the user didn't At the same time,terraform schema has it's own limitations.

I think there are multiple ways to tackle this.
Struct holding schema and serialization options per field, ie:

type Field struct {
  *schema.Schema
  *routeros.Serialization 
}

Or having object types per resource with tags, ie:

type InterfaceBridgeXX struct {
   MacAddress string `routeros:"ommitempty, ...."`
}

Any option in this sense would be a really big swing and a big task (that could be done progressively). Are you open to any of this ideas? (or any other way to take on the problem)?

[vaerh]

Hi @jlpedrosa

The current serealization is flawed and that's a fact. I don't mind trying to improve it. But at the same time I would like to keep it simple and readable.
And I don't want to go in the direction of plugin framework now.
Let's take a specific resource and work with it.
The only thing is that I'm away from my computer for the rest of the week.

[jlpedrosa]

@vaerh feel free to add things (edit comment) or add in a response and I'll join them here.|

Requirements:

• needs to support ALL data types, bool, int, string, list, object.
• should not generate drift on re-aplying the same hcl.

Scenarios:

• User has not defined a field in hcl, the router sets a value and it's read back.
• User has not defined a field in hcl, but it's mandatory and the default must be sent.

[vaerh]

I'll just give my thoughts on the problems I'm having with the provider and router interaction:
The biggest technical problem that prevents us from migrating to a fresh framework is the inconsistency of values between the user declarative configuration and the resource state that MT returns. This is unacceptable in the new version: as much as there is in the script, so much should be in the state.

Another problem is the overcomplication of creating uniform resources, but there is granular control over the data. The funny thing about this is that this is the approach under which the original gfenn-newbury and terraform-provider-mikrotik was written. One resource - one static data exchange structure.

It seems to me that MT sometimes resembles a large coral reef with long-term layers from which it is impossible to escape.
If I understood the REST API appliance correctly, then it's just a serializer, above the CLI interpreter. This is associated with problems with the lack of data types in JSON, when at first glance a numeric field may unexpectedly return "none".

I don't understand what to do if the time is returned to the user in a format that does not match the fields in WinBox.
It is unclear to me how to react to values that are not in hcl, but they have changed over time and there is a difference between the state file and the MT returned values.
And such problems appear with enviable regularity. How many gray hairs did the CAPsMAN configuration cost me? In the MT world, the resource configuration can return both the xxx attribute and the xxx.yyy attribute at the same time. But in the TF paradigm, this is impossible!
I do not know how to describe it beautifully in a schema without using constant hacks.

But maybe you have some fresh ideas? Or we can solve problems as they come in ))))