Method output arguments return <nil> Value() for ExtensionObject arrays representing custom structure dataTypes

[hbrackel]

When calling a method whose output argument[0] is an array of custom structure dataType values, the corresponding output argument returns a slice of []*ua.ExtensionObject, where the ua.ExtensionObject.Value() == nil for each slice element. The same method called from e.g. UaExpert returns the correct decoded custom structure; that is the method returns the correct []*ua.ExtensionObject. Tested with multiple clients on multiple custom OPC UA servers. As a result, the ExtensionObject value cannot be decoded.

pseudo code:

...
oID := objectNodeId
mID := methodNodeId // method with zero input arguments; one output argument
req := &ua.CallMethodRequest{
		ObjectID:       oID,
		MethodID:       mID,
		InputArguments: []*ua.Variant{},
	}

resp, err := c.Call(ctx, req)
if err != nil {
	log.Fatal(err)
}
if got, want := resp.StatusCode, ua.StatusOK; got != want {
	log.Fatalf("got status %v want %v", got, want)
}
out := resp.OutputArguments[0].Value() 
// first output argument shall be an array of values of a custom structure dataType; 
// thus encoded as extensionObjects

extObjects, ok := out.([]*ua.ExtensionObject) // works up to here; 
// ...error handling omitted
for _, extObj range extObjects {
  value := extObj.Value(). // value is always nil??
}

[magiconair]

Just to clarify:

• The method is returning ONE value
• The value is an array of extension objects

correct?

[hbrackel]

in this concrete case, the method returns a single output argument, where the arg's variant is expected to contain an array of ExtensionObjects (encoded structured dataType values).

So: correct

[magiconair]

Thank you. @hbrackel could you have a look at the test I've added in #770?

Is this a correct description of your problem?

[hbrackel]

gopcua-issue-768.xml.zip

@magiconair the signatures of the test are correct, but the output arguments would best be encoded custom OPC UA structure dataType values, while the test uses go structure types.
For a 100% reproduction of the problem, the server would define a custom OPC UA structure dataType (none of the OPC UA predefined structures) and return an array of values of that structured OPC UA dataType.

I don't know whether the problem is specifically about receiving encoded custom structure ExtensionObjects or just "any" ExtensionObjects.

I created a tiny OPC UA nodeset.xml with a sample dataType ad a method (see attached). Maybe this helps.

Thanks for taking the time and effort!

[magiconair]

I thought that the Complex data type is a custom data type encoded as an ExtensionObject but maybe I'm mistaken. I'll have a look at the nodeset.

class Complex(uatypes.FrozenClass):
    ua_types = [
        ('i', 'Int64'),
        ('j', 'Int64'),
    ]

    def __init__(self, i=0, j=0):
        self.i = i
        self.j = j
        self._freeze = True

    def __str__(self):
        return f'Complex(i:{self.i}, j:{self.j})'

    __repr__ = __str__
...

if __name__ == "__main__":
    server = Server()
    server.set_endpoint("opc.tcp://0.0.0.0:4840/")

    ns = server.register_namespace("http://gopcua.com/")
    uatypes.register_extension_object('Complex', ua.NodeId("ComplexType", ns), Complex)
...

[magiconair]

Could you capture a tcpdump of the UAExpert method call and attach it here?

Maybe something like this:

tcpdump -n -i eth0 -s 1520 -w dump.pcap port 4840

[hbrackel]

Sorry for the confusion - the term "complex" was meant to indicate complex (non-trivial) data structures not a complex number. My bad. I'll see what I can do about the tcpdump.

[magiconair]

I understood you to mean "complex" to be a struct type. The "Complex" data type in the test is just a complex number by accident.

Your structure is also a struct. The only difference is that you use 4 instead of 2 fields and the field data types are not just numbers. But strings are length encoded byte arrays, DateTime fields are 8 byte values (int64) and a NodeID is a built-in data type. So this should work exactly like the "Complex" type in the test.

    <UADataType BrowseName="1:Issue768DataType" NodeId="ns=1;i=3003">
        <DisplayName>Issue768DataType</DisplayName>
        <References>
            <Reference ReferenceType="HasSubtype" IsForward="false">i=22</Reference>
            <Reference ReferenceType="HasEncoding">ns=1;i=5001</Reference>
            <Reference ReferenceType="HasEncoding">ns=1;i=5002</Reference>
        </References>
        <Definition Name="1:Issue768DataType">
            <Field DataType="String" Name="Name"/>
            <Field DataType="NodeId" Name="NodeId"/>
            <Field DataType="DateTime" Name="LowerBound"/>
            <Field DataType="DateTime" Name="UpperBound"/>
        </Definition>
    </UADataType>

type Issue768DataType struct {
	Name       string
	NodeId     *ua.NodeID
	LowerBound time.Time
	UpperBound time.Time
}

[hbrackel]

The other difference is that ComplexNumberType is an OPC UA defined type (namespace 0, known to gopcua, while the custom structure is not. I am not familiar enough with Python OPC UA to predict what the ua.Variant([Complex(1,2), Complex(3,4]) will be encoded to. (line 41 in your test)

[magiconair]

OK. If you could send me the tcpdump file then I can see what the protocol looks on the wire and see why the code doesn't decode it into an array of extension objects with the Issue768DataType from your nodeset.

[hbrackel]

It may take until next Tuesday, as I'll be traveling from tomorrow morning. But I will obtain the information.

[hbrackel]

@magiconair I hope the attached *.pcab is what you are looking for. Given limited admin rights on my company notebook, I created a server from the nodeset on my personal macOS machine. The recording is from a method call of the issue768 method using ProsysOPC's "Browser" client. The call succeeded and returned a 2 element array of the properly decoded Issue768DataType values.

Obviously, the securityMode is None.

I also wrote a simple gopcua client to call the method (code below). The returned output arguments correctly return an []*ua.ExtensionObject, but the Value of the ExtensionObjects is nil.

func main() {
	endpoint := "opc.tcp://localhost:4840"
	ctx := context.Background()

	c, err := opcua.NewClient(endpoint, opcua.SecurityMode(ua.MessageSecurityModeNone))
	if err != nil {
		log.Fatal(err)
	}
	if err = c.Connect(ctx); err != nil {
		log.Fatal(err)
	}
	defer c.Close(ctx)

	req := &ua.CallMethodRequest{
		ObjectID:       ua.NewNumericNodeID(0, 85),
		MethodID:       ua.NewNumericNodeID(2, 7001),
		InputArguments: []*ua.Variant{},
	}

	resp, err := c.Call(ctx, req)
	if err != nil {
		log.Fatal(err)
	}
	if got, want := resp.StatusCode, ua.StatusOK; got != want {
		log.Fatalf("got status %v want %v", got, want)
	}
	outValue, ok := resp.OutputArguments[0].Value().([]*ua.ExtensionObject)
	if !ok {
		log.Fatalf("Received unexpected type %T, want []*ua.ExtensionObject", outValue)
	}
	firstArrayElement := outValue[0]
	fmt.Printf("firstArrayElement.TypeId: %v, Value: %v\n", firstArrayElement.TypeID, firstArrayElement.Value)
	// prints 'firstArrayElement.TypeId: ns=2;i=5001, Value: <nil>'
}

dump.pcap.zip

[magiconair]

In gopcua you need to register the extension objects before you can use them. I think you are used to Java where the client SDK discovers the type from the server and dynamically creates a corresponding class. Try adding this to your code:

	type Issue768DataType struct {
		Name       string
		NodeId     *NodeID
		LowerBound time.Time
		UpperBound time.Time
	}

	RegisterExtensionObject(NewNumericNodeID(2, 5001), new(Issue768DataType))

I've written a test case and hand-decoded the byte array and this works:

func TestIssue768(t *testing.T) {
	type Issue768DataType struct {
		Name       string
		NodeId     *NodeID
		LowerBound time.Time
		UpperBound time.Time
	}

	RegisterExtensionObject(NewNumericNodeID(2, 5001), new(Issue768DataType))

	data := []byte{
		// variant

		// encoding mask: VariantArrayValues  (0x80) | TypeIDExtensionObject (0x16)
		0x96,

		// array length
		0x01, 0x00, 0x00, 0x00,

		// ExtensionObject[0]

		// TypeID
		// NodeIDTypeFourByte (0x01)
		0x01,
		// Namespace: 2
		0x02,
		// identifier: 5001
		0x89, 0x13,

		// EncodingMask: Binary (0x01)
		0x01,

		// Length: 43
		0x2b, 0x00, 0x00, 0x00,

		// Value

		// Name
		// length: 8
		0x08, 0x00, 0x00, 0x00,
		// value: Issue768
		0x49, 0x73, 0x73, 0x75, 0x65, 0x37, 0x36, 0x38,

		// NodeId
		// type: string
		0x03,
		// namespace: 123
		0x7b, 0x00,
		// lentgth: 8
		0x08, 0x00, 0x00, 0x00,
		// value: myNodeId
		0x6d, 0x79, 0x4e, 0x6f, 0x64, 0x65, 0x49, 0x64,

		// LowerBound
		0xf0, 0x49, 0x58, 0x0f, 0x1c, 0x7e, 0xdb, 0x01,

		// UpperBound
		0xf0, 0x49, 0x58, 0x0f, 0x1c, 0x7e, 0xdb, 0x01,
	}

	t.Run("codec", func(t *testing.T) {
		ts := 0x01DB7E1C0F5849F0
		now := time.Unix(0, int64((ts-116444736000000000)*100)).UTC()

		cases := []CodecTestCase{
			{
				Name: "[]*ua.ExtensionObject",
				Struct: MustVariant([]*ExtensionObject{
					NewExtensionObject(&Issue768DataType{
						"Issue768",
						NewStringNodeID(123, "myNodeId"),
						now,
						now,
					}),
				}),
				Bytes: data,
			},
		}
		RunCodecTest(t, cases)
	})

	t.Run("decode", func(t *testing.T) {
		var v Variant
		_, err := Decode(data, &v)
		require.NoError(t, err)

		val := v.Value().([]*ExtensionObject)
		require.Len(t, val, 1)
		t.Logf("val[0]: %#v", val[0].Value)
	})
}

> go test -timeout 5s -run ^TestIssue768 -v github.com/gopcua/opcua/ua
=== RUN   TestIssue768
=== RUN   TestIssue768/codec
=== RUN   TestIssue768/codec/[]*ua.ExtensionObject
=== RUN   TestIssue768/codec/[]*ua.ExtensionObject/decode
=== RUN   TestIssue768/codec/[]*ua.ExtensionObject/encode
=== RUN   TestIssue768/decode
    variant_test.go:580: val[0]: &ua.Issue768DataType{Name:"Issue768", NodeId:(*ua.NodeID)(0x14000222210), LowerBound:time.Date(2025, time.February, 13, 13, 34, 51, 919000000, time.UTC), UpperBound:time.Date(2025, time.February, 13, 13, 34, 51, 919000000, time.UTC)}
--- PASS: TestIssue768 (0.00s)
    --- PASS: TestIssue768/codec (0.00s)
        --- PASS: TestIssue768/codec/[]*ua.ExtensionObject (0.00s)
            --- PASS: TestIssue768/codec/[]*ua.ExtensionObject/decode (0.00s)
            --- PASS: TestIssue768/codec/[]*ua.ExtensionObject/encode (0.00s)
    --- PASS: TestIssue768/decode (0.00s)
PASS
ok  	github.com/gopcua/opcua/ua	0.302s

[magiconair]

Since Go is a compiled language the current implementation needs to pre-register all extensions object types at compile time. You need to use the Node ID for the binary encoding to register it (2, 5001) in your case.

So far this has worked mostly for us but I've since written some code which dynamically creates a type at runtime using reflection to support decoding. This is only useful for serializing it to JSON IMHO or some other form with reflection since you don't have access to the fields.

We use this internally to decode generic UDTs from PLCs and forward them as JSON to an API. There is a discussion thread about this. (#441)

[magiconair]

It's in the examples but maybe easy to miss

https://github.com/gopcua/opcua/blob/main/examples/udt/udt.go#L78-L97

[hbrackel]

Once again, thanks a lot for explaining this.
I am actually connecting to an aggregating server, which in turn dynamically forwards and "aggregates" underlying servers. This presents 2 challenges: UDTs can only be discovered at runtime, not compile time, and the absolute NodeIds of types are not predictable, only ExpandedNodeIds (the namespaceArray is growing as new servers are connected). My idea was to dynamically (on demand) read the DataTypeDefinition attribute of a UDT (possibly recursively) and then decode an extension object based on that knowledge. This would require that the extensionObject at least contains the UDT values as an opaque byte array, without the need to pre-register the nodes.

I would like to avoid the need to pre-register UDTs as there are literally hundreds of them. Where in the code is the ExtensionObject's payload of non-registered UDTs set to nil? If an unregistered UDTs ExtensionObject's Value() would provide the raw payload bytes, I may be able to decode "manually".

[hbrackel]

PS: How could I contribute to the gopcua project? (not on this topic, rather some other suggestions). Fork and PR?

[magiconair]

OK, that's what I thought. We compiled the structures into our mapping applications since at some point we need to understand the schema anyway and have access to the actual fields and data types.

One could argue that you can write a small app which discovers the types, generates the Go code including the registration call and stores them in a schema repo that you just import instead of dealing with map[string]any and converting to JSON only. That's not how Go usually works. Bit of a mind shift if you come from another language.

The approach is the one outlined in #691 (reply in thread). It would work well as a starting point for Go type discovery and generation as well as dynamic handling of types.

Maybe it is time to generalize this approach and add it to the library to support both use cases. You could help develop/test it if you want. Let me see if there is a good starting point.

Contributions usually start with fork and PR and over time a small number of people have taken on larger portions of the code (@kung-foo as general maintainer, @danomagnum for the server, @dwhutchison for the initial crypto code) and these people have more access to the repo, create branches without forking etc. Especially, for the server code which is fairly new and maintained by @danomagnum.

[magiconair]

Hundreds of data types don't really scare me as long as you can automate the process. It also depends on how often they change. Go code generation works well if the type definitions are fairly static but might be less good if they change frequently.

However, at some point you will have to decode this into an actual type or do you only need some fields with common names?

[magiconair]

I would like to avoid the need to pre-register UDTs as there are literally hundreds of them. Where in the code is the ExtensionObject's payload of non-registered UDTs set to nil? If an unregistered UDTs ExtensionObject's Value() would provide the raw payload bytes, I may be able to decode "manually".

There is no code that sets it to nil. It is nil because the code knows that it is an Array of Extension Objects (0x96) with length 1 and type id 2,5001 but has no corresponding type definition to decode it. So it never gets set.

It should probably return either an error or have access to the raw payload. However, if you are that far you might as well define a Go type and let the decoder do the work.

[magiconair]

As a former Java programmer with now 14 years of Go under the belt I remember that it felt really nice to go back to engineering principles and hand code certain things because it isn't really that much work. Sometimes it just seems like it.

[hbrackel]

I would like to avoid the need to pre-register UDTs as there are literally hundreds of them. Where in the code is the ExtensionObject's payload of non-registered UDTs set to nil? If an unregistered UDTs ExtensionObject's Value() would provide the raw payload bytes, I may be able to decode "manually".

There is no code that sets it to nil. It is nil because the code knows that it is an Array of Extension Objects (0x96) with length 1 and type id 2,5001 but has no corresponding type definition to decode it. So it never gets set.

It should probably return either an error or have access to the raw payload. However, if you are that far you might as well define a Go type and let the decoder do the work.

This was actually one of my questions: how can I access the raw payload of the extension object? I thought Value would be delivering it, but here is where I get nil.

I actually don't need to convert it back into the original Go structure rather only need a sort of map[key]value per dataType value instance for further e.g. JSON export processing.

Back on Tuesday - on the road now.

[hbrackel]

Hi,

after registering the DataTypeEncoding ID the method returns the proper array of UDTs.

@magiconair Thanks for your patience and support. I will close this issue then with this comment.

I would have a couple of suggestions on this topic, which I probably create separate Github issues for:
a) the registration of ExtensionObjects could support ExpandedNodeIds, as the actual namespaceIndex may only be available after connecting to the server (I am connecting to an aggregating server with a non-static namespaceArray)
b) the registration should be "per client". I am creating multiple clients connecting to multiple servers within in the same application . Each server could have a different UDT for the same DataTypeEncodingId
c) as you already suggested, if the client can't find a registered encoding, the extension object could contain the raw payload []byte data
d) if the previous would be available, one could implement dynamic decoding into a map[any]any for

I could implement the last bullet point, but would certainly need quite some guidance on the 3rd one.

Cheers, Hans-Uwe