[mongodb_atlas]: Parsing errors in multiple datasets/datastreams and agent instability in Beta version

[rheppe]

Integration Name

MongoDB Atlas [mongodb_atlas]

Dataset Name

mongodb_atlas-alert,mongodb_atlas-hardware,mongodb_atlas-mongod_database,mongodb_atlas-organization,mongodb_atlas-process,mongodb_atlas-project

Integration Version

0.0.8

Agent Version

8.15.1

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Elastic Docker Image hosted on Azure Container Apps (x86_64/linux)

Software/API Version

No response

Error Message

Please see the error-log.json

Event Original

Please see the request-trace.json

What did you do?

I did some digging and isolated the error to a particular request. In the attached 'error-log.json' you can see the error being logged. The 'request-trace.json' file contains the corresponding request to the MongoDB API.

If I understand the error logs correctly, the error occurs when parsing the response from MongoDB in line 384 of the file input.go:

`"log.origin": {
"file.line": 384,
"[file.name](https://file.name/)": "cel/input.go",
"function": "[github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1](https://github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1)"
},`

When browsing Elastic GitHub I can fin the line here: https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/cel/input.go#L384:

`if err, ok := e["error"]; ok {
env.UpdateStatus(status.Degraded, fmt.Sprintf("single event error object returned by evaluation: %s", mapstr.M{"error": err}))
} else {
env.UpdateStatus(status.Degraded, "single event object returned by evaluation")
}`

I don't have any knowledge of GO, so I can't determine what exactly is going wrong. But I think your developers should be able to quickly find out what the issue is with this information.

What did you see?

I did some digging and isolated the error to a particular request. In the attached 'error-log.json' you can see the error being logged. The 'request-trace.json' file contains the corresponding request to the MongoDB API.

If I understand the error logs correctly, the error occurs when parsing the response from MongoDB in line 384 of the file input.go:

`"log.origin": {
"file.line": 384,
"[file.name](https://file.name/)": "cel/input.go",
"function": "[github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1](https://github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1)"
},`

When browsing Elastic GitHub I can fin the line here: https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/cel/input.go#L384:

`if err, ok := e["error"]; ok {
env.UpdateStatus(status.Degraded, fmt.Sprintf("single event error object returned by evaluation: %s", mapstr.M{"error": err}))
} else {
env.UpdateStatus(status.Degraded, "single event object returned by evaluation")
}`

I don't have any knowledge of GO, so I can't determine what exactly is going wrong. But I think your developers should be able to quickly find out what the issue is with this information.

What did you expect to see?

Parsed output

Anything else?

[efd6]

@rheppe The code that you are linking to is working as intended. That it is triggering the logging that you see is telling you that an error condition happened in attempts to collect the data from the API. The message in the log line would indicate what that error was. I've taken a look at the relevant code in the integration and I see that the way the collectors are written, fewer of these errors are possible than should be possible according to the input's design. This should probably be improved.

Providing the message (and only the message) of the logged errors would be helpful for the integration maintainers to be able to address the issue that you have raised here.