Support OpenTelemetry logs in S3 source

[danhli]

Is your feature request related to a problem? Please describe.
It would be nice to provide native support for OTLP JSON logs in S3 source. OTLP JSON logs confirms to the OTLP specification and are stored as JSON files (sample file). When the original log messages are in JSON format, the OTel collector would store the log messages in escaped JSON format under logRecords/body/stringValue.

For example:
"logRecords": [
  {
    "body": {
      "stringValue": "{\"key1\":\"val1\",\"key2\":\"val2\"}"
    }
  }
]

The log format is currently supported in otel_logs_source when the OTel JSON logs are ingested through http endpoints. However, if the OTLP JSON logs are stored in files in a S3 bucket, there isn't a way to parse the escaped JSON.

Describe the solution you'd like
Add a new codec option otel_logs in the S3 source configuration. With the following pipeline configuration, the OTLP JSON logs will be ingested into OpenSearch with one document per log record and all the fields in the original log messages, e.g. key1, key2, and the attributes will be stored in separate document.

For example:
version : "2"
s3-log-pipeline:
  source:
    s3:
      acknowledgments : true
      notification_type : "sqs"
      compression : "none"
      codec:
        otel_logs:
          format: "json"
      workers : 3
      sqs:
        queue_url : ""
        maximum_messages : 10
        visibility_timeout : "60s"
        visibility_duplication_protection : true
      aws:
        region : "us-west-2"
        sts_role_arn : ""
  processor:
    - parse_json:
        source : "/body"
    - delete_entries:
        with_keys : ["body"]

Expected result

• Each log record is stored in a separate document in an OpenSearch index
• Each document contains all the attributes that are shared by all the log records, i.e. the attributes under resourceLogs/resource

For example:
{
  "_index": "my-index",
  "_id": "qyamVZIBXzpFSvHE1m14",
  "_score": 1,
  "_source": {
    "traceId": "",
    "spanId": "",
    "schemaUrl": "https://opentelemetry.io/schemas/1.6.1",
    "key1": "val1",
    "key2": "val2",
    "key3": "ke3",
    "resource.attributes.service.name": "my.service"
  }
}

[dlvenable]

We took the approach of implementing this as:

codec:
  otel_logs:
    format: json

This will allow us to also support a binary version in the future.

codec:
  otel_logs:
    format: binary