Add an example for ingesting CloudWatch logs into OpenSearch using OpenSearch Ingestion pipeline. #1056
Conversation
…bscription filter.
Signed-off-by: Souvik Bose <[email protected]>
| for logEvent in logEvents: | ||
| request = {} | ||
| request['@id'] = logEvent['id'] | ||
| request['@timestamp'] = str(datetime.now().year) + '0' + str(datetime.now().month) + '0' + str(datetime.now().day) |
There was a problem hiding this comment.
Each log event has a timestamp property. Why not use that?
| logEvents = cwLogs['logEvents'] | ||
| for logEvent in logEvents: | ||
| request = {} | ||
| request['@id'] = logEvent['id'] |
There was a problem hiding this comment.
Why did you include @ in here? I think this may be confusing and require additional processing to remove. I think we would be fine without the @.
Maybe keep it on @timestamp if anything.
| def cw_subscription_handler(event, context): | ||
|
|
||
| """Extract the data from the event""" | ||
| data = jmespath.search("awslogs.data", event) |
There was a problem hiding this comment.
Is there any limit to the amount of data in each call? OSI only accepts sizes of 10mb or less by default.
|
|
||
| // Create a dashboard access role | ||
| const dashboardAccessRole = new Role(this, `${this.STACK_RESOURCE_NAMING_PREFIX}DashboardAccessRole`, { | ||
| assumedBy: new ServicePrincipal('ec2.amazonaws.com'), |
There was a problem hiding this comment.
Why is this role assumable by EC2? Wouldn't we have the account be the one to assume it?
| id = str(randrange(10000)) | ||
| source['id'] = id | ||
| source['timestamp'] = str(datetime.now()) | ||
| source['message'] = 'Hello world' |
There was a problem hiding this comment.
It may be nice to make this more interesting and look like some application log.
| source: | ||
| http: | ||
| path: /logs/ingest | ||
| sink: |
There was a problem hiding this comment.
I think this pipeline example would be more helpful if we ran grok. See my comment above about application logs. We could possibly have a log that you could grok here.
Fixes #1055
The purpose of this pull request is to provide an example on how OpenSearch Ingestion Pipeline could be used to ingest CloudWatch logs into Opensearch using a CloudWatch Lambda subscription filter.
This example creates the following resources:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.