-
Notifications
You must be signed in to change notification settings - Fork 52
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* feat: cleaning up logs docs * next set of logs * updated docker logs docs * heroku logs * heroku logs fix * http logs fixed * syslogs * tomcat logs * aws-lambda-nodejs * otel sdk java * otel python sdk * log file * existing collectors to signoz * aws docs * azure docs * minor fix * gcp docs * gcp docs fix * fix app logs * logs docs fix * docs fix
- Loading branch information
Showing
66 changed files
with
2,117 additions
and
1,999 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,81 +1,82 @@ | ||
| --- | ||
| date: 2024-06-06 | ||
| date: 2024-12-19 | ||
| id: ec2-logs | ||
| title: Send Application/Server logs from EC2 to SigNoz | ||
| title: Send Application/Server Logs from EC2 to SigNoz | ||
| --- | ||
|
|
||
| ## Introduction | ||
| This guide provides detailed instructions on how to send application and server logs from an EC2 instance to **SigNoz Cloud**. By integrating with SigNoz, you can efficiently collect, monitor, and analyze your logs for better insights into your applications and servers. | ||
|
|
||
| This documentation provides detailed instructions on how to send application and server logs from an EC2 instance to **SigNoz**. By integrating with SigNoz, you can | ||
| efficiently collect, monitor, and analyze your logs for better insights into your applications and servers. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - A Linux-based EC2 instance | ||
| - An active [SigNoz Cloud](http://localhost:3000/teams/) account | ||
|
|
||
| Sending your server/application logs to SigNoz Cloud broadly involves these two simple steps: | ||
| - Install OpenTelemetry Collector(OTel collector) | ||
| - Configure filelog receiver | ||
|
|
||
| <Tabs entityName="plans"> | ||
|
|
||
| ## Install OpenTelemetry Collector | ||
| <TabItem value="signoz-cloud" label="SigNoz Cloud" default> | ||
|
|
||
| The OpenTelemetry collector provides a vendor-neutral way to collect, process, and export your telemetry data such as logs, metrics, and traces. | ||
| ### Install OpenTelemetry Collector | ||
|
|
||
| You can install OpenTelemetry collector as an agent on your Virtual Machine by following this [guide](https://signoz.io/docs/tutorial/opentelemetry-binary-usage-in-virtual-machine/). | ||
| The OpenTelemetry collector provides a vendor-neutral way to collect, process, and export telemetry data such as logs, metrics, and traces. | ||
|
|
||
| ### Dummy Log File | ||
|
|
||
| ## Dummy log file | ||
|
|
||
| As an example, we can use a sample log file called `app.log` with the following dummy data: | ||
| As an example, use a sample log file called `app.log` with the following dummy data: | ||
|
|
||
| ``` | ||
| This is log line 1 | ||
| This is log line 2 | ||
| This is log line 3 | ||
| This is log line 4 | ||
| This is log line 5 | ||
| ``` | ||
|
|
||
| This file represents a log file of your application/server. | ||
|
|
||
| ## Configure filelog receiver | ||
| ### Configure Filelog Receiver | ||
|
|
||
| Receivers are used to get data into the collector. A filelog receiver collects logs from files. | ||
| Modify the `config.yaml` file that you created while installing OTel collector in the previous step to include the filelog receiver. This involves specifying the path to your `app.log` file (or your log file) and setting the `start_at` parameter. For more fields that are available for filelog receiver please check [this link](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). | ||
| Receivers are used to get data into the collector. A filelog receiver collects logs from files. Modify the `config.yaml` file that you created while installing | ||
| the OTel collector to include the filelog receiver. Specify the path to your `app.log` file (or your actual log file) and set the `start_at` parameter. | ||
|
|
||
|
|
||
| ```yaml | ||
| ```yaml:config.yaml | ||
| receivers: | ||
| ... | ||
| filelog/app: | ||
| include: [ /tmp/app.log ] #include the full path to your log file | ||
| include: [ /tmp/app.log ] # include the full path to your log file | ||
| start_at: end | ||
| ... | ||
| ``` | ||
|
|
||
| <Admonition> | ||
|
|
||
| The `start_at: end` configuration ensures that only newly added logs are transmitted. If you wish to include historical logs from the file, remember to modify `start_at` to `beginning`. | ||
|
|
||
| <Admonition> | ||
| The `start_at: end` configuration ensures that only newly added logs are transmitted. If you wish to include historical logs from the file, set `start_at` to | ||
| `beginning`. | ||
| </Admonition> | ||
|
|
||
| ## Update pipeline configuration | ||
| ### Update Pipeline Configuration | ||
|
|
||
| Receivers must be enabled via pipelines within the service section of the collector config file. In the same `config.yaml` file mentioned above, update the pipeline settings to include the new filelog receiver. This step is crucial for ensuring that the logs are correctly processed and sent to SigNoz. | ||
| Receivers must be enabled via pipelines within the `service` section of the collector config file. Update the pipeline configuration in `config.yaml`: | ||
|
|
||
| ```yaml {4} | ||
| service: | ||
| .... | ||
| logs: | ||
| receivers: [otlp, filelog/app] | ||
| processors: [batch] | ||
| exporters: [otlp] | ||
| ```yaml:config.yaml | ||
| service: | ||
| pipelines: | ||
| logs: | ||
| receivers: [otlp, filelog/app] | ||
| processors: [batch] | ||
| exporters: [otlp] | ||
| ``` | ||
|
|
||
| Now restart the OTel collector so that new changes are applied. The steps to run the OTel collector can be found [here](https://signoz.io/docs/tutorial/opentelemetry-binary-usage-in-virtual-machine/) | ||
| Restart the OpenTelemetry Collector by following the steps outlined in [this guide](https://signoz.io/docs/tutorial/opentelemetry-binary-usage-in-virtual-machine/). | ||
|
|
||
| ### Verifying the Exported Logs | ||
|
|
||
| ## Verifying the exported logs | ||
| The logs will be exported to SigNoz UI. If you add more entries to your app.log file they will also be visible in SigNoz UI. | ||
| The logs will be exported to the SigNoz Cloud UI. If you add more entries to your `app.log` file, they will also appear in the SigNoz Logs Explorer. | ||
|
|
||
| <figure data-zoomable align='center'> | ||
| <img src="/img/docs/ec2-application-server-logs.webp" alt="Logs of the dummy app.log file visible in SigNoz"/> | ||
| <figcaption><i>Dummy log file data shown in SigNoz Logs Explorer</i></figcaption> | ||
| </figure> | ||
| </figure> | ||
|
|
||
| </TabItem> | ||
|
|
||
| </Tabs> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.