Skip to content

JSON logger formatter with support for Google Cloud, DataDog and other for Elixir.

License

Notifications You must be signed in to change notification settings

qdentity/logger_json

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

LoggerJSON

Build Status Coverage Status Module Version Hex Docs Hex Download Total License

A collection of formatters and utilities for JSON-based logging for various cloud tools and platforms.

Supported formatters

Installation

Add logger_json to your list of dependencies in mix.exs:

def deps do
  [
    # ...
    {:logger_json, "~> 6.0"}
    # ...
  ]
end

and install it running mix deps.get.

Then, enable the formatter in your config.exs:

config :logger, :default_handler,
  formatter: {LoggerJSON.Formatters.Basic, []}

or during runtime (eg. in your application.ex):

:logger.update_handler_config(:default, :formatter, {Basic, []})

You might also want to format the log messages when migrations are running:

config :domain, MyApp.Repo,
  # ...
  start_apps_before_migration: [:logger_json]

Additionally, you may also be try redirecting otp reports to Logger (see "Configuration" section).

Configuration

Configuration can be set using 2nd element of the tuple of the :formatter option in Logger configuration. For example in config.exs:

config :logger, :default_handler,
  formatter: {LoggerJSON.Formatters.GoogleCloud, metadata: :all, project_id: "logger-101"}

or during runtime:

:logger.update_handler_config(:default, :formatter, {Basic, metadata: {:all_except, [:conn]}})

Docs

The docs can be found at https://hexdocs.pm/logger_json.

Examples

Basic

{
  "message": "Hello",
  "metadata": {
    "domain": ["elixir"]
  },
  "severity": "notice",
  "time": "2024-04-11T21:31:01.403Z"
}

Google Cloud Logger

Follows the Google Cloud Logger LogEntry format, for more details see special fields in structured payloads.

{
  "logging.googleapis.com/trace": "projects/my-projectid/traces/0679686673a",
  "logging.googleapis.com/spanId": "000000000000004a",
  "logging.googleapis.com/operation": {
    "pid": "#PID<0.29081.0>"
  },
  "logging.googleapis.com/sourceLocation": {
    "file": "/Users/andrew/Projects/os/logger_json/test/formatters/google_cloud_test.exs",
    "function": "Elixir.LoggerJSON.Formatters.GoogleCloudTest.test logs an LogEntry of a given level/1",
    "line": 44
  },
  "message": {
    "domain": ["elixir"],
    "message": "Hello"
  },
  "severity": "NOTICE",
  "time": "2024-04-12T15:07:55.020Z"
}

and this is how it looks in Google Cloud Logger:

{
  "insertId": "1d4hmnafsj7vy1",
  "jsonPayload": {
    "message": "Hello",
    "logging.googleapis.com/spanId": "000000000000004a",
    "domain": ["elixir"],
    "time": "2024-04-12T15:07:55.020Z"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "zone": "us-east1-d",
      "project_id": "firezone-staging",
      "instance_id": "3168853301020468373"
    }
  },
  "timestamp": "2024-04-12T15:07:55.023307594Z",
  "severity": "NOTICE",
  "logName": "projects/firezone-staging/logs/cos_containers",
  "operation": {
    "id": "F8WQ1FsdFAm5ZY0AC1PB",
    "producer": "#PID<0.29081.0>"
  },
  "trace": "projects/firezone-staging/traces/bc007e40a2e9edffa23785d8badc43b8",
  "sourceLocation": {
    "file": "lib/phoenix/logger.ex",
    "line": "231",
    "function": "Elixir.Phoenix.Logger.phoenix_endpoint_stop/4"
  },
  "receiveTimestamp": "2024-04-12T15:07:55.678986520Z"
}

Exception that can be sent to Google Cloud Error Reporter:

{
  "httpRequest": {
    "protocol": "HTTP/1.1",
    "referer": "http://www.example.com/",
    "remoteIp": "",
    "requestMethod": "PATCH",
    "requestUrl": "http://www.example.com/",
    "status": 503,
    "userAgent": "Mozilla/5.0"
  },
  "logging.googleapis.com/operation": {
    "pid": "#PID<0.250.0>"
  },
  "logging.googleapis.com/sourceLocation": {
    "file": "/Users/andrew/Projects/os/logger_json/test/formatters/google_cloud_test.exs",
    "function": "Elixir.LoggerJSON.Formatters.GoogleCloudTest.test logs exception http context/1",
    "line": 301
  },
  "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
  "context": {
    "httpRequest": {
      "protocol": "HTTP/1.1",
      "referer": "http://www.example.com/",
      "remoteIp": "",
      "requestMethod": "PATCH",
      "requestUrl": "http://www.example.com/",
      "status": 503,
      "userAgent": "Mozilla/5.0"
    },
    "reportLocation": {
      "filePath": "/Users/andrew/Projects/os/logger_json/test/formatters/google_cloud_test.exs",
      "functionName": "Elixir.LoggerJSON.Formatters.GoogleCloudTest.test logs exception http context/1",
      "lineNumber": 301
    }
  },
  "domain": ["elixir"],
  "message": "Hello",
  "serviceContext": {
    "service": "nonode@nohost"
  },
  "stack_trace": "** (EXIT from #PID<0.250.0>) :foo",
  "severity": "DEBUG",
  "time": "2024-04-11T21:34:53.503Z"
}

Datadog

Adheres to the default standard attribute list as much as possible.

{
  "domain": ["elixir"],
  "http": {
    "method": "GET",
    "referer": "http://www.example2.com/",
    "request_id": null,
    "status_code": 200,
    "url": "http://www.example.com/",
    "url_details": {
      "host": "www.example.com",
      "path": "/",
      "port": 80,
      "queryString": "",
      "scheme": "http"
    },
    "useragent": "Mozilla/5.0"
  },
  "logger": {
    "file_name": "/Users/andrew/Projects/os/logger_json/test/formatters/datadog_test.exs",
    "line": 239,
    "method_name": "Elixir.LoggerJSON.Formatters.DatadogTest.test logs http context/1",
    "thread_name": "#PID<0.225.0>"
  },
  "message": "Hello",
  "network": {
    "client": {
      "ip": "127.0.0.1"
    }
  },
  "syslog": {
    "hostname": "MacBook-Pro",
    "severity": "debug",
    "timestamp": "2024-04-11T23:10:47.967Z"
  }
}

Elastic

Follows the Elastic Common Schema (ECS) format.

{
  "@timestamp": "2024-05-21T15:17:35.374Z",
  "ecs.version": "8.11.0",
  "log.level": "info",
  "log.logger": "Elixir.LoggerJSON.Formatters.ElasticTest",
  "log.origin": {
    "file.line": 18,
    "file.name": "/app/logger_json/test/logger_json/formatters/elastic_test.exs",
    "function": "test logs message of every level/1"
  },
  "message": "Hello"
}

When an error is thrown, the message field is populated with the error message and the error. fields will be set:

Note: when throwing a custom exception type that defines the fields id and/or code, then the error.id and/or error.code fields will be set respectively.

{
  "@timestamp": "2024-05-21T15:20:11.623Z",
  "ecs.version": "8.11.0",
  "error.message": "runtime error",
  "error.stack_trace": "** (RuntimeError) runtime error\n    test/logger_json/formatters/elastic_test.exs:191: anonymous fn/0 in LoggerJSON.Formatters.ElasticTest.\"test logs exceptions\"/1\n",
  "error.type": "Elixir.RuntimeError",
  "log.level": "error",
  "message": "runtime error"
}

Any custom metadata fields will be added to the root of the message, so that your application can fill any other ECS fields that you require:

Note that this also allows you to produce messages that do not strictly adhere to the ECS specification.

// Logger.info("Hello") with Logger.metadata(:"device.model.name": "My Awesome Device")
// or Logger.info("Hello", "device.model.name": "My Awesome Device")
{
  "@timestamp": "2024-05-21T15:17:35.374Z",
  "ecs.version": "8.11.0",
  "log.level": "info",
  "log.logger": "Elixir.LoggerJSON.Formatters.ElasticTest",
  "log.origin": {
    "file.line": 18,
    "file.name": "/app/logger_json/test/logger_json/formatters/elastic_test.exs",
    "function": "test logs message of every level/1"
  },
  "message": "Hello",
  "device.model.name": "My Awesome Device"
}

Copyright and License

Copyright (c) 2016 Andrew Dryga

Released under the MIT License, which can be found in LICENSE.md.

About

JSON logger formatter with support for Google Cloud, DataDog and other for Elixir.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Elixir 100.0%