💡 [Feature] Log download stats from THREDDS server

[huard]

Description

It would be useful for reporting purposes to monitor data downloads from THREDDS:

• total volume of downloads (Gb/day);
• per-file download volume (Gb/day);
• per-datasets opendap streaming volumes (Gb/day);

References

This information can be parsed from NGINX logs, but those logs need to be exposed to Prometheus to be aggregated and archived within the current architecture.

Possible solutions:

• https://www.martin-helmich.de/en/blog/monitoring-nginx.html: parses logs and exposes to Prometheus
• Loki (Prometheus for logs, integrates with Graphana), seems possible to configure metrics from logs and expose them to Prometheus
• https://github.com/nginxinc/nginx-prometheus-exporter: only exposes content from the API, which is very limited
• https://github.com/google/mtail (general purpose log scraper)

Additional info

• https://www.martin-helmich.de/en/blog/monitoring-nginx.html

See also:

• Need to add some monitoring and notification to the automated deployment system and PAVICS in general #12
• 💡 [Feature] monitoring: Add a second Prometheus to scrap the first Prometheus to keep stats for long term at a lower granularity #277

Concerned Organizations

[fmigneault]

Consider downloads from WPS outputs and STAC data proxy endpoints as well for the same reasons.

[huard]

ESGF uses Beats and Logstash to collect logs and compute their stats. See https://drive.google.com/drive/folders/1LbvoYeQ_6L_bzTsO-EEhwqjIx1jZ-G1k

[fmigneault]

If the "node collector" can be located on the same instance, logstash seems like an interesting candidate. If there is no distinction between beats or logstash as "log producers", I would favor the 2nd architecture to limit the number of configurations/technologies involved.

[huard]

Parser for nginx logs and prometheus counter
https://gist.github.com/huard/25ca5be3479f72546f748da54f7097e7

[mishaschwartz]

I've created two PRs to implement log parsing in different ways. I'd like to summarize each below and briefly discuss the pros and cons of each. We should decide here which one we are interested in.

Prometheus log parser: #473

Summary: reads log files with a lightweight python library and converts log lines to metrics using python functions that create metrics using the prometheus python client

Pros:

• lightweight and easy to configure
• we write the underlying log parser code so we can update it as needed
• write custom python functions to convert lines to metrics (simple to write a basic parser and can be as complex as needed within the limits of python)
• the prometheus python client is well maintained and is an official prometheus product
• the prometheus python client is well documented and very simple so its easy to learn.
• can easily be deployed on multiple machines (supports a federated architecture)

Cons:

• we write the underlying log parser code so we need to maintain it
• slightly slower than promtail (written in python vs. go)

Promtail and Loki #474

Summary: reads log files with the promtail component and converts log lines to metrics using the metrics pipeline stage. Optionally supports shipping the parsed logs themselves to grafana (through loki) for custom log inspection.

Pros:

• slightly faster than the prometheus log parser (written in go vs. python)
• part of the grafana stack and an official grafana product
• can easily be deployed on multiple machines (supports a federated architecture)
• we don't need to maintain the underlying code

Cons:

• we can't customize the underlying code
• officially, promtail cannot be run without loki so if we just want to generate metrics and nothing else, we can do it but it's a bit of a hack.
• writing pipeline stages to extract log lines into metrics is complex and (in my opinion) very poorly documented which makes it very difficult to learn

Why not something else... logstash, beats, fluentbit ...

These could totally work as well... probably. I didn't have time to investigate them all. The main reason why I didn't choose to investigate these options is because for most of them, exporting log data to metrics required additional plugins and the complexity to set them up seemed much higher.

For our goals, I think we can achieve what we want with promtail or the prometheus log exporter. Unless there's a use-case that we can't achieve with either of those two I'm happy to look into other technologies but I'd rather stick with these two options for now.

[huard]

Thanks for the overview. I think one challenge we're having by plugging together different servers is the expertise required to configure each one. I don't think we have within our group someone fluent in Grafana for example. I'm concerned that as we add component, ẁe're going to make the problem worse.

In that sense, I'm leaning toward your first approach, which is simple and can be easily extended without delving into yet another configuration format.

[fmigneault]

I agree with @huard for the same reasons.

[mishaschwartz]

Thanks @huard and @fmigneault for your input. I'm happy with that decision as well. I'll un-draft #473 and close #474.

Once we're happy with #473 we can start adding some of the other metrics discussed here

[tlvu]

I agree with @huard for the same reasons.

Same here.