feat!: configure metrics implementation as a module instead of shippi…

…ng a built-in version (#1471) Instead of a built-in metrics object, allow the user to pass the metrics collection method of their choice. Currenltly there's a `@libp2p/prometheus-metrics` implementation, there may be others in the future. Before: ```js import { createLibp2p } from 'libp2p' const node = createLibp2p({ metrics: { enabled: true } }) ``` After: ```js import { createLibp2p } from 'libp2p' import { prometheusMetrics } from '@libp2p/prometheus-metrics' const node = createLibp2p({ metrics: prometheusMetrics() }) ``` BREAKING CHANGE: the libp2p opts have changed to accept a metrics object factory function instead of a config object
libp2p · Nov 5, 2022 · 5e9dcf3 · 5e9dcf3
1 parent 074118a
commit 5e9dcf3
Show file tree

Hide file tree

Showing 21 changed files with 131 additions and 1,368 deletions.
diff --git a/doc/METRICS.md b/doc/METRICS.md
@@ -9,8 +9,6 @@ Metrics allow you to gather run time statistics on your libp2p node.
   - [Enable metrics](#enable-metrics)
   - [Stream Metrics](#stream-metrics)
   - [Component Metrics](#component-metrics)
-  - [Application metrics](#application-metrics)
-  - [Integration](#integration)
 - [Extracting metrics](#extracting-metrics)
 
 ## Overview
@@ -31,15 +29,14 @@ Although designed to primarily integrate with tools such as [Prometheus](https:/
 
 ### Enable metrics
 
-First enable metrics tracking:
+First enable metrics tracking by supplying a [Metrics](https://www.npmjs.com/package/@libp2p/interface-metrics) implementation:
 
 ```js
 import { createLibp2pNode } from 'libp2p'
+import { prometheusMetrics } from '@libp2p/prometheus-metrics'
 
 const node = await createLibp2pNode({
-  metrics: {
-    enabled: true
-  }
+  metrics: prometheusMetrics()
   //... other config
 })
 ```
@@ -87,133 +84,94 @@ class MyClass {
 }
 ```
 
-Metrics are updated by calling [`Metrics.updateComponentMetric`](https://github.com/libp2p/js-libp2p-interfaces/blob/master/packages/interface-metrics/src/index.ts#L192) and passing an object that conforms to the [`ComponentMetricsUpdate`](https://github.com/libp2p/js-libp2p-interfaces/blob/master/packages/interface-metrics/src/index.ts#L122-L152) interface:
+A tracked metric can be created by calling either `registerMetric` on the metrics object:
 
 ```ts
-metrics.updateComponentMetric({
-  system: 'libp2p',
-  component: 'connection-manager',
-  metric: 'incoming-connections',
-  value: 5
+const metric = metrics.registerMetric('my_metric', {
+  // an optional label
+  label: 'label',
+  // optional help text
+  help: 'help'
 })
-```
 
-If several metrics should be grouped together (e.g. for graphing purposes) the `value` field can be a [`ComponentMetricsGroup`](https://github.com/libp2p/js-libp2p-interfaces/blob/master/packages/interface-metrics/src/index.ts#L159):
+// set a value
+metric.update(5)
 
-```ts
-metrics.updateComponentMetric({
-  system: 'libp2p',
-  component: 'connection-manager',
-  metric: 'connections',
-  value: {
-    incoming: 5,
-    outgoing: 10
-  }
-})
+// increment by one, optionally pass a number to increment by
+metric.increment()
+
+// decrement by one, optionally pass a number to increment by
+metric.decrement()
+
+// reset to the default value
+metric.reset()
+
+// use the metric to time something
+const stopTimer = metric.timer()
+// later
+stopTimer()
 ```
 
-If the metrics are expensive to calculate, a [`CalculateComponentMetric`](https://github.com/libp2p/js-libp2p-interfaces/blob/master/packages/interface-metrics/src/index.ts#L164) function can be set as the value instead - this will need to be invoked to collect the metrics (see [Extracting metrics](#extracting-metrics) below):
+A metric that is expensive to calculate can be created by passing a `calculate` function that will only be invoked when metrics are being scraped:
 
 ```ts
-metrics.updateComponentMetric({
-  system: 'libp2p',
-  component: 'connection-manager',
-  metric: 'something-expensive',
-  value: () => {
-    // valid return types are:
-    // number
-    // Promise<number>
-    // ComponentMetricsGroup
-    // Promise<ComponentMetricsGroup>
+metrics.registerMetric('my_metric', {
+  async calculate () {
+    return 5
   }
 })
 ```
 
-### Application metrics
-
-You can of course piggy-back your own metrics on to the lib2p metrics object, just specify a different `system` as part of your `ComponentMetricsUpdate`:
+If several metrics should be grouped together (e.g. for graphing purposes) `registerMetricGroup` can be used instead:
 
 ```ts
-metrics.updateComponentMetric({
-  system: 'my-app',
-  component: 'my-component',
-  metric: 'important-metric',
-  value: 5
+const metric = metrics.registerMetricGroup('my_metric', {
+  // an optional label
+  label: 'label',
+  // optional help text
+  help: 'help'
 })
-```
 
-### Integration
+metric.update({
+  key1: 1,
+  key2: 1
+})
 
-To help with integrating with metrics gathering software, a `label` and `help` can also be added to your `ComponentMetricsUpdate`. These are expected by certain tools such as [Prometheus](https://prometheus.io/).
+// increment one or more keys in the group
+metric.increment({
+  key1: true
+})
 
-```ts
-metrics.updateComponentMetric({
-  system: 'libp2p',
-  component: 'connection-manager',
-  metric: 'incoming-connections',
-  value: 5,
-  label: 'label',
-  help: 'help'
+// increment one or more keys by passed value
+metric.increment({
+  key1: 5
 })
+
+// reset to the default value
+metric.reset()
+
+// use the metric to time something as one of the keys
+const stopTimer = metric.timer('key1')
+// later
+stopTimer()
 ```
 
 ## Extracting metrics
 
-Metrics can be extracted from the metrics object and supplied to a tracking system such as [Prometheus](https://prometheus.io/). This code is borrowed from the `js-ipfs` metrics endpoint which uses [prom-client](https://www.npmjs.com/package/prom-client) to format metrics:
+Metrics implementations will allow extracting the values for presentation in an external system. For example here is how to use the metrics implementation from `@libp2p/prometheus-metrics` to enable scraping stats to display in [Prometheus](https://prometheus.io/) or a [Graphana](https://grafana.com/) dashboard:
 
 ```ts
+import { prometheusMetrics } from '@libp2p/prometheus-metrics'
 import client from 'prom-client'
 
 const libp2p = createLibp2pNode({
-  metrics: {
-    enabled: true
-  }
+  metrics: prometheusMetrics()
   //... other config
 })
 
 // A handler invoked by express/hapi or your http framework of choice
 export default async function metricsEndpoint (req, res) {
-  const metrics = libp2p.metrics
-
-  if (metrics) {
-    // update the prometheus client with the recorded metrics
-    for (const [system, components] of metrics.getComponentMetrics().entries()) {
-      for (const [component, componentMetrics] of components.entries()) {
-        for (const [metricName, trackedMetric] of componentMetrics.entries()) {
-          // set the relevant gauges
-          const name = `${system}-${component}-${metricName}`.replace(/-/g, '_')
-          const labelName = trackedMetric.label ?? metricName.replace(/-/g, '_')
-          const help = trackedMetric.help ?? metricName.replace(/-/g, '_')
-          const gaugeOptions = { name, help }
-          const metricValue = await trackedMetric.calculate()
-
-          if (typeof metricValue !== 'number') {
-            // metric group
-            gaugeOptions.labelNames = [
-              labelName
-            ]
-          }
-
-          if (!gauges[name]) {
-            // create metric if it's not been seen before
-            gauges[name] = new client.Gauge(gaugeOptions)
-          }
-
-          if (typeof metricValue !== 'number') {
-            // metric group
-            Object.entries(metricValue).forEach(([key, value]) => {
-              gauges[name].set({ [labelName]: key }, value)
-            })
-          } else {
-            // metric value
-            gauges[name].set(metricValue)
-          }
-        }
-      }
-    }
-  }
-
-  // done updating, write the metrics into the response
+  // prom-client metrics are global so extract them from the client
   res.send(await client.register.metrics())
 }
 ```
diff --git a/doc/migrations/v0.40-v0.41.md b/doc/migrations/v0.40-v0.41.md
@@ -0,0 +1,38 @@
+# Migrating to libp2p@41 <!-- omit in toc -->
+
+A migration guide for refactoring your application code from libp2p v0.40.x to v0.41.0.
+
+## Table of Contents <!-- omit in toc -->
+
+- [Modules](#modules)
+- [Autodial](#autodial)
+
+## Metrics
+
+libp2p no longer ships a built-in metrics object, allowing the user to configure an implemnetation of their choice or not at all.
+
+Currently an [implementation](https://www.npmjs.com/package/@libp2p/prometheus-metrics) exists for [Prometheus](https://prometheus.io/)/[Graphana](https://grafana.com/), others may follow.
+
+**Before**
+
+```js
+import { createLibp2p } from 'libp2p'
+
+const node = await createLibp2p({
+  metrics: {
+    enabled: true,
+    // ... other options
+  }
+})
+```
+
+**After**
+
+```js
+import { createLibp2p } from 'libp2p'
+import { prometheusMetrics } from '@libp2p/prometheus-metrics'
+
+const node = await createLibp2p({
+  metrics: prometheusMetrics()
+})
+```
diff --git a/package.json b/package.json
@@ -106,7 +106,7 @@
     "@libp2p/interface-connection-manager": "^1.1.1",
     "@libp2p/interface-content-routing": "^1.0.2",
     "@libp2p/interface-dht": "^1.0.1",
-    "@libp2p/interface-metrics": "^3.0.0",
+    "@libp2p/interface-metrics": "^4.0.0",
     "@libp2p/interface-peer-discovery": "^1.0.1",
     "@libp2p/interface-peer-id": "^1.0.4",
     "@libp2p/interface-peer-info": "^1.0.3",
@@ -124,7 +124,7 @@
     "@libp2p/peer-id-factory": "^1.0.18",
     "@libp2p/peer-record": "^4.0.3",
     "@libp2p/peer-store": "^5.0.0",
-    "@libp2p/tracked-map": "^2.0.1",
+    "@libp2p/tracked-map": "^3.0.0",
     "@libp2p/utils": "^3.0.2",
     "@multiformats/mafmt": "^11.0.2",
     "@multiformats/multiaddr": "^11.0.0",
@@ -150,7 +150,6 @@
     "it-stream-types": "^1.0.4",
     "merge-options": "^3.0.4",
     "multiformats": "^10.0.0",
-    "mutable-proxy": "^1.0.0",
     "node-forge": "^1.3.1",
     "p-fifo": "^1.0.0",
     "p-retry": "^5.0.0",
@@ -169,20 +168,20 @@
   },
   "devDependencies": {
     "@chainsafe/libp2p-noise": "^9.0.0",
-    "@chainsafe/libp2p-yamux": "^3.0.0",
+    "@chainsafe/libp2p-yamux": "^3.0.3",
     "@libp2p/bootstrap": "^5.0.0",
     "@libp2p/daemon-client": "^3.0.5",
     "@libp2p/daemon-server": "^3.0.4",
     "@libp2p/floodsub": "^5.0.0",
     "@libp2p/interface-compliance-tests": "^3.0.2",
     "@libp2p/interface-connection-encrypter-compliance-tests": "^3.0.0",
-    "@libp2p/interface-mocks": "^7.0.1",
+    "@libp2p/interface-mocks": "^8.0.0",
     "@libp2p/interop": "^3.0.1",
-    "@libp2p/kad-dht": "^5.0.1",
+    "@libp2p/kad-dht": "^6.0.0",
     "@libp2p/mdns": "^5.0.0",
     "@libp2p/mplex": "^7.0.0",
     "@libp2p/pubsub": "^5.0.0",
-    "@libp2p/tcp": "^5.0.0",
+    "@libp2p/tcp": "^6.0.0",
     "@libp2p/topology": "^3.0.1",
     "@libp2p/webrtc-star": "^5.0.2",
     "@libp2p/websockets": "^5.0.0",

diff --git a/src/config.ts b/src/config.ts
@@ -37,17 +37,6 @@ const DefaultConfig: Partial<Libp2pInit> = {
   transportManager: {
     faultTolerance: FaultTolerance.FATAL_ALL
   },
-  metrics: {
-    enabled: false,
-    computeThrottleMaxQueueSize: 1000,
-    computeThrottleTimeout: 2000,
-    movingAverageIntervals: [
-      60 * 1000, // 1 minute
-      5 * 60 * 1000, // 5 minutes
-      15 * 60 * 1000 // 15 minutes
-    ],
-    maxOldPeersRetention: 50
-  },
   peerRouting: {
     refreshManager: {
       enabled: true,

diff --git a/src/connection-manager/dialer/index.ts b/src/connection-manager/dialer/index.ts
@@ -27,16 +27,12 @@ import { getPeer } from '../../get-peer.js'
 import sort from 'it-sort'
 import map from 'it-map'
 import type { AddressSorter, PeerStore } from '@libp2p/interface-peer-store'
-import type { ComponentMetricsTracker, Metrics } from '@libp2p/interface-metrics'
+import type { Metrics } from '@libp2p/interface-metrics'
 import type { Dialer } from '@libp2p/interface-connection-manager'
 import type { TransportManager } from '@libp2p/interface-transport'
 
 const log = logger('libp2p:dialer')
 
-const METRICS_COMPONENT = 'dialler'
-const METRICS_PENDING_DIALS = 'pending-dials'
-const METRICS_PENDING_DIAL_TARGETS = 'pending-dial-targets'
-
 export interface DialTarget {
   id: string
   addrs: Multiaddr[]
@@ -84,7 +80,6 @@ export interface DialerInit {
    * Multiaddr resolvers to use when dialing
    */
   resolvers?: Record<string, Resolver>
-  metrics?: ComponentMetricsTracker
 }
 
 export interface DefaultDialerComponents {
@@ -115,13 +110,11 @@ export class DefaultDialer implements Startable, Dialer {
     this.tokens = [...new Array(init.maxParallelDials ?? MAX_PARALLEL_DIALS)].map((_, index) => index)
     this.components = components
     this.pendingDials = trackedMap({
-      component: METRICS_COMPONENT,
-      metric: METRICS_PENDING_DIALS,
-      metrics: init.metrics
+      name: 'libp2p_dialler_pending_dials',
+      metrics: components.metrics
     })
     this.pendingDialTargets = trackedMap({
-      component: METRICS_COMPONENT,
-      metric: METRICS_PENDING_DIAL_TARGETS,
+      name: 'libp2p_dialler_pending_dial_targets',
       metrics: components.metrics
     })