[investigation] Find a less fragile way of using otel dependencies

[tsmethurst]

Currently our otel deps seem to break intermittently when we do updates, which sort of suggests that either it's a really fragile library with a lot of moving parts, or we're not using it correctly in terms of imports and vendoring. For example, updating from v0.29.0 to v0.30.0 caused unexpected runtime breakages with the following panic:

Sep 23 21:26:37 eu1 gotosocial[345919]: panic: runtime error: invalid memory address or nil pointer dereference
Sep 23 21:26:37 eu1 gotosocial[345919]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e0d77a]
Sep 23 21:26:37 eu1 gotosocial[345919]: goroutine 1 [running]:
Sep 23 21:26:37 eu1 gotosocial[345919]: go.opentelemetry.io/otel/internal/global.(*registration).setDelegate(0xc005e88e00, {0x30bb540, 0xc00059c580})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/go.opentelemetry.io/otel/internal/global/meter.go:492 +0x19a
Sep 23 21:26:37 eu1 gotosocial[345919]: go.opentelemetry.io/otel/internal/global.(*meter).setDelegate(0xc000db9560, {0x30995e0?, 0xc00003f450?})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/go.opentelemetry.io/otel/internal/global/meter.go:138 +0x1e5
Sep 23 21:26:37 eu1 gotosocial[345919]: go.opentelemetry.io/otel/internal/global.(*meterProvider).setDelegate(0xc00090a4e0, {0x30995e0, 0xc00003f450})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/go.opentelemetry.io/otel/internal/global/meter.go:47 +0x165
Sep 23 21:26:37 eu1 gotosocial[345919]: go.opentelemetry.io/otel/internal/global.SetMeterProvider.func1()
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/go.opentelemetry.io/otel/internal/global/state.go:171 +0x2f
Sep 23 21:26:37 eu1 gotosocial[345919]: sync.(*Once).doSlow(0x1?, 0x0?)
Sep 23 21:26:37 eu1 gotosocial[345919]:         /usr/local/go/src/sync/once.go:74 +0xc2
Sep 23 21:26:37 eu1 gotosocial[345919]: sync.(*Once).Do(...)
Sep 23 21:26:37 eu1 gotosocial[345919]:         /usr/local/go/src/sync/once.go:65
Sep 23 21:26:37 eu1 gotosocial[345919]: go.opentelemetry.io/otel/internal/global.SetMeterProvider({0x30995e0, 0xc00003f450})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/go.opentelemetry.io/otel/internal/global/state.go:169 +0x1de
Sep 23 21:26:37 eu1 gotosocial[345919]: go.opentelemetry.io/otel.SetMeterProvider(...)
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/go.opentelemetry.io/otel/metric.go:41
Sep 23 21:26:37 eu1 gotosocial[345919]: github.com/superseriousbusiness/gotosocial/internal/metrics.Initialize({0x31189d0, 0xc005f4e008})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/internal/metrics/metrics.go:73 +0x465
Sep 23 21:26:37 eu1 gotosocial[345919]: github.com/superseriousbusiness/gotosocial/cmd/gotosocial/action/server.init.func1({0x30a3110, 0x610ae60})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/cmd/gotosocial/action/server/server.go:321 +0x148e
Sep 23 21:26:37 eu1 gotosocial[345919]: main.run({0x30a3110, 0x610ae60}, 0x2bf3d60)
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/cmd/gotosocial/common.go:82 +0x1ad
Sep 23 21:26:37 eu1 gotosocial[345919]: main.serverCommands.func2(0xc000af9700?, {0x2902ea3?, 0x4?, 0x2902ea7?})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/cmd/gotosocial/server.go:39 +0x2b
Sep 23 21:26:37 eu1 gotosocial[345919]: github.com/spf13/cobra.(*Command).execute(0xc000a66608, {0xc00090cba0, 0x2, 0x2})
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/github.com/spf13/cobra/command.go:985 +0xaca
Sep 23 21:26:37 eu1 gotosocial[345919]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000a66008)
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff
Sep 23 21:26:37 eu1 gotosocial[345919]: github.com/spf13/cobra.(*Command).Execute(...)
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/vendor/github.com/spf13/cobra/command.go:1041
Sep 23 21:26:37 eu1 gotosocial[345919]: main.main()
Sep 23 21:26:37 eu1 gotosocial[345919]:         /drone/src/cmd/gotosocial/main.go:74 +0x226
Sep 23 21:26:37 eu1 systemd[1]: gotosocial.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

and:

invalid observable: from different implementation

For now we can just bump back down to v0.29.0, but this isn't sustainable, so we should investigate what we're doing wrong and try to make the whole otel stack a bit more robust to updates.

[tsmethurst]

Related: #3117