Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core/services/streams: create new Vars to avoid race #15997

Merged
merged 1 commit into from
Jan 20, 2025
Merged

core/services/streams: create new Vars to avoid race #15997

merged 1 commit into from
Jan 20, 2025

Conversation

jmank88
Copy link
Contributor

@jmank88 jmank88 commented Jan 20, 2025 •
edited
Loading

https://github.com/smartcontractkit/chainlink/actions/runs/12861990746/job/35856232378

data race 
==================
WARNING: DATA RACE
Write at 0x00c010670330 by goroutine 129593:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.23.4/x64/src/runtime/map_faststr.go:223 +0x0
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.Vars.Set()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/variables.go:80 +0xa4
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.(*scheduler).Run()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/scheduler.go:[20](https://github.com/smartcontractkit/chainlink/actions/runs/12861990746/job/35856232378#step:17:21)1 +0x64e
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.(*runner).run.gowrap1()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/runner.go:385 +0x33

Previous read at 0x00c010670330 by goroutine 129613:
  runtime.mapiterinit()
      /opt/hostedtoolcache/go/1.23.4/x64/src/runtime/map.go:877 +0x0
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.Vars.Copy()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/variables.go:90 +0x72e
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.newScheduler()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/scheduler.go:103 +0x6b3
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.(*runner).run()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/runner.go:384 +0x6a4
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.(*runner).ExecuteRun()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/runner.go:316 +0x17c
  github.com/smartcontractkit/chainlink/v2/core/services/streams.(*multiStreamPipeline).executeRun()
      /home/runner/work/chainlink/chainlink/core/services/streams/pipeline.go:125 +0x12c
  github.com/smartcontractkit/chainlink/v2/core/services/streams.(*multiStreamPipeline).Run()
      /home/runner/work/chainlink/chainlink/core/services/streams/pipeline.go:106 +0x53
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*observationContext).run()
      /home/runner/work/chainlink/chainlink/core/services/llo/observation_context.go:186 +0x48c
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*observationContext).Observe()
      /home/runner/work/chainlink/chainlink/core/services/llo/observation_context.go:53 +0x7c
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*dataSource).Observe.func2()
      /home/runner/work/chainlink/chainlink/core/services/llo/data_source.go:118 +0x15c
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*dataSource).Observe.gowrap1()
      /home/runner/work/chainlink/chainlink/core/services/llo/data_source.go:138 +0x3e

Goroutine 129593 (running) created at:
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.(*runner).run()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/runner.go:385 +0x72a
  github.com/smartcontractkit/chainlink/v2/core/services/pipeline.(*runner).ExecuteRun()
      /home/runner/work/chainlink/chainlink/core/services/pipeline/runner.go:316 +0x17c
  github.com/smartcontractkit/chainlink/v2/core/services/streams.(*multiStreamPipeline).executeRun()
      /home/runner/work/chainlink/chainlink/core/services/streams/pipeline.go:125 +0x12c
  github.com/smartcontractkit/chainlink/v2/core/services/streams.(*multiStreamPipeline).Run()
      /home/runner/work/chainlink/chainlink/core/services/streams/pipeline.go:106 +0x53
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*observationContext).run()
      /home/runner/work/chainlink/chainlink/core/services/llo/observation_context.go:186 +0x48c
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*observationContext).Observe()
      /home/runner/work/chainlink/chainlink/core/services/llo/observation_context.go:53 +0x7c
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*dataSource).Observe.func2()
      /home/runner/work/chainlink/chainlink/core/services/llo/data_source.go:118 +0x15c
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*dataSource).Observe.gowrap1()
      /home/runner/work/chainlink/chainlink/core/services/llo/data_source.go:138 +0x3e

Goroutine 129613 (running) created at:
  github.com/smartcontractkit/chainlink/v2/core/services/llo.(*dataSource).Observe()
      /home/runner/work/chainlink/chainlink/core/services/llo/data_source.go:116 +0x62c
  github.com/smartcontractkit/chainlink-data-streams/llo.(*Plugin).observation()
      /home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected]/llo/plugin_observation.go:150 +0x2243
  github.com/smartcontractkit/chainlink-data-streams/llo.(*Plugin).Observation()
      /home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected]/llo/plugin.go:306 +0x111
  github.com/smartcontractkit/chainlink/v2/core/services/ocr3/promwrapper.(*reportingPlugin[go.shape.struct { LifeCycleStage github.com/smartcontractkit/chainlink-common/pkg/types/llo.LifeCycleStage; ReportFormat github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportFormat }]).Observation.func1()
      /home/runner/work/chainlink/chainlink/core/services/ocr3/promwrapper/plugin.go:58 +0x11d
  github.com/smartcontractkit/chainlink/v2/core/services/ocr3/promwrapper.withObservedExecution[go.shape.struct { LifeCycleStage github.com/smartcontractkit/chainlink-common/pkg/types/llo.LifeCycleStage; ReportFormat github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportFormat },go.shape.[]uint8]()
      /home/runner/work/chainlink/chainlink/core/services/ocr3/promwrapper/plugin.go:143 +0x72
  github.com/smartcontractkit/chainlink/v2/core/services/ocr3/promwrapper.(*reportingPlugin[go.shape.struct { LifeCycleStage github.com/smartcontractkit/chainlink-common/pkg/types/llo.LifeCycleStage; ReportFormat github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportFormat }]).Observation()
      /home/runner/work/chainlink/chainlink/core/services/ocr3/promwrapper/plugin.go:57 +0x1a4
  github.com/smartcontractkit/chainlink/v2/core/services/ocr3/promwrapper.(*reportingPlugin[github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportInfo]).Observation()
      /home/runner/work/chainlink/chainlink/core/services/ocr3/promwrapper/plugin.go:56 +0xe4
  github.com/smartcontractkit/libocr/offchainreporting2plus/internal/shim.LimitCheckOCR3ReportingPlugin[go.shape.struct { LifeCycleStage github.com/smartcontractkit/chainlink-common/pkg/types/llo.LifeCycleStage; ReportFormat github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportFormat }].Observation()
      /home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected][21](https://github.com/smartcontractkit/chainlink/actions/runs/12861990746/job/35856232378#step:17:22)5956-e5b78d8e3919/offchainreporting2plus/internal/shim/ocr3_reporting_plugin.go:39 +0x10d
  github.com/smartcontractkit/libocr/offchainreporting2plus/internal/shim.(*LimitCheckOCR3ReportingPlugin[github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportInfo]).Observation()
      <autogenerated>:1 +0x21b
  github.com/smartcontractkit/libocr/offchainreporting2plus/internal/ocr3/protocol.(*outcomeGenerationState[go.shape.struct { LifeCycleStage github.com/smartcontractkit/chainlink-common/pkg/types/llo.LifeCycleStage; ReportFormat github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportFormat }]).backgroundObservation.func1()
      /home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected][22](https://github.com/smartcontractkit/chainlink/actions/runs/12861990746/job/35856232378#step:17:23)3215956-e5b78d8e3919/offchainreporting2plus/internal/ocr3/protocol/outcome_generation_follower.go:241 +0x158
  github.com/smartcontractkit/libocr/offchainreporting2plus/internal/ocr3/protocol.callPluginFromOutcomeGenerationBackground[go.shape.[]uint8,go.shape.struct { LifeCycleStage github.com/smartcontractkit/chainlink-common/pkg/types/llo.LifeCycleStage; ReportFormat github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportFormat }].func1()
      /home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected][23](https://github.com/smartcontractkit/chainlink/actions/runs/12861990746/job/35856232378#step:17:24)215956-e5b78d8e3919/offchainreporting2plus/internal/ocr3/protocol/outcome_generation.go:391 +0xb5
  github.com/smartcontractkit/libocr/offchainreporting2plus/internal/ocr3/protocol.callPluginFromBackground[go.shape.[]uint8]()
      /home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected][24](https://github.com/smartcontractkit/chainlink/actions/runs/12861990746/job/35856232378#step:17:25)1223215956-e5b78d8e3919/offchainreporting2plus/internal/ocr3/protocol/common.go:72 +0x227
  github.com/smartcontractkit/libocr/offchainreporting2plus/internal/ocr3/protocol.callPluginFromOutcomeGenerationBackground[go.shape.[]uint8,go.shape.struct { LifeCycleStage github.com/smartcontractkit/chainlink-common/pkg/types/llo.LifeCycleStage; ReportFormat github.com/smartcontractkit/chainlink-common/pkg/types/llo.ReportFormat }]()
      /home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected]/offchainreporting2plus/internal/ocr3/protocol/outcome_generation.go:381 +0x[29](https://github.com/smartcontractkit/chainlink/actions/runs/12861990746/job/35856232378#step:17:30)7

@jmank88 jmank88 requested a review from samsondav January 20, 2025 19:11
@jmank88 jmank88 marked this pull request as ready for review January 20, 2025 19:13
@jmank88 jmank88 requested review from a team as code owners January 20, 2025 19:13
@jmank88 jmank88 requested a review from agparadiso January 20, 2025 19:13
@github-actions GitHub Actions
Copy link
Contributor

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

🎖️ No JIRA issue number found in: PR title, commit message, or branch name. Please include the issue ID in one of these.

@jmank88
Copy link
Contributor Author

jmank88 commented Jan 20, 2025

@samsondav Does this also suggest that the observationContext allowed to executions to occur concurrently? It looks synced to me, so maybe there were multiple instances?

@cl-sonarqube-production
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@jmank88 jmank88 enabled auto-merge January 20, 2025 22:29
@jmank88 jmank88 added this pull request to the merge queue Jan 20, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 20, 2025
@jmank88 jmank88 added this pull request to the merge queue Jan 20, 2025
Merged via the queue into develop with commit 8bbb77f Jan 20, 2025
169 of 171 checks passed
@jmank88 jmank88 deleted the streams-pipeline-race branch January 20, 2025 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrickhuie19 patrickhuie19 patrickhuie19 approved these changes

@samsondav samsondav samsondav approved these changes

@agparadiso agparadiso Awaiting requested review from agparadiso agparadiso is a code owner automatically assigned from smartcontractkit/foundations

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jmank88 @samsondav @patrickhuie19