Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Triple failure rate 15%: AssertionFailed("confirmation from Participant(x) did not match expectation") #923

Open
Tracked by #926
ppca opened this issue Nov 6, 2024 · 1 comment
Labels
Emerging Tech Emerging Tech flying formation at Pagoda Near BOS NEAR BOS team at Pagoda

Comments

@ppca
Copy link
Contributor

ppca commented Nov 6, 2024

Most of faled to generate some triples in the logs, the reason is not triple protocol time out, but
[AssertionFailed("confirmation from Participant(x) did not match expectation")]

Take dev 6 node as an example. From 8.31 am to 9.31 am Nov 6 PST time(gcp log link), there are 156 times of faled to generate some triples, but only 13 among those are timeouts. So (156-13)/156 = 91.7% of all failed triple generators failed because of AssertionFailed("confirmation from Participant(x) did not match expectation").

We need to figure out why this error would happen.

@ppca ppca added Near BOS NEAR BOS team at Pagoda Emerging Tech Emerging Tech flying formation at Pagoda labels Nov 6, 2024
@ppca ppca changed the title Triple failure rate 15%: AssertionFailed("confirmation from Participant(x) did not match expectation") Triple failure rate 15%: AssertionFailed("confirmation from Participant(x) did not match expectation") Nov 6, 2024
@volovyks
Copy link
Collaborator

If you compare the occurrence of this log in GCP with data from Grafana, you will see, that it mostly acores when we are actively generating triples. The issue may be on the messaging layer, or some other place affected by high load.
Image
Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Emerging Tech Emerging Tech flying formation at Pagoda Near BOS NEAR BOS team at Pagoda
Projects
Status: Backlog
Development

No branches or pull requests

2 participants