Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify the logic for the acceptance tests #1076

Open
isabelle-dr opened this issue Dec 10, 2021 · 2 comments
Open

Modify the logic for the acceptance tests #1076

isabelle-dr opened this issue Dec 10, 2021 · 2 comments
Labels
enhancement New feature request or improvement on an existing feature status: Needs discussion We need a discussion on requirements before calling this issue ready tests Anything related to our tests.

Comments

@isabelle-dr
Copy link
Contributor

isabelle-dr commented Dec 10, 2021

As discussed in this comment:

Modify the logic of the acceptance tests so a PR is flagged if there is at least one additional error in one dataset from the Mobility Database (currently it's flagged if there is at least one additional error in at least 1% of datasets).

@isabelle-dr isabelle-dr added the enhancement New feature request or improvement on an existing feature label Dec 10, 2021
@barbeau
Copy link
Member

barbeau commented Dec 10, 2021

Modify the logic of the acceptance tests so a PR is flagged if there is at least one additional error in one dataset from the Mobility Database

Is this one additional error type, or one additional occurrence of an error type (or both)?

If you're treating this more like a unit test and expecting the same output from both validators, then IMHO count occurrences of errors is important. In other words, you'd want to know if a code change made the output jump from 1 occurrence of block_trips_with_overlapping_stop_times to 1000 occurrences.

@isabelle-dr
Copy link
Contributor Author

👋 Hello,

Is this one additional error type, or one additional occurrence of an error type (or both)?

We are thinking one additional occurrence of an error type. Thanks for clarifying this!

@isabelle-dr isabelle-dr added Enhancement - Performance and removed enhancement New feature request or improvement on an existing feature labels Jan 7, 2022
@isabelle-dr isabelle-dr modified the milestones: Future work, v3.2.0 Jan 7, 2022
@isabelle-dr isabelle-dr added enhancement New feature request or improvement on an existing feature and removed Enhancement - Performance labels Jan 9, 2022
@isabelle-dr isabelle-dr modified the milestones: v3.2.0 - improve acceptance tests, v3.3.0 - improvements for the acceptance test Jan 9, 2022
@isabelle-dr isabelle-dr added tests Anything related to our tests. status: Ready An issue that is ready to be worked on. labels Aug 10, 2022
@isabelle-dr isabelle-dr added status: Needs discussion We need a discussion on requirements before calling this issue ready and removed status: Ready An issue that is ready to be worked on. labels Oct 3, 2022
@isabelle-dr isabelle-dr removed this from the Acceptance test improvements milestone Oct 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature request or improvement on an existing feature status: Needs discussion We need a discussion on requirements before calling this issue ready tests Anything related to our tests.
Projects
None yet
Development

No branches or pull requests

2 participants