Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve testing infra for fetchTree/fetchGit #9388

Open
DavHau opened this issue Nov 19, 2023 · 3 comments
Open

Improve testing infra for fetchTree/fetchGit #9388

DavHau opened this issue Nov 19, 2023 · 3 comments
Labels
feature Feature request or proposal fetching Networking with the outside (non-Nix) world, input locking idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. tests

Comments

@DavHau
Copy link
Member

DavHau commented Nov 19, 2023

Is your feature request related to a problem? Please describe.
None of the code paths that could trigger actual networking seem to be tested as of now, limiting the test coverage to only file:// based actions.
I would like to add tests for potential bugs like this.

Describe the solution you'd like
Git already offers the git-http-backend CGI script. Therefore, all that's needed is a cgi capable http server. Python could be used, as it has all of that already built in, but having python as a dependency might be undesired due to bootstrapping complexity. Maybe there is a simpler cpp based http server that can be used.

Describe alternatives you've considered
Not testing network related code paths -> Instead rely on humans to find bugs during code review -> slow and unreliable.

Additional context

Priorities

Add 👍 to issues you find important.
@DavHau DavHau added the feature Feature request or proposal label Nov 19, 2023
@DavHau DavHau mentioned this issue Nov 19, 2023
Merged
@roberth
Copy link
Member

roberth commented Nov 19, 2023

I'd be happy to upstream into nixpkgs these modules that help set up a gitea. https://github.com/hercules-ci/hercules-ci-effects/tree/master/effects/testsupport

gitea is fairly quick to launch (unlike, say, gitlab) VM+gitea take about 10 s.

OTOH these a lot of value in keeping the test setup minimal.

might be undesired due to bootstrapping complexity.

Bootstrapping should be done without tests.

  1. bootstrap without tests
  2. have a Nix
  3. bootstrap with tests, using Nix to get the test deps

Also this could be a checks test instead of an installCheckPhase test.

@roberth roberth added fetching Networking with the outside (non-Nix) world, input locking tests labels Nov 19, 2023
@roberth roberth added this to Nix team Nov 19, 2023
@DavHau
Copy link
Member Author

DavHau commented Nov 19, 2023 •
edited
Loading

Yes, it would be nice if you could upstream these modules. I played around a bit wit the python http server, but it seems like advanced path rewriting capabilities are required in order to use thegit-http-backend. Just using gitea might be simpler.

@thufschmitt thufschmitt added the idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. label Dec 1, 2023
@thufschmitt thufschmitt removed this from Nix team Dec 1, 2023
@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-12-01-nix-team-meeting-minutes-108/36402/1

DavHau added a commit to DavHau/nix that referenced this issue Dec 29, 2023
@DavHau
initialize test suite for git fetchers
84af38d 
solves NixOS#9388

This utilizes nixos vm tests to allow:
- writing tests for fetchTree and fetchGit involving actual networking.
- writing small independent test cases by automating local and remote repository setup per test case.

This adds:
  - a gitea module setting up a gitea server
  - a setup module that simplifies writing test cases by automating the repo setup.
  - a simple git http test case

Other improvements:
For all nixos tests, add capability of overriding the nix version to test against.
This should make it easier to prevent regressions. If a new test is added it can simply be ran against any older nix version without having to backport the test.
For example, for running the container tests against nix 2.12.0:
`nix build "$(nix eval --raw .#hydraJobs.tests.containers --impure --apply 't: (t.forNix "2.12.0").drvPath')^*" -L`
@DavHau DavHau mentioned this issue Dec 29, 2023
Merged
DavHau added a commit to DavHau/nix that referenced this issue Dec 29, 2023
@DavHau
initialize test suite for git fetchers
73ae9ab 
solves NixOS#9388

This utilizes nixos vm tests to allow:
- writing tests for fetchTree and fetchGit involving actual networking.
- writing small independent test cases by automating local and remote repository setup per test case.

This adds:
  - a gitea module setting up a gitea server
  - a setup module that simplifies writing test cases by automating the repo setup.
  - a simple git http test case

Other improvements:
For all nixos tests, add capability of overriding the nix version to test against.
This should make it easier to prevent regressions. If a new test is added it can simply be ran against any older nix version without having to backport the test.
For example, for running the container tests against nix 2.12.0:
`nix build "$(nix eval --raw .#hydraJobs.tests.containers --impure --apply 't: (t.forNix "2.12.0").drvPath')^*" -L`
DavHau added a commit to DavHau/nix that referenced this issue Jan 9, 2024
@DavHau
initialize test suite for git fetchers
572179b 
solves NixOS#9388

This utilizes nixos vm tests to allow:
- writing tests for fetchTree and fetchGit involving actual networking.
- writing small independent test cases by automating local and remote repository setup per test case.

This adds:
  - a gitea module setting up a gitea server
  - a setup module that simplifies writing test cases by automating the repo setup.
  - a simple git http test case

Other improvements:
For all nixos tests, add capability of overriding the nix version to test against.
This should make it easier to prevent regressions. If a new test is added it can simply be ran against any older nix version without having to backport the test.
For example, for running the container tests against nix 2.12.0:
`nix build "$(nix eval --raw .#hydraJobs.tests.containers --impure --apply 't: (t.forNix "2.12.0").drvPath')^*" -L`
DavHau added a commit to DavHau/nix that referenced this issue Jan 11, 2024
@DavHau
initialize test suite for git fetchers
813c113 
solves NixOS#9388

This utilizes nixos vm tests to allow:
- writing tests for fetchTree and fetchGit involving actual networking.
- writing small independent test cases by automating local and remote repository setup per test case.

This adds:
  - a gitea module setting up a gitea server
  - a setup module that simplifies writing test cases by automating the repo setup.
  - a simple git http test case

Other improvements:
For all nixos tests, add capability of overriding the nix version to test against.
This should make it easier to prevent regressions. If a new test is added it can simply be ran against any older nix version without having to backport the test.
For example, for running the container tests against nix 2.12.0:
`nix build "$(nix eval --raw .#hydraJobs.tests.containers --impure --apply 't: (t.forNix "2.12.0").drvPath')^*" -L`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature Feature request or proposal fetching Networking with the outside (non-Nix) world, input locking idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@roberth @thufschmitt @DavHau @nixos-discourse