Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git-protocol: git protocol v2 machinery #724

Merged
merged 1 commit into from
Jul 28, 2021
Merged

git-protocol: git protocol v2 machinery #724

merged 1 commit into from
Jul 28, 2021

Conversation

kim
Copy link
Contributor

@kim kim commented Jul 2, 2021

Introduces a new crate with the lower-level machinery for handling
replication based on git protocol v2.

Signed-off-by: Kim Altintop [email protected]

@kim
Copy link
Contributor Author

kim commented Jul 2, 2021

@Byron would you mind taking a look?

It seems to work basically, but then hangs forever reading the packfile. By replacing copy with a loop, I can see that it reads exactly 95 bytes in three read calls, and then waits indefinitely ... for the flush pkt?

@kim
Copy link
Contributor Author

kim commented Jul 2, 2021

@Byron nevermind -- somewhat surprisingly, I need to spawn upload-pack in stateful mode, otherwise it doesn't send the capabilities advertisement. Now, I need to close its stdin after the packfile is sent to make the process terminate... 🤔

git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
@kim
Copy link
Contributor Author

kim commented Jul 2, 2021

@Byron nevermind the nevermind -- it's pretty bad, though, as the client can keep upload-pack running by simply not closing the connection. I wonder if I should just send made-up capabilities, so I can use --stateless-rpc.

@Byron
Copy link
Contributor

Byron commented Jul 2, 2021

@Byron nevermind the nevermind -- it's pretty bad, though, as the client can keep upload-pack running by simply not closing the connection. I wonder if I should just send made-up capabilities, so I can use --stateless-rpc.

Sorry for the hassle!

I will take a closer look tomorrow. It could be that this setup is different enough from the tests (which are against the daemon and against github) to cause something to be a little off. Failing to flush a flush packet line could certainly be a cause of this. The daemon might theoretically run into the same issue but somehow cleans out upload-pack when the connection is dropped, and I never noticed.

@kim
Copy link
Contributor Author

kim commented Jul 2, 2021

It does work as designed, except that the git-upload-pack executable doesn’t allow to turn on capability advertisement without also turning on refs advertisement: https://github.com/git/git/blob/master/builtin/upload-pack.c#L53

Sending a flush pkt would also cause the server to exit, but is not satisfying in our environment. So, until we have a replacement for upload-pack, we’ll need to assemble the capabilities advertisement by hand, and then fork upload-pack in stateless mode. I don’t see a better solution atm.

Copy link
Contributor

@Byron Byron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now I took a look at all code except for the tests which are merely postponed to another session.

All looks good to me and it's nice to see that git_protocol::client::fetch() does indeed compose.

I can't say anything about what to do with git-upload-pack, you seem to know what works and what doesn't, but if you think that gitoxide isn't speaking git correctly confusing the server in the process then it's definitely something to reproduce and fix.

git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
git-protocol/src/fetch.rs Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
git-protocol/src/upload_pack.rs Outdated Show resolved Hide resolved
git-protocol/src/upload_pack.rs Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@Byron Byron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the integration test was very helpful as I came to believe that the server hanging is a shortcoming of gitoxide that should be investigated and hopefully fixed. I don't fully understand what the expectations in the server are though as I think you mentioned it should not shut down when sending the flush packet, nor do I fully understand the interplay between stateless RPC and the daemon sending capabilities (instead of the upload-pack program).

test/src/test/integration/git_protocol.rs Outdated Show resolved Hide resolved
git-protocol/src/upload_pack.rs Show resolved Hide resolved
test/src/test/integration/git_protocol.rs Outdated Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
@kim kim force-pushed the kim/git2 branch 5 times, most recently from 6da938e to 9263108 Compare July 26, 2021 16:51
@kim kim marked this pull request as ready for review July 26, 2021 16:52
@kim kim requested a review from a team as a code owner July 26, 2021 16:52
@kim
Copy link
Contributor Author

kim commented Jul 26, 2021

This is now ready, I think.

FintanH
FintanH previously approved these changes Jul 27, 2021
Copy link
Contributor

@FintanH FintanH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👌

Byron
Byron previously approved these changes Jul 28, 2021
Copy link
Contributor

@Byron Byron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably it's best to use libgit2 pack writing for now as the gitoxide implementation of pack writing exists in isolation, that is isn't yet aware of any additional requirements that might apply to writing packs while dealing with locks, multi-pack indices and the likes.
Some more research should be done to be sure there is no unwanted side effects.

On the other hand, all it does it to write tmp files and rename the finished files to move into their desired spot. This should be find if git is indeed backwards compatible with itself.

I also left a comment about using gitoxide for setting up the test fixture, maybe that could be done in a separate PR contribute by me if there is interest.

Besides that: YAY 🎉, gitoxide lands in Radicle-Link :)

git-protocol/src/packwriter.rs Outdated Show resolved Hide resolved
git-protocol/src/fetch.rs Outdated Show resolved Hide resolved
test/src/test/integration/git_protocol.rs Show resolved Hide resolved
}

#[test]
fn thin_pack_gitoxide() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am definitely loving these tests and am glad it does indeed work as expected.
Currently thin pack support is tested on a thin pack of gitoxide's own making only.


let tree = {
let empty = repo.treebuilder(None).unwrap();
let oid = empty.write().unwrap();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually all of work done here except for Repository::init_bare(…) and .set_namespace() can already be done in gitoxide. Maybe once all is supported I could contribute a PR which oxidizes the test setup.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That'd be awesome.

Introduces a new crate with the lower-level machinery for handling
replication based on git protocol v2.

Signed-off-by: Kim Altintop <[email protected]>
@kim kim dismissed stale reviews from Byron and FintanH via e3e127d July 28, 2021 07:08
@kim kim merged commit e3e127d into master Jul 28, 2021
@kim kim deleted the kim/git2 branch July 28, 2021 07:16
@github-pages github-pages bot temporarily deployed to github-pages July 28, 2021 07:16 Inactive
Byron added a commit to GitoxideLabs/gitoxide that referenced this pull request Aug 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants