git-lfs support

[extrawurst]

Our current bug #2809 (our broken pre-push hook support) led me down the rabbit hole of "what else is required to properly support git-lfs in gitui.

We are looking at basically three things:

1. properly run pre-push hook (thats Pushing throws pre-push hook error on repository with LFS. #2809)
2. hook into filter logic and run them on all checkout-like operations (smudge)
3. hook into filter logic and run them on all worktree->index operations (clean)

Now libgit2 exposes an API to hook into filters but git2-rs does not expose this right now (open issue).

Since our end goal is to be libgit2 independent I call out whats needed to do this all using gitoxide:

1. we need to migrate all our git-checkout locations to gitoxide
2. migrate all worktree->index operations to gitoxide
3. use gitoxide machinery to hook into filters (turns out gitoxide will already invoke filters for us)
4. forward filter logic via shellout (git-lfs clean, git-lfs smudge or git-lfs filter-process)
5. finally we support basic git-lfs
6. add support for more file locking relevant lfs features via hooks: post-checkout, post-merge, post-commit

This issue is more of an epic, I would love to see contributions to take on individual steps as separate PRs.

cc @Byron @cruessler correct me if I am wrong but from the gitoxide side all this is ready to be used or am I wrong?

Further reading: #1089

[Byron]

Thanks for the summary! Here are my notes:

we need to migrate all our git-checkout locations to gitoxide

This will already work as long as you only checkout individual files, see filter_pipeline.

migrate all worktree->index operations to gitoxide

This will also work as long as checkins are per-file, see filter_pipeline.

use gitoxide machinery to hook into filters

I don't understand where this requirement is coming from, I am just missing a hint on the feature that this supports.
My potshot here is that in theory, as filters are invoked by gitui on a per-file basis, pre- and post- filter hooks are naturally present.

forward filter logic via shellout (git-lfs clean, git-lfs smudge or git-lfs filter-process)

Standard Git filters are naturally supported and invoked, there is no extra action needed unless gitui wants to support git-lfs specifics that aren't covered by standard Git filters.

finally we support git-lfs

I think the crux here would be that gitoxide has no mechanism of doing a worktree checkout yet, and the one that's planned will be a brute force "apply the changes that are needed to turn tree A into tree B", overwriting anything in its path. It will be very much on the plumbing side of things. GitButler needs such a primitive as it does the analysis on what to do with changed or untracked files that would be in the way. Something like it would have to be ported to over then, while just adhering to the 'Git-style' of doing this (i.e. force, non-force, for starters).

cc @Byron @cruessler correct me if I am wrong but from the gitoxide side all this is ready to be used or am I wrong?

Besides the checkout, there are weaknesses when interacting with the .git/index - it's very much expert mode and plumbing level. But GitButler uses that to get what it needs, so it can be done.

[extrawurst]

I don't understand where this requirement is coming from

I guess I was under the assumption that gitoxide would make me implement the filter logic myself just like libgit2.

filters are naturally supported and invoked, there is no extra action needed

That is awesome. Does it support both smudge/clean and the more modern filter-process?

gitoxide has no mechanism of doing a worktree checkout yet

I understand but if we checkout a worktree on a per-file-basis it should already work? Does that have some specific downsides other than being more laborious?

[Byron]

That is awesome. Does it support both smudge/clean and the more modern filter-process?

Yes, if more modern means the single-invocation ones. For all I know, filters are fully supported.
It's implemented as the caller choosing a direction - convert data for storage in Git, or for placement in the worktree.

I understand but if we checkout a worktree on a per-file-basis it should already work? Does that have some specific downsides other than being more laborious?

I wouldn't recommend it - the worktree is holy and needs a lot of care. If a gix client was to implement this 'properly', they would write the implementation that gix should have had in the first place. And everything below 'properly' really shouldn't be unleashed into this world 😁. Trust me 🙇‍♂️.

With that said, when gix clone is used, gix will checkout the worktree, but it can only do it safely (and very quickly) because it has exclusive access to a known-to-be-empty directory. When the user also has access to it, it gets way more involved.
With that said, all the code to do it 'properly' is there already, it just needs to be combined and tested to death to not be worse than Git (or better: implement all the knowledge that Git gathered in its 20 years or so. This is what makes it so dangerous - treacherously simple, and full of mines right next to foot guns.)

Yeah, so ensuring its correctness is certainly the most time consuming part, along with implementing it in single-threaded and multi-threaded mode to have another reason to exist (why do it if you can't make it better/faster).

Thanks for nerd-sniping me into analysing this more than I have ever consciously done 🙏.

[extrawurst]

With that said, all the code to do it 'properly' is there already

@Byron is there a gitoxide issue to track for proper git checkout support? For now this seems like the big thing this is hinging on and what I would wait for first.