Remove binary blobs from history?

[mbrukman]

The repo is currently 274MB when cloned, and just the src tree (i.e., excluding past versions in .git) is 85MB — the actual code takes up negligible space, and the vast majority of that space is taken up by src/test/resources/gcd-head.zip.

This makes it take an unusually large amount of time to clone the repo, and it will likely grow every time this blob is updated to a new version.

Would it be possible to not version these large files (and ideally remove them from history while there are few forks), and get them from Maven or other sources, pinned to a specific version?

[aozarov]

I am not sure how to exclude past versions of a file (and was unaware that they are actually pulled when repo is cloned). I don't think it is likely that we will have much changes in gcd-head but the plan is to be depended on gcloud SDK which in the future will include the gcd tool and hopefully will be available via maven.

[ludoch]

I think it's about time to do this clean up before we are too successful:-)
Never done that, but I guess http://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery#Removing-Objects will help?

[aozarov]

This sounds dangerous (and I wonder if such a change, that is not a changelist could then be applied back on the github repository...)

In any case, I followed the directions there on my local branch.

$ git verify-pack -v .git/objects/pack/pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.
pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.idx pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.pack

$ git verify-pack -v .git/objects/pack/pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.idx | sort -k 3 -n | tail -3
997af117c72074594a71ed6b141f9675709ec948 blob 931373 52784 90364920
484475ed6fc310dc3eaf115309719344fcda6791 blob 89098873 89111817 273395
ac05237a254e2892d0eb3b497d12f54515df775c blob 99807981 99811510 127031733

$ git rev-list --objects --all | grep ac05237a254e2892d0eb3b497d12f54515df775c
ac05237a254e2892d0eb3b497d12f54515df775c src/test/resources/gcd-v1beta2-rev1-2.1.1.zip

(gcd-v1beta2-rev1-2.1.1.zip is what we want to prune).

$ git log --oneline --branches -- src/test/resources/gcd-v1beta2-rev1-2.1.1.zip
e68feca replace local gcd hack with a fixed version of gcd
c79887b Automatically invoke gcd for testing

$ git filter-branch --index-filter 'git rm --ignore-unmatch --cached src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' -- c79887b^..
$ rm -Rf .git/refs/original
$ rm -Rf .git/logs
$ git prune --expire now
$ git count-objects -v

But could not see any difference (I verified and repeated the steps but still no change).

I then found this: https://help.github.com/articles/remove-sensitive-data/
and use it to do the cleanup of the old /gcd-v1beta2-rev1-2.1.1.zip
$ git filter-branch --force --index-filter \

'git rm --cached --ignore-unmatch src/test/resources/gcd-v1beta2-rev1-2.1.1.zip'
--prune-empty --tag-name-filter cat -- --all

But this created a scary/nasty change-list per forked branch (I tried it in my fork) touching all files (this may be safe but was too scary for me, so I dropped it). Not to mention that for this to work all forks has to be re-based (rather than merge) to avoid re-introducing the object again...

[aozarov]

Any more suggestions on how to remove src/test/resources/gcd-v1beta2-rev1-2.1.1.zip which was introduced by this changelist and later got replaced with src/test/resources/gcd-head.zip by this changelist?

[jgeewax]

@ryanseys : Any input here? Is there a "best practice way" to remove large binaries from history? The link @ludoch provided seems pretty straightforward.

We'll be totally re-writing history (which sucks, I know) and therefore fucking over the 11 forks that exist today, but I think making 11 people unhappy is better than having giant binaries hanging around in the git repository...

[jgeewax]

@aozarov

But this created a scary/nasty change-list per forked branch (I tried it in my fork) touching all files (this may be safe but was too scary for me, so I dropped it).

I think thats supposed to happen. You're effectively re-writing history, no?

[ryanseys]

Yep that github link to remove sensitive data looks on point. You are rewriting all history when deleting the references to the files so the SHAs will change and screw over any forks or clones so potentially way more than 11 people. But this is the only way to permanently remove large blobs from your repo.

[jgeewax]

Yep, sounds like a necessary evil :(

[aozarov]

Is it really a necessary evil? When I clone my fork where I applied all this git scary magic I get:

remote: Counting objects: 26043, done.
remote: Compressing objects: 100% (204/204), done.
remote: Total 26043 (delta 17), reused 0 (delta 0), pack-reused 25829
Receiving objects: 100% (26043/26043), 195.57 MiB | 42.28 MiB/s, done.
Resolving deltas: 100% (22834/22834), done.

When I clone the non-fork (GoogleCloudPlatform/gcloud-java) I get:

cloning into 'gcloud-java'...
remote: Counting objects: 70531, done.
remote: Compressing objects: 100% (1818/1818), done.
remote: Total 70531 (delta 1666), reused 0 (delta 0), pack-reused 68705
Receiving objects: 100% (70531/70531), 224.72 MiB | 42.19 MiB/s, done.
Resolving deltas: 100% (65692/65692), done.

So the % difference is not that big and honestly I am not comfortable to do it blindly (as
my knowledge in git is very basic, I just follow the steps without really understanding them
or their implication).

I still didn't manage to recover my fork which keep saying that it is "This branch is 171 commits ahead of GoogleCloudPlatform:master" (without any file change). Yes, I understand that it is re-writing history (though I would expect less
history to be behind not ahead).

Any advice on how to reset my fork to a fresh start would be much appreciated. After that (and after I know how to recover from it) I don't mind to give it another try (and let someone that understand review that push request or better yet do it himself).

[jgeewax]

OK I think the bigger problem is that this library is 200 MB...

That's an order of magnitude larger than gcloud-python, which has been around for quite a bit longer:

Cloning into 'gcloud-python'...
remote: Counting objects: 16892, done.
remote: Compressing objects: 100% (139/139), done.
remote: Total 16892 (delta 91), reused 0 (delta 0), pack-reused 16753
Receiving objects: 100% (16892/16892), 29.53 MiB | 5.69 MiB/s, done.
Resolving deltas: 100% (12957/12957), done.

It also seems weird that the number of objects is so large. From the logs above, it looks like your fork had 26,043, whereas GCP/gcloud-java had 70,531? How did we get a 3x multiplier there? Are we sure there aren't more blobs floating around?

The following stats just don't look quite right to me...

Library	Services	Commits	Start date	Size
gcloud-java	2	259	Nov 2014	224.72 MiB
gcloud-python	3	1408	Jan 2014	29.53 MiB

[ryanseys]

Yeah there's a definite difference.

Before:

After:

$ git clone https://github.com/ryanseys/gcloud-java.git ryanseys-gcoud-java

Results:

Cloning into 'ryanseys-gcoud-java'...
remote: Counting objects: 4017, done.
remote: Compressing objects: 100% (795/795), done.
remote: Total 4017 (delta 1994), reused 4017 (delta 1994), pack-reused 0
Receiving objects: 100% (4017/4017), 788.63 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1994/1994), done.
Checking connectivity... done.

[jgeewax]

@ryanseys : Which commits / blobs did you chop out ?

[ryanseys]

src/test/resources/gcd-v1beta2-rev1-2.1.1.zip
src/test/resources/gcd-head.zip

[ryanseys]

You don't really chop out commits, you chop out files (listed above) from the index and as a result, you affect every commit that follows the earliest commit that added those files to the index.

How to reproduce:

Step 0. Fork and clone repo.

git clone git@github.com:{{username}}/gcloud-java.git

Step 1. Remove src/test/resources/gcd-v1beta2-rev1-2.1.1.zip from index.

git filter-branch --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' --tag-name-filter cat -- --all

Step 2. Remove src/test/resources/gcd-head.zip from index.

git filter-branch --force --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-head.zip' --tag-name-filter cat -- --all

Step 3. Ignore those files from now on, make a new commit.

echo 'src/test/resources/gcd-head.zip' >> .gitignore
echo 'src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' >> .gitignore
git commit -am 'Remove files'

Step 4. Garbage collect outdated blobs and force push back to your fork.

(WARNING THIS WILL REWRITE HISTORY IN YOUR FORK)

git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
git push origin --force --all

Step 5. Clone it under a new name, enjoy the speed up!

git clone git@github.com:{{username}}/gcloud-java.git gcloud-java-new

Let me know if this doesn't work for you!