Corrupt Objects in Git Object Database

This past week I tried converting the Git repositories for our projects to Mercurial, why we are converting is another story, and I ran into this message

 fatal: git cat-file 4429f4f6f77856bb00e66dc24121d565528f97ff: bad file
 transaction about!
 rollback completed
 about: cannot read 'blob' object at 4429f4f6f77856bb00e66dc24121d565528f97ff

about three hundred commits into the conversion. Some quick searching of this error message via my favorite search engine and I found postings about unpacking the Git object database and other general rooting around under-the-hood to replace bad objects by hand. There is a document here where Linus himself discussing this process, and gives a brief comment on how corrupted objects happen.

I'm not too proud to admit it took me some time to pierce the abstraction of working directly with the Git object database which is why I'm writing this post, so hopefully I will save others the same trouble as me. This is kind of a for-dummies manual, of which I'm a card carrying member, to help those of us of average ability cope in a world of super-geniuses which I seem to be surrounded by.

The first step was to figure out what was going on here, the 'blob' is a 'file-blob' representing the change-set that was added to the object database when a commit was made. Some how this file blob has been corrupted, when the checksum for this 'file-blob' is looked up in some register upon retrieval the checksum in the register and the checksum generated by the corrupt blob do not match and this error is thrown. That is my understanding of the problem anyway I may still be wrong.

Moving along, using the Git command 'fsck' from inside the Git repository that I'm trying to convert, I will be able to find the commit and object database object that are associated with this 'file-blob' that is bad.

 $ git fsck --tags 4429f4f6f77856bb00e66dc24121d565528f97ff
 tagged commit b1ef38c0aa901524362786a6405d60d1d7bd80a0 (v1.0.0) in 5adad5a9f49f396e21881ce8b85e77c729a4f050

The second checksum output is the object database object that is associated with the bad 'file-blob', now to gain access to the object in the object database, and to do that you have to unpack the object database.
The object database is in a 'pack' several directories down into the repository (the hidden .git directory), you are first going to move the pack, then open it, Git needs either the packed or unpacked object database to function. We are going to move the directory containing the packed object database out of the way so that we can work directory with the unpacked object database. The follow commands are all relative to the root of the Git repository containing the .git directory, first we are going to move the directory containing the packed object database out of the way, and there may be several object database packs in this directory depending on how old your clone of the repository is.

 $ mv .git/objects/pack/ .git/objects/pack.old

Once the pack directory has been renamed to pack.old we are going to unpack the object database, you may need to repeat the steps from here on out to find the object associated with your corrupt 'file-blob'.

 $ git unpack-objects -r < .git/objects/pack.old/pack-1ca2ace5e54b94cf0312806533612ec0c5ee1249.pack

This will unpack the object database inside the .git/objects database, you will see several dozen directories with two character names.

NOTE: This is very import to realize and was one of the major stumbling blocks for me, the two characters in the directory names are the first two characters of the database objects contained within. This means that is when using 'git fsck --tags' you get the checksum

 5adad5a9f49f396e21881ce8b85e77c729a4f050

The database object will be in the unpacked directory '5a' and be in the file 'dad5a9f49f396e21881ce8b85e77c729a4f050', so using 'find' this is how I located the file containing the database object which contains the bad 'file-blob'.

 $ find .git/objects/ -name dad5a9f49f396e21881ce8b85e77c729a4f050
 .git/objects/5a/dad5a9f49f396e21881ce8b85e77c729a4f050

So there it is, the database object containing the bad 'file-blod'. Because there maybe several packed copies of the object database you might want to rerun the same command that generated the error message and see if it has gone away, you may have unpacked a packed object database that contains a good copy of that database object before the corruption took place.

The trick to all this is you need to find good copy of the database object to fix the problem, either from backups or from older packs that contain a non-corrupted version of the object. Since the SHA1 checksum is so secure there is no way I know of to backwards engineer database object from scratch.