Question on work-flow for big binary files

[Marcus Lindblom]

Shridhar Daithankar wrote:
> On Thursday 18 December 2008 19:06:50 Marcus Lindblom wrote:
>> It boils down to this: is it feasible with a distributed versioning
>> system of really large files? Perhaps I should run some tests, but it's
>> hard to predict performance a few years down the line.
>>
>> Anyone with experience in that field? How much data can one feasibly
>> work with in a DVCS? Am I trying to use the wrong tool for the job?
> 
> I would like to put another angle on it, just a thought that passed by :)
> 
> The situation is similar with databases. Lots of people ask if it is good to 
> put such large files in database, say as a part of website, images or 
> audio/video etc., to gain support from database transactions and ACID 
> guarantees. Many a times, acceptable solution is to put the file name in the 
> database and put the file in the file system.(I think similar solution is 
> suggested before, in terms of hg).

Yes. That would be the HgExternalBinaries as Douglas mentioned.

> Another thing you could look at, is ZFS/solaris which provides seamless 
> snapshots and it could replace your need for versioning in the short period of 
> time and could be space efficient too.

Does rsnapshot offer something similar? (I don't know much about it). 
IIRC we did some fast-backups on an OpenBSD server using rsnapshot, 
which was rumored to be space efficient by using links.

There still needs to be some client tool to 'commit' stuff to the 
server, which could then version stuff by snapshotting with hardlinks 
and filter out old intermediaries.

> Just a thought...

An interesting one.

Cheers,
/Marcus