Question on work-flow for big binary files

[Marcus Lindblom]

  Arne Babenhauserheide wrote:
> Am Mittwoch 17 Dezember 2008 23:26:16 schrieb Marcus Lindblom:
>> It seems to be no big problem in using Mercurial to manage sources, but
>> there's also the matter of content files (3d-models, textures, whatnot)
>> that have a different characteristics/usage patterns than source code:
>>
>>   * are big (10-100:s of megabytes)
>>   * change in bursts (a few weeks with 1-4 commits/day), then are stable
>> for a long time (so some near-time rollback is necessary, but ancient
>> history is not important)
>>   * are often binary
>>
>> With svn, I could imagine a scheme where you'd store content-files in a
>> separate repo which was processed (dumped/filtered/restored) every week
>> or so to remove any intermediate copies above a certain age, of course
>> keeping heads of any tags and branches.
> 
> Since you only need short-time version tracking of these files, I'd just 
> create a clone of the main repo for working on certain content files, and once 
> they are stable just copy them into main (without their history). 

[snip]

> If you need to rework the character later on, you just create another clone 
> from main and work there. 
> 
> After the files get stable, you can just throw away (or backup) the clone and 
> create a new one for the next set of content items. 

I see. Thanks for the informative reply. :) It would indeed work well 
for small-to-average content files, and I hadn't thought of it from that 
perspective.

But, thinking about it, I have some additional concerns (which again 
stem from svn-experience, I'm very thankful for the input):

  * The total content size will probably become huge (2-20 gb) over 
time, so a clone is not _that_ cheap.
  * Not all content needs to be accessible all the time (different 
applications have different content, but use the same platform), so 
every developer doesn't need all content all the time.
  * If one is not on-site, cloning a multi-gig repo probably isn't feasible.

Perhaps I should have several content-repos per application, and use 
hgforest or something with selective checkouts?

This seems to touch on ShallowClone, PartialClone, TrimmingHistory and 
EditingHistory, a bit here and there. :)

> Always remember that cloning is cheap - and safe. 

I'm trying to come to terms with that. :) But cloning repositories where 
the working copy is 2-20 gb (and the entire thing, with changes and 
branches might be even bigger) is not really cheap?

Or perhaps it's cheap enough, compared to the alternatives?

I'm thinking a custom tool & server db for just storing and sharing 
large file-tree, while avoiding putting too much stuff on the clients. 
This is what SVN and other centralized systems offers. OTOH, you can't 
do local commits and rollbacks, which is nice. Perhaps a local 
mercurial-repo with some system for centralized sharing?

Thinking out loud a bit:

Using Hg as-is will incur a bit of an administrative procedure since we 
want to share the clone amongst developers, so if we need to clone and 
work, someone needs to create a repo on our server that people use to 
share. Not that it's much work. (mkdir, hg clone). I could setup a small 
webapp for that.

It will take some diskspace (but.. the server is linux, clients are 
windows, so the server will be able to use hardlinks, and the clients 
too, once vista-hardlink-support is in mercurial). We have quite a lot 
of diskspace currently anyway. :)

...

I think I'm learning the differences between distributed and centralized 
systems. :)

It boils down to this: is it feasible with a distributed versioning 
system of really large files? Perhaps I should run some tests, but it's 
hard to predict performance a few years down the line.

Anyone with experience in that field? How much data can one feasibly 
work with in a DVCS? Am I trying to use the wrong tool for the job?

Cheers,
/Marcus