Call for comments on new dirstate format contents

[Raphaël Gomès]

Hello all,

As you probably know my colleagues at Octobus and I have been working on 
a new version of the dirstate, and we're coming pretty close to 
something usable in production, so we need to freeze the format soon. 
This email is not meant to discuss the exact byte-per-byte layout 
details of the format, but rather its contents: what do you think should 
be included (or at least have space reserved for) in the new version?

We have already discussed this at previous sprints and various other 
discussion channels, but I thought it'd be better to give a "last call" 
chance for people to get their voices heard.

I remember Google people saying they'd like to separate information that 
is frequently written to a separate file to help with their filesystem 
shenanigans. What exactly would be the plan and can we do it easily? I 
may be pessimistic, but this looks like it would require a lot of work 
which (so far) no one wants to sponsor, though I'm happy to be proven 
wrong either way.

To Matt Harbison: you said something about storing exec bit and symlink 
info explicitly to help platforms like Windows that don't have them, 
could you please elaborate?

As a general recap (and to help understand some decisions), the new 
format will be an append-only tree with no stem compression for 
performance reasons. The Python implementation will be functional but 
very basic and will offer no purposeful performance improvements (unless 
someone wants to have fun!), as we currently only have the bandwidth for 
optimizing the Rust implementation.

An overview of the current target (some implementation-detail level 
contents omitted):

 Â Â Â  - A docket file that contains global metadata about the dirstate:
 Â Â Â Â Â Â Â  - NodeID of the parents (32 bytes reserved, 20 used for now)
 Â Â Â Â Â Â Â  - A total count of files (including Removed ones)
 Â Â Â Â Â Â Â  - A count of dead (unreachable) bytes
 Â Â Â Â Â Â Â  - A count of alive (reachable) bytes
 Â Â Â Â Â Â Â  - A hash of ignore patterns (see 
https://phab.mercurial-scm.org/D10836)
 Â Â Â  - In the data file, for each directory/file (it can be both at the 
same time):
 Â Â Â Â Â Â Â  - The full path in bytes of the file (or directory)
 Â Â Â Â Â Â Â  - The full path of the copy source (optional)
 Â Â Â Â Â Â Â  - How many tracked recursive descendants it has
 Â Â Â Â Â Â Â  - How many recursive copies it has
 Â Â Â Â Â Â Â  - Exec bit
 Â Â Â Â Â Â Â  - mtime (probably up to nanosecond precision, both files and 
directories)
 Â Â Â Â Â Â Â  - Clean file size when applicable
 Â Â Â Â Â Â Â  - Its state: if it's removed, added, clean, etc.
 Â Â Â Â Â Â Â  - Whether it's from p1 or p2
 Â Â Â Â Â Â Â  - Whether it's ambiguous (it appears clean but the mtime is the 
same as the last status, probably will only happen with the Python 
implementation)
 Â Â Â Â Â Â Â  - All of the info needed to get the previous state of a Removed 
file in case we `hg add` it back
 Â Â Â Â Â Â Â  - (My idea as I type this: ) store the "raw bytes" version of 
the OS path if it differs from the normalized hg version (on Windows and 
MacOS for example) to cache the filefoldmap.

I *think* that's it? I might be wrong, if so, please tell me!

Raphaël