Status of lfs and lock extensions

[Matt Harbison]

On Thu, Jan 21, 2021 at 7:12 AM Daniele Benegiamo
<danielebenegiamo at fastwebnet.it> wrote:
>
> On 2021/01/19 19:33, Matt Harbison wrote:
> >>> [...]
> >>> The LFS extension is shipped with Mercurial.  Don't let the fact that it
> >>> is marked experimental scare you off- I use it in production.  The
> >>> experimental tag is mostly so that we can change some of the
> >>> fileset/revset/template functionality without worrying about backward
> >>> compatibility.  I have no intention of changing the storage layout in an
> >>> incompatible way.  There is a TODO list of future plans that I hope to
> >>> get to some day:
> >>>
> >>>      https://www.mercurial-scm.org/repo/hg/file/5.6.1/hgext/lfs/TODO.rst
>
>  From a quick look at the history of the pointed directory, it seems
> there's not much activity on the extension since ~1 year (at least on
> that branch/repo). Are there any plans on if/when it will move out from
> the "experimental" status? [just to understand the long term plan about
> this feature in the Mercurial's team vision]

Yeah, this was something started by Facebook, and then I did enough
work on it to make it meet my needs for work.  I don't think the other
developers use it, so they're only changing it when refactoring things
requires it.  The "plan" is to complete most/all of the TODO list (or
at least the parts that would be backwards incompatible changes like
the filesets/templates) before removing the experimental label.  Right
now I'm pretty busy with python3 porting on Windows and TortoiseHg, so
I don't have a date in mind.  But again, don't let the label scare you
off- it's just that there are planned behavior changes noted in the
TODO.  The only sharp edge you may run into is with `hg grep` wanting
to download the files to search, but it's probably pretty easy to make
that command use the raw data instead (which will cause it to search
the lfs pointer data instead of the blob, effectively skipping it).

> Many devs working on multimedia/interactive projects (including
> simulation, VR and video games of course - projects that often rely on
> game engines and proprietary binary files) are migrating from Subversion
> and Perforce to git+git-lfs because DVCSs have very pleasant advantages.
> It would be great to have Mercurial in the set of the alternatives.

Agreed.  If you'd like to try your hand at implementing some of it,
I'm happy to mentor/give advice.

> > [...]
> >> I'll test it for sure. Can I follow the instructions in the extension
> >> documentation to set it up?
> >>
> >>          https://www.mercurial-scm.org/repo/hg/file/5.6.1/hgext/lfs/__init__.py
> >
> > Yes, that should work.  I'd ignore the `lfs.track` config, and track
> > `.hglfs` instead.  You won't need `lfs.url` either if you use
> > Mercurial as the server, but it sounds like you may need to point to a
> > git server if you want locking.  (Unless you get the lock extension
> > working, and transition to lfs locking in the future.  But IDK the
> > state of that extension.)
>
> Thanks! Accordingly to the docs, in case I need to specify the lfs.url
> to test other git-lfs backends, I must set it in the global config file,
> right?

You can set the config in any config file.  If you have multiple
repos, you probably want it in the repo level .hg/hgrc, since the url
is repo specific.

[...]

> > The protocol is basically a client request and reply sequence:
> >
> > HTTP POST:
> > C: "I'd like to upload these blobs, here are the hashes"
> > S: "OK, here is a list of URLs, one per blob that needs to be
> > uploaded, ignoring stuff I already have"
> >
> > HTTP PUT:
> > C: "Here's blob 1 content"
> > S: success/failure status
> > C: "Here's blob2 content"
> > ....
>
> It sounds easy. But usually problems are in the details :)

Yeah, but this isn't something you need to worry about unless you're
doing development work or looking at wireshark traces to debug.

> >> Just to be sure to have understood correctly, the current version of
> >> "hgweb" can be used as a remote git-lfs endpoint, and so it can be used
> >> directly with the "lfs" extension on the clients? Or we need other
> >> specialized software on server to run the tests?
> >
> > Correct.  Obviously you need to also enable the extension on the
> > server also.  The only caveat is that it doesn't currently support the
> > locking protocol.
>
> If some existing git-lfs implementation can manage the lock protocol
> separately from the data storage (not sure this is possible at all...),
> transparently forwarding requests to it could provide a quick
> implementation of the protocol on the Mercurial server (at prototype
> level at least) [of course after adding the needed functionalities on
> the client].

I'm not sure I follow this.  It seems like from reading the spec, the
client makes the request to the server to lock the file, and gets a
yes/no result back.  It's not separate from the data storage (unless
the server supports telling the client to store it in some other
location, but the lfs-server is still in control).  The hg test file
for `lfs-test-server` never initializes a git repo, so in theory a git
repo isn't required with at least some 3rd party server, if that's
what you mean.

I wasn't thinking too deeply about how you could handle the locking
outside of Mercurial.  I figured maybe write a quick and dirty tool
that could either use the API to lock it, or maybe some of the fancier
servers have a web interface to manage the locks?  Maybe that's too
much hassle though, and it might just be easier to implement it (the
client part anyway) in Mercurial.  It might be worth looking at the
Facebook repo to see if they have locking support yet.  If so, it
could be really easy to borrow.

I did a quick read through the spec, and I'm a bit puzzled by it.  It
seems much stripped down from the original proposal (which talked
about checking out files read-only unless they were unlocked, among
other things).  But otherwise, I couldn't figure out what the benefit
was.  It lets you edit and commit the files locally, and only yells
when you go to push.  But at that point, you've made the edits, and
potentially have to redo them if someone else did as well.  If there
were support (or you were using git), do you have a workflow in mind?

> >>> I didn't try any file locking (and it's not implemented in Mercurial
> >>> client), but maybe if you've already got a 3rd party server and can send
> >>> it the right commands to lock it on the server, it will (mostly) work? I
> >>> didn't look at any of the lfs extensions or proposals when working on
> >>> the server, because I needed the bare bones implementation.
> >>
> >> So the "lfs" extension could work with the current git-lfs v2 reference
> >> implementation (https://git-lfs.github.com)?
> >
> > I don't remember a v2 reference when I implemented the server support
> > in early 2018.  The batch command and basic-transfers specs I used
> > look basically unchanged since then:
> >
> >      https://github.com/git-lfs/git-lfs/tree/master/docs/api
> >
>
> Accordingly to the note "Added: v2.0" in their documents, I think
> locking was the main addition to the v2 specs.

Yeah, I think I saw that, but didn't see any other references to v2.0 specs.

> >> I see the locking API is documented here:
> >> https://github.com/git-lfs/git-lfs/blob/master/docs/api/locking.md
> >>
> >> If I have understood, an hypothetical client "lfs-lock" extension on
> >> Mercurial could use that API to manage locks on server (if supported by
> >> the server of course). The server should then enforce the locks on push.
> >
> > Yes, though the functionality should probably just be built into the
> > lfs extension.
>
> I don't know how Mercurial extensions works, but I suppose that an
> extension can define distinct "hooks" for when it's running on the
> client and the server?

Yeah, extensions have free reign to do what they want inside the
Mercurial process.  I'm just saying if someone adds `hg lfs-lock`, `hg
lfs-unlock`, etc, it should go into the existing extension.  Hooks are
mostly for external things, but the extension already wraps some
internal stuff for `hg push` for example.  So that's probably a
natural place for the messages about things that were pushed that need
to be unlocked.  And that's likely true of other areas that need
support.