When does it make sense to use subrepositories?

[Boggess, Rod]

-----Original Message-----
From: mercurial-bounces at selenic.com
[mailto:mercurial-bounces at selenic.com] On Behalf Of Angel Ezquerra
Sent: Friday, April 19, 2013 8:37 AM
To: Matt Mackall
Cc: v; Mercurial
Subject: Re: When does it make sense to use subrepositories?

On Thu, Apr 18, 2013 at 11:40 PM, Matt Mackall <mpm at selenic.com> wrote:
> On Thu, 2013-04-18 at 23:04 +0200, Angel Ezquerra wrote:
>> On Thu, Apr 18, 2013 at 8:52 PM, Matt Mackall <mpm at selenic.com>
wrote:
>> > On Thu, 2013-04-18 at 09:36 -0700, v wrote:
>> >> I've just backed out of several hours' work transitioning a set of

>> >> related projects from one repository to a group of 
>> >> subrepositories. Having been through this a few times before, I 
>> >> always seem to end up with one of two
>> >> scenarios:
>> >>
>> >> - If changes to one project are usually associated with changes to

>> >> other related projects, put them in the same repository.
>> >> - Otherwise, use separate repositories, and handle the 
>> >> relationship with you favourite dependency management solution. If

>> >> you really want to do this with your SCM, then use guestrepos.
>> >>
>> >> I do acknowledge that subrepos are now declared "a feature of last

>> >> resort", but I'm wondering what this situation is. Someone paid 
>> >> for them to be developed, so they must be useful somewhere!
>> >
>> > Most people are unclear on the fundamental purpose of version
control.
>> > They think of VCSes as first and foremost a tool to synchronize 
>> > code between developers. But this is wrong; it is a description of 
>> > rsync. The primary purpose of an VCS is to precisely track old 
>> > states of the project for future reference.
>> >
>> > With that in mind, ask yourself the question:
>> >
>> > "Do I need to be able to reproduce an exact combination of 
>> > independent projects from the past in the future?"
>> >
>> > In other words, "do I need _version control_ on my combination of 
>> > projects?"
>> >
>> > If the answer is yes, your options are:
>> >
>> > - check everything into one repo and hope you don't have to merge 
>> > in "upstream" changes often
>> > - use subrepos
>>
>> I could not have said it better! I think subrepos cover the 
>> requirement that Matt mentioned pretty well, and that is a very 
>> important and common requirement.
>>
>> I'm very happy to see that Matt did not immediately refer to them as 
>> a feature of last resort.
>
> It is absolutely a feature of last resort. I do not know why people 
> have so much trouble parsing the semantics of that.

Personally when I read that something is "a feature of last resort" I
used to imagine Bruce Willis jumping down a 40 story building, guns
blazing, while the bad guys shoot at him in one of the Die Hard
movies... Using subrepos is just not that exciting :-)

> It is a feature: it does things you can't do with normal Mercurial.
> It is a last resort: you do not want to use it if you don't need to.

I think now I get what you mean, and under that definition I agree with
you. However, on the "FeaturesOfLastResort" wiki page, it says:

"These features exist for supporting some (usually legacy) use cases,
but are not considered best practice. Needing (or thinking you need)
these features is often an indicator of a _bad practice_ somewhere else
in your workflow." (emphasis mine).

This is a much stronger statement than what you said above (IMHO). The
key word is "often" but I think most people (me included) would
interpret that statement as "they are almost always an indicator of a
bad practice". Do you think that subrepos fit that definition? Are they
really an indicator of a bad practice? If so, what is the alternative?

> Given the number of n00bs who have tried to use subrepos when they 
> shouldn't and get teh sad, we need to advertise the latter fact to 
> scare them off for their own good.

I understand what you mean and I even think you are right, but (to me)
subrepos do not fit the definition of "feature of last resort"
provided on the Wiki.

It is one thing to say that you should be careful and not decide to use
subrepos lightly, and quite another to suggest that if you use them you
are doing something wrong, which is the impression I get from many
messages on these mailing lists. That is why I think subrepos get an
undeserved bad rap.

Angel
_______________________________________________
Mercurial mailing list
Mercurial at selenic.com
http://selenic.com/mailman/listinfo/mercurial

From: mercurial-bounces at selenic.com
[mailto:mercurial-bounces at selenic.com] On Behalf Of Angel Ezquerra
Sent: Friday, April 19, 2013 8:37 AM
To: Matt Mackall
Cc: v; Mercurial
Subject: Re: When does it make sense to use subrepositories?

On Thu, Apr 18, 2013 at 11:40 PM, Matt Mackall <mpm at selenic.com> wrote:
> On Thu, 2013-04-18 at 23:04 +0200, Angel Ezquerra wrote:
>> On Thu, Apr 18, 2013 at 8:52 PM, Matt Mackall <mpm at selenic.com>
wrote:
>> > On Thu, 2013-04-18 at 09:36 -0700, v wrote:
>> >> I've just backed out of several hours' work transitioning a set of

>> >> related projects from one repository to a group of 
>> >> subrepositories. Having been through this a few times before, I 
>> >> always seem to end up with one of two
>> >> scenarios:
>> >>
>> >> - If changes to one project are usually associated with changes to

>> >> other related projects, put them in the same repository.
>> >> - Otherwise, use separate repositories, and handle the 
>> >> relationship with you favourite dependency management solution. If

>> >> you really want to do this with your SCM, then use guestrepos.
>> >>
>> >> I do acknowledge that subrepos are now declared "a feature of last

>> >> resort", but I'm wondering what this situation is. Someone paid 
>> >> for them to be developed, so they must be useful somewhere!
>> >
>> > Most people are unclear on the fundamental purpose of version
control.
>> > They think of VCSes as first and foremost a tool to synchronize 
>> > code between developers. But this is wrong; it is a description of 
>> > rsync. The primary purpose of an VCS is to precisely track old 
>> > states of the project for future reference.
>> >
>> > With that in mind, ask yourself the question:
>> >
>> > "Do I need to be able to reproduce an exact combination of 
>> > independent projects from the past in the future?"
>> >
>> > In other words, "do I need _version control_ on my combination of 
>> > projects?"
>> >
>> > If the answer is yes, your options are:
>> >
>> > - check everything into one repo and hope you don't have to merge 
>> > in "upstream" changes often
>> > - use subrepos
>>
>> I could not have said it better! I think subrepos cover the 
>> requirement that Matt mentioned pretty well, and that is a very 
>> important and common requirement.
>>
>> I'm very happy to see that Matt did not immediately refer to them as 
>> a feature of last resort.
>
> It is absolutely a feature of last resort. I do not know why people 
> have so much trouble parsing the semantics of that.

Personally when I read that something is "a feature of last resort" I
used to imagine Bruce Willis jumping down a 40 story building, guns
blazing, while the bad guys shoot at him in one of the Die Hard
movies... Using subrepos is just not that exciting :-)

> It is a feature: it does things you can't do with normal Mercurial.
> It is a last resort: you do not want to use it if you don't need to.

I think now I get what you mean, and under that definition I agree with
you. However, on the "FeaturesOfLastResort" wiki page, it says:

"These features exist for supporting some (usually legacy) use cases,
but are not considered best practice. Needing (or thinking you need)
these features is often an indicator of a _bad practice_ somewhere else
in your workflow." (emphasis mine).

This is a much stronger statement than what you said above (IMHO). The
key word is "often" but I think most people (me included) would
interpret that statement as "they are almost always an indicator of a
bad practice". Do you think that subrepos fit that definition? Are they
really an indicator of a bad practice? If so, what is the alternative?

> Given the number of n00bs who have tried to use subrepos when they 
> shouldn't and get teh sad, we need to advertise the latter fact to 
> scare them off for their own good.

I understand what you mean and I even think you are right, but (to me)
subrepos do not fit the definition of "feature of last resort"
provided on the Wiki.

It is one thing to say that you should be careful and not decide to use
subrepos lightly, and quite another to suggest that if you use them you
are doing something wrong, which is the impression I get from many
messages on these mailing lists. That is why I think subrepos get an
undeserved bad rap.

Angel
_______________________________________________
Mercurial mailing list
Mercurial at selenic.com
http://selenic.com/mailman/listinfo/mercurial

Most of the projects I work on these days (dot net) have DLLs by the
nature of how they're built. Not all of those warrant a separate
project, but many do. So, the concept of a sub-repository is very
inviting as a solution for keeping the various add-ins in synch with the
main project(s). I also work on a client-server solution, where the
client's message parsing depends greatly on shared data structures with
the server, so those two need to be maintained in synch. (This last was
my primary interest in looking at Subrepository as a feature that might
help in doing this.)

Because of the warning on the Wiki that this is a feature of last
resort, I backed off. That was a Good Thing. Please don't remove the
warning.

As you can imagine, I've been watching related conversations about how
to do this closely, and experimenting with nested repositories, named
branches, and even the guestrepo extension as try to grok a working
solution. For now, my best solution appears to be the use of either tags
or bookmarks to note paired releases of client and server, and manually
managing nested repositories appears a workable (if cumbersome) solution
to shared libraries. (I'm convinced that shared libraries will work
better with named branches, but I haven't had the time to experiment
enough to work out how that would look.)

I think that subrepository and guestrepo extensions might solve the
libraries problem, but make no mistake, they add a layer of complexity
that should be well understood before enabling them -- Caveat Emptor!
Since the current recommended practice for subrepository linking is via
a shell repo, managing these as separate, nested repos appears to work
in line with this until I figure it out.

As you can imagine, I'd be the first to welcome a more detailed write up
of how-to set this all up, but I think the first problem, and the
problem that appears to plague subrepositories in general, is that we
don't have a clear definition on what it should do or should behave.
Being lost is so much more problematic when you don't know where you're
going to begin with.