Configurable resource usage profile and repository role

[Pierre-Yves David]

I create a MR to create such option, not fully convinced with the naming 
yet, but that is a starting point.


https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/682

On 9/13/23 16:45, Mathias De Mare (Nokia) wrote:
>
>> -----Original Message-----
>> From: Pierre-Yves David <pierre-yves.david at ens-lyon.org>
>> Sent: Wednesday, September 13, 2023 12:36 PM
>> To: mercurial-devel <mercurial-devel at mercurial-scm.org>
>> Cc: Joerg Sonnenberger <joerg at bec.de>; Mathias De Mare (Nokia)
>> <mathias.de_mare at nokia.com>
>> Subject: Configurable resource usage profile and repository role
>>
>>
>> CAUTION: This is an external email. Please be very careful when clicking links or
>> opening attachments. See the URL nok.it/ext for additional information.
>>
>>
>>
>> Hello everyone,
>>
>> I would like to submit some development though and possible directions for
>> Mercurial performance-behavior in the future.
>>
>>
>> One of the key principle of Mercurial since its conception is "versatility".
>> Mercurial is built to be as suitable for small repositories, large repositories,
>> small teams, large teams, small hardware, large hardware. This approach is
>> pretty nice and have been working reasonably well so far.
>>
>> However, when it comes to performance optimization, this approach reach
>> some limitation. For example, using more memory for caches can have a big
>> impact on some operation, but could cripples them on smaller hardware. In
>> the same ways, some expensive computation are necessary for smooth server
>> operation, but would signicantly slow down operation on developer machine.
>>
>> So I think it would make sense to introduce two sets of configuration:
>>
>> - the first one for finer control about resource profiles, (that would adjust the
>> default setting for some configuration),
>>
>> - the second one to clearer declaration of the repository intended usage, (that
>> would adjust the default setting for some configuration),
> Having resource profiles and usage profiles looks like it would have a few nice benefits:
> - It would make a lot of configuration easier (I think we still have some settings behind flags, or indeed caches set to relatively low values).
> - It would help have less 'missed configuration': every so often, I notice an interesting performance-related flag that we simply did not have set.
> - It would perhaps introduce new settings to at least a subset of users sooner (by enabling them for the "high" level, for example).
>
> I'm definitely in favour of this.
>
> Both for resource profile and usage profile, I think it would be also be useful to print some kind of warning message when doing a new clone if no resource or usage profile is set.
>
> One thing that worries me a bit: do we let "low", "medium" and "high" evolve in the future? I know we have backwards-compatibility to keep in mind, so maybe we should clearly define this as something that is not set in stone (and never will be)?
>
>>
>> # About resource profile
>>
>> I can see about three areas were resource usage could be adjusted:
>> memory, cpu and storage.
>>
>> Having three levels would be a good start, "low", "medium" (the default) and
>> "high". (maybe with a fourth option that control all other at the same time)
>>
>>   From this configuration, we could adjust some of the current value (especially
>> cache size) and some behavior. For example if the storage is marked as "low"
>> and the "cpu" is marked as "high", more time can be spent optimizing the
>> storage.
>>
>> I could go into more example of what we could adjust here, but I did not want
>> this initial email to grow too large.
>>
>>
>> # About usage profile :
>>
>> The way a repository is used can change the tradeoff that works best for it. For
>> example, if you make a disposable clone for a CI runner, we really don't care
>> about optimizing the storage information received from the server, that clone
>> will be dead in a couple of minutes, on the other hand, if you are the server
>> holding the main copy of a repository, it make sense to carefully validate and
>> optimize the small content you receive from your client pushes.
> We would definitely be interested in profiles like this, especially for the two use cases you mention. Anything that makes our CI runners faster is welcome 😉
>
> Greetings,
> Mathias
>> I can think of the following role we could declare (with different level of
>> precision).
>>
>> - server
>>     - main
>>     - mirror
>> - client
>>     - developer
>>     - read-only
>>     - ci
>>       - temporary
>>       - persistent
>>
>>
>> (again, I am not going into too much details to keep this email short.
>>
>> I guess I am not the only one to think about these problems, so I am curious to
>> hear your though.
>>
>> --
>> Pierre-Yves David
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at lists.mercurial-scm.org
> https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel

-- 
Pierre-Yves David