Problem with bookmarks in version 2.5

[Angel Ezquerra]

On Wed, Feb 6, 2013 at 9:24 PM, Matt Mackall <mpm at selenic.com> wrote:
> On Wed, 2013-02-06 at 19:42 +0100, Angel Ezquerra wrote:
>> On Wed, Feb 6, 2013 at 7:14 PM, Kevin Bullock
>> <kbullock+mercurial at ringworld.org> wrote:
>> > On Feb 6, 2013, at 11:54 AM, Mads Kiilerich wrote:
>> >
>> >> On 02/06/2013 06:35 PM, Angel Ezquerra wrote:
>> >>> If you are using Apache on windows I would actually recommend against migrating to WSGI. In my experience the multi-user performance is much worse than when using CGI.
>> >>
>> >> Can you elaborate on that?
>> >>
>> >> Without further evidence I would assume that it was a WSGI misconfiguration and that no conclusion should be made from that experience.
>> >
>> > I'm curious as well, given the number of different moving parts that can go into a WSGI deployment (Apache, lighttpd, nginx, IIS? mod_wsgi, gunicorn, ...? etc.)
>> >
>> > pacem in terris / мир / शान्ति / ‎‫سَلاَم‬ / 平和
>> > Kevin R. Bullock
>> >
>>
>> I started a lenghty thread about this last year:
>>
>> http://selenic.com/pipermail/mercurial/2012-July/043539.html
>>
>> The performance when multiple users were trying to access our
>> Apache+WSGI windows server simultaneously was very bad. In particular
>> all access to the web interface would stall when a user was doing a
>> large pull or push operation.
>>
>> It is hard to tell where our performance problems came from, since
>> there were/are so many moving parts (windows, apache, mod_wsgi,
>> mercurial, our particular network configuration, etc). It could well
>> be due to a wsgi or Apache misconfiguration. That being said one of
>> the main suggestions was to try to use CGI rather than WSGI. We tried
>> it and the result was that the performance improved dramatically. The
>> "stalls" were basically eliminated when we moved to CGI.
>>
>> Now this is just a single data point, and the our performance issues
>> with WSGI could be due to our specific setup. However this proves that
>> under certain conditions WSGI may perform worse than CGI on Windows.
>> IMHO this means that we should not do a blanket recommendation of WSGI
>> over CGI on Windows without further proof that it really offers better
>> performance in most cases.
>
> It looks like you're stuck using the winnt mpm. That gets you one
> process with one Python interpreter and one GIL. It's important to note
> that Python threads are not quite the same as OS threads. Only one
> thread can ever be executing Python code at a time so you'll never
> really take advantage of multiple cores this way.
>
> Python is.. not so good at threading and you're feeling it here. Some
> operations are especially bad here because they can block the GIL for
> large blocks of time, thus starving the other threads. There's basically
> no attempt to fairly allocate the GIL either, so CPU-intensive threads
> will tend to monopolize it.
>
> We've improved this some since 1.9 by explicitly dropping the lock in
> some of our C code (see 70b5e25f1598, for instance) but more could be
> done. But it's really best to have multiple Python processes so you have
> multiple GILs.
>
> By switching to CGI, you now have multiple Python interpreters which
> mean you don't block waiting for one thread. On the other hand, you have
> to start a brand new instance of Python and Mercurial for each request.
> For multiple repos with wildcard configs, you end up crawling the
> filesystem on each request. For single repos, it also means you're
> reloading the changelog index on each request too, as we cache this.
>
> This document:
>
> https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
>
> ..describes a feature that lets it emulate the UNIX-style
> multiprocess/multithread model, with a setting like:
>
> WSGIDaemonProcess example processes=10 threads=10
>

Matt,

thanks for the detailed explanation. I think that explains our
performance problems.

I looked at the modewsgi wiki page you mentioned and I got the
impression that the WSGIDaemonProcess configuration that you suggest
is not supported on windows because the setting wsgi.multiprocess
cannot be True.

The text is not very clear on this point though so maybe this is not the case...
It would be great to be able to avoid spawning a new python process per request.

For now to improve the CGI performance we have stopped using wildcards
on our path definitions. Instead we have a script that every 15
minutes crawls the filesystem and generates a hgpath.config file that
contains the list of repositories that we are serving (~800!) in the
format that is expected by hgweb.

Cheers,

Angel