Erratic and slow hgweb performance compared to a windows network share

[Adrian Buehlmann]

On 2012-07-10 12:10, Angel Ezquerra Moreu wrote:
> On Tue, Jul 10, 2012 at 9:21 AM, Adrian Buehlmann <adrian at cadifra.com> wrote:
>> On 2012-07-09 22:35, Angel Ezquerra wrote:
>>> On Mon, Jul 9, 2012 at 8:17 PM, Bryan O'Sullivan <bos at serpentine.com> wrote:
>>>> On Fri, Jul 6, 2012 at 2:32 PM, Angel Ezquerra <angel.ezquerra at gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> So mod_wsgi.so was compiled for python 2.6.2. but we are using 2.6.5.
>>>>> Other than that I don't see anything weird. However I am no expert on
>>>>> any of this so I could be missing something trivial of course.
>>>>
>>>>
>>>> Take a look at the mod_wsgi docs:
>>>> http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
>>>>
>>>> I'm pretty sure you need to set a number of threads or processes to use, as
>>>> otherwise they're likely to default to 1.
>>>
>>> Thank you Brian. I had read that document before and I thought that
>>> the default configuration already uses multiple threads.
>>>
>>> If I understand that document correctly the only available 'MPM' on
>>> windows is 'winnt'. According to the document "To emulate the same
>>> process/thread model as the 'winnt' MPM, that is, a single process
>>> with multiple threads, the following configuration would be used:
>>>
>>> WSGIDaemonProcess example threads=25"
>>>
>>> So I thought that on winnt we would already be using 25 threads by
>>> default. I'll see if setting it explicitly improves things though.
>>>
>>> Am I the only one running an apache based hgweb server on windows? I'm
>>> a bit surprised that other people have not experienced this issue.
>>
>> I wouldn't be that surprised. I guess some people are using IIS or if
>> they are interested in Apache run it on Linux anyway. The barrier to
>> switch to using Linux gets very low pretty quickly, given all the
>> combined quirks you most likely run into with trying to run Apache on
>> Windows. There's nothing wrong with having a Linux server and people
>> using Windows desktops. In fact, it even spares you a Windows server
>> license. IMHO, Linux on servers is way more common than on desktops.
>>
>> I don't run Apache here either, so you most likely have more experience
>> than I have.
>>
>> What strikes me as odd, is that you have such a powerful 64-bit server
>> (16 GB RAM, IIRC) but you still use 32-bit Apache and thus 32-bit Python
>> and 32-bit Mercurial.
>>
>> I see that ASF (through their official Apache website) provides only
>> 32-bit Windows binaries for Apache. It seems others like
>> http://www.apachelounge.com/download/win64/ are jumping in to help, but
>> then you still have a problem with getting 64 bit mod_wsgi.
> 
> That is exactly the reason why we are using a 32 bit Apache on a 64
> bit windows box.
> 
>> But I'm not
>> really sure whether mod_wsgi is really such a good idea to use.I
>> wouldn't be surprised if wsgi requires more tuning than plain simple
>> stupid cgi. apachelounge seems to be providing fcgi 64-bit modules for
>> Apache, so you might have more luck using plain simple cgi. It might
>> even be interesting to see if there is a difference by using cgi vs wsgi
>> with your existing 32-bit Apache.
> 
> I never considered using plain old CGI. Everything I read when I
> looked into this indicated that WSGI was the way to go.
> 
> When using CGI, is an hg process created for every server request?

I think so, yes. But it might be interesting to see how bad that really
is, compared to the current problems you have.

But it would certainly be sort of sidestepping the problem you have.

> Could it improve the "concurrent access" behavior?

I suspect that, yes. Why don't you give it a try?

If you want to see how much "better" wsgi really is compared to cgi,
then giving cgi a shot shouldn't be that much of a problem.

I think using cgi would also allow to combine your 32-bit Apache with a
64-bit Mercurial. After all, it runs in a separate process anyway.

>> It seems the projects haven't yet caught up with the reality of 64-bit
>> now being ubiquitous.
> 
> Very true. There are plenty of very popular python modules that do not
> have 64 versions (or at least 64 bit windows installers) yet.
> 
>> I'm not saying your problems will go away by using 64-bit Apache instead
>> of 32-bit. But a 32-bit process has a very hard memory limit due to the
>> limit imposed by pointer sizes (4 GB). If you have only one process with
>> lots of threads, these threads are still limited into the same process
>> space regarding memory.
> 
> The thing is it feels as if the problem is more a certain lack of
> "multi-threading" for lack of a better way to put. Isn't it weird that
> a big pull operation blocks (or at least considerably slows down) any
> access to the web server itself?

It probably is, but you are most likely on your own to find out why.