[PATCH V2] procutil: ensure that all stdio file objects are flushed at interpreter exit

Manuel Jacob me at manueljacob.de
Fri Jul 10 21:27:39 UTC 2020


On 2020-07-10 15:16, Yuya Nishihara wrote:
> On Fri, 10 Jul 2020 13:55:50 +0200, Manuel Jacob wrote:
>> On 2020-07-10 13:19, Yuya Nishihara wrote:
>> > On Fri, 10 Jul 2020 07:11:04 +0200, Manuel Jacob wrote:
>> >> # HG changeset patch
>> >> # User Manuel Jacob <me at manueljacob.de>
>> >> # Date 1594346556 -7200
>> >> #      Fri Jul 10 04:02:36 2020 +0200
>> >> # Node ID 284a80ed05e7f4724062c9eeac95933a86257f38
>> >> # Parent  034feda7f6afcb0ae973569207a8268487793962
>> >> # EXP-Topic stdio
>> >> procutil: ensure that all stdio file objects are flushed at
>> >> interpreter exit
>> >>
>> >> Changeset 8403cc54bc83 introduced code that opens a second file object
>> >> referring to the stderr file descriptor. This broke tests on Windows.
>> >> The
>> >> reason is that on Windows, sys.stderr is buffered and procutil.stderr
>> >> closed
>> >> the file descriptor when it got garbage collected before sys.stderr
>> >> had the
>> >> chance to flush buffered data.
>> >>
>> >> Possibly, stdout had the same problem for a long time, but we didn’t
>> >> realize,
>> >> as in CI test runs, stdout is not a TTY and therefore no second file
>> >> object was
>> >> opened.
>> >
>> > Yep, but IIUC, the major difference is that the Python interpreter
>> > needs
>> > sys.stderr to print traceback, etc. So stderr has to remain open after
>> > the
>> > procutil module gets GC-ed. On the other hand, stdout wouldn't have
>> > such
>> > problem, and nothing should be printed after dispatch.run(), which
>> > flushes
>> > the both streams.
>> 
>> There are various scripts, especially in contrib, that use the 
>> sys.std*
>> streams.
>> 
>> > Suppose 8403cc54bc83 wouldn't make a real difference in our codebase, I
>> > suggest reverting the change.
>> 
>> There are various small tools using procutils.stderr directly that 
>> could
>> benefit from it being unbuffered (one was referred to in the changeset
>> description of 8403cc54bc83).
>> 
>> Furthermore, it would enable to remove the manual flush of stderr in 
>> the
>> ui object that works around the fact the procutil.stderr wasn’t
>> unbuffered before on Windows (and Python 3).
> 
> My feeling is that these improvements are far more trivial compared to 
> the
> complexity introduced by atexit.

I’m uncomfortable with the idea of having a block-buffered 
mercurial.utils.procutils.stderr on Python 3 and Windows. We should 
abstract the platform differences as much as possible. However, it’s 
worse to introduce regressions.

I think that we anyway need to have the complexity introduced by atexit, 
to flush stdout. But as you mentioned, sys.stderr is sometimes used by 
the Python interpreter itself to print stuff; that could even be after 
the atexit hook.

There seems to be some weak consensus that Python 2 support should be 
dropped for 5.6. Therefore, I would be fine with backing out 
8403cc54bc83 (make mercurial.utils.procutil.stderr unbuffered) for now, 
and reapply a Python 3-only solution (that doesn’t need the atexit hook) 
once Python 2 support was dropped.

What do you think? I might decide to send a patch for backing out 
8403cc54bc83 later today. Feel free to go ahead and apply it if you 
think it’s a good idea.



More information about the Mercurial-devel mailing list