[PATCH V2] procutil: ensure that all stdio file objects are flushed at interpreter exit
Manuel Jacob
me at manueljacob.de
Fri Jul 10 21:27:39 UTC 2020
On 2020-07-10 15:16, Yuya Nishihara wrote:
> On Fri, 10 Jul 2020 13:55:50 +0200, Manuel Jacob wrote:
>> On 2020-07-10 13:19, Yuya Nishihara wrote:
>> > On Fri, 10 Jul 2020 07:11:04 +0200, Manuel Jacob wrote:
>> >> # HG changeset patch
>> >> # User Manuel Jacob <me at manueljacob.de>
>> >> # Date 1594346556 -7200
>> >> # Fri Jul 10 04:02:36 2020 +0200
>> >> # Node ID 284a80ed05e7f4724062c9eeac95933a86257f38
>> >> # Parent 034feda7f6afcb0ae973569207a8268487793962
>> >> # EXP-Topic stdio
>> >> procutil: ensure that all stdio file objects are flushed at
>> >> interpreter exit
>> >>
>> >> Changeset 8403cc54bc83 introduced code that opens a second file object
>> >> referring to the stderr file descriptor. This broke tests on Windows.
>> >> The
>> >> reason is that on Windows, sys.stderr is buffered and procutil.stderr
>> >> closed
>> >> the file descriptor when it got garbage collected before sys.stderr
>> >> had the
>> >> chance to flush buffered data.
>> >>
>> >> Possibly, stdout had the same problem for a long time, but we didn’t
>> >> realize,
>> >> as in CI test runs, stdout is not a TTY and therefore no second file
>> >> object was
>> >> opened.
>> >
>> > Yep, but IIUC, the major difference is that the Python interpreter
>> > needs
>> > sys.stderr to print traceback, etc. So stderr has to remain open after
>> > the
>> > procutil module gets GC-ed. On the other hand, stdout wouldn't have
>> > such
>> > problem, and nothing should be printed after dispatch.run(), which
>> > flushes
>> > the both streams.
>>
>> There are various scripts, especially in contrib, that use the
>> sys.std*
>> streams.
>>
>> > Suppose 8403cc54bc83 wouldn't make a real difference in our codebase, I
>> > suggest reverting the change.
>>
>> There are various small tools using procutils.stderr directly that
>> could
>> benefit from it being unbuffered (one was referred to in the changeset
>> description of 8403cc54bc83).
>>
>> Furthermore, it would enable to remove the manual flush of stderr in
>> the
>> ui object that works around the fact the procutil.stderr wasn’t
>> unbuffered before on Windows (and Python 3).
>
> My feeling is that these improvements are far more trivial compared to
> the
> complexity introduced by atexit.
I’m uncomfortable with the idea of having a block-buffered
mercurial.utils.procutils.stderr on Python 3 and Windows. We should
abstract the platform differences as much as possible. However, it’s
worse to introduce regressions.
I think that we anyway need to have the complexity introduced by atexit,
to flush stdout. But as you mentioned, sys.stderr is sometimes used by
the Python interpreter itself to print stuff; that could even be after
the atexit hook.
There seems to be some weak consensus that Python 2 support should be
dropped for 5.6. Therefore, I would be fine with backing out
8403cc54bc83 (make mercurial.utils.procutil.stderr unbuffered) for now,
and reapply a Python 3-only solution (that doesn’t need the atexit hook)
once Python 2 support was dropped.
What do you think? I might decide to send a patch for backing out
8403cc54bc83 later today. Feel free to go ahead and apply it if you
think it’s a good idea.
More information about the Mercurial-devel
mailing list