Unicode stdout/stderr replacement on Windows fails to print large strings

[davidsarah]

While trying to debug #1045, I tried to print a ResponseCache object and got the following exception:

  File "d:\tahoe\tahoe-1.8.0c2\src\allmydata\windows\fixups.py", line 134, in write
    raise IOError("WriteConsoleW returned %r, n.value = %r" % (retval, n.value))
exceptions.IOError: WriteConsoleW returned 0, n.value = 0L

This is a bug in the Unicode stdout/stderr replacement in src/allmydata/windows/fixups.py​ that was added in Tahoe-LAFS v1.8 beta. On my machine it fails when trying to print more than 26608 characters at once to stdout or stderr. This appears to be an undocumented limitation of the Windows WriteConsoleW API call.

Technically this is a regression since v1.7.1, although we very rarely print a string that large. The fix is trivial; just limit the length of data passed in a single call to WriteConsoleW.

[davidsarah]

Correction: the ​MSDN documentation for WriteConsoleW does document a length limitation, but it says 64 KiB (which would be 32767 UTF-16 characters and a terminating NUL), not 26608 characters. The fix is the same regardless; I'll limit it to 10000 [UTF-16] characters.

[davidsarah]

windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.

[warner]

looks good to me.

[david-sarah@…]

In 25d8103dde95d784:

(The changeset message doesn't reference this ticket)

[david-sarah@…]

In 2610f8e0aa6e2221:

NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232

[davidsarah]

Thread about this issue that seems to suggest the limit varies between machines:

​http://www.mail-archive.com/log4net-dev@logging.apache.org/msg00661.html

I have not seen any reports of it failing with < 10000 characters, so our current code should work. However, I'm attempting to file the bug with Microsoft's bug reporting system, such as it is.

[davidsarah]

It is possible for calls to the console functions by multiple threads to cause this error even when the amount written in each call is limited to 10000 characters (20000 bytes). However Tahoe-LAFS is single-threaded (almost; there are a few uses of deferToThread, but I don't think we print from those threads), so this shouldn't affect us. I'm pointing it out here because I know this ticket is referenced from tickets in other projects.

# Warning: this test may DoS your system.

from threading import Thread
import sys
from ctypes import WINFUNCTYPE, windll, POINTER, byref
from ctypes.wintypes import BOOL, HANDLE, DWORD, LPVOID, LPCVOID

GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)(("GetStdHandle", windll.kernel32))
WriteFile = WINFUNCTYPE(BOOL, HANDLE, LPCVOID, DWORD, POINTER(DWORD), LPVOID) \
                        (("WriteFile", windll.kernel32))
GetLastError = WINFUNCTYPE(DWORD)(("GetLastError", windll.kernel32))
STD_OUTPUT_HANDLE = DWORD(-11)
INVALID_HANDLE_VALUE = DWORD(-1).value

hStdout = GetStdHandle(STD_OUTPUT_HANDLE)
assert hStdout is not None and hStdout != INVALID_HANDLE_VALUE

LENGTH = 20000 #bytes
THREADS = 10

data = b'\x08'*LENGTH

def run():
    n = DWORD(0)
    while True:
        ret = WriteFile(hStdout, data, LENGTH, byref(n), None)
        if ret == 0 or n.value != LENGTH:
            print("WriteFile returned %d, bytes written = %d, last error = %d"
                  % (ret, n.value, GetLastError()))
            sys.exit(1)

for i in range(THREADS):
    Thread(target=run).start()