#1232 closed defect (fixed)

Unicode stdout/stderr replacement on Windows fails to print large strings

Reported by: davidsarah Owned by: davidsarah
Priority: major Milestone: 1.8.1
Component: code Version: 1.8.0
Keywords: unicode regression easy news-done Cc:
Launchpad Bug:

Description

While trying to debug #1045, I tried to print a ResponseCache object and got the following exception:

  File "d:\tahoe\tahoe-1.8.0c2\src\allmydata\windows\fixups.py", line 134, in write
    raise IOError("WriteConsoleW returned %r, n.value = %r" % (retval, n.value))
exceptions.IOError: WriteConsoleW returned 0, n.value = 0L

This is a bug in the Unicode stdout/stderr replacement in src/allmydata/windows/fixups.py that was added in Tahoe-LAFS v1.8 beta. On my machine it fails when trying to print more than 26608 characters at once to stdout or stderr. This appears to be an undocumented limitation of the Windows WriteConsoleW API call.

Technically this is a regression since v1.7.1, although we very rarely print a string that large. The fix is trivial; just limit the length of data passed in a single call to WriteConsoleW.

Attachments (1)

workaround-windows-writeconsole-suckage.darcs.patch (5.3 KB) - added by davidsarah at 2010-10-27T02:42:39Z.
windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.

Download all attachments as: .zip

Change History (9)

comment:1 Changed at 2010-10-27T02:36:12Z by davidsarah

Correction: the MSDN documentation for WriteConsoleW does document a length limitation, but it says 64 KiB (which would be 32767 UTF-16 characters and a terminating NUL), not 26608 characters. The fix is the same regardless; I'll limit it to 10000 [UTF-16] characters.

Last edited at 2010-10-27T02:46:07Z by davidsarah (previous) (diff)

Changed at 2010-10-27T02:42:39Z by davidsarah

windows/fixups.py: limit length of string passed in a single call to WriteConsoleW. fixes #1232.

comment:2 Changed at 2010-10-27T02:43:59Z by davidsarah

  • Keywords review-needed added

comment:3 Changed at 2010-10-29T09:13:19Z by warner

  • Keywords review-needed removed
  • Owner changed from somebody to davidsarah

looks good to me.

comment:4 Changed at 2010-10-29T19:43:09Z by david-sarah@…

  • Resolution set to fixed
  • Status changed from new to closed

In 25d8103dde95d784:

(The changeset message doesn't reference this ticket)

comment:5 Changed at 2010-10-29T19:43:14Z by david-sarah@…

In 2610f8e0aa6e2221:

NEWS: clarify (strengthen) description of what backdoors.rst declares, and add bugfix entries for 'tahoe cp' and Windows console bugs. refs #1216, #1224, #1232

comment:6 Changed at 2010-10-29T19:52:14Z by davidsarah

  • Keywords news-done added

comment:7 Changed at 2011-01-09T08:47:29Z by davidsarah

Thread about this issue that seems to suggest the limit varies between machines:

http://www.mail-archive.com/log4net-dev@logging.apache.org/msg00661.html

I have not seen any reports of it failing with < 10000 characters, so our current code should work. However, I'm attempting to file the bug with Microsoft's bug reporting system, such as it is.

comment:8 Changed at 2011-03-27T14:13:00Z by davidsarah

It is possible for calls to the console functions by multiple threads to cause this error even when the amount written in each call is limited to 10000 characters (20000 bytes). However Tahoe-LAFS is single-threaded (almost; there are a few uses of deferToThread, but I don't think we print from those threads), so this shouldn't affect us. I'm pointing it out here because I know this ticket is referenced from tickets in other projects.

# Warning: this test may DoS your system.

from threading import Thread
import sys
from ctypes import WINFUNCTYPE, windll, POINTER, byref
from ctypes.wintypes import BOOL, HANDLE, DWORD, LPVOID, LPCVOID

GetStdHandle = WINFUNCTYPE(HANDLE, DWORD)(("GetStdHandle", windll.kernel32))
WriteFile = WINFUNCTYPE(BOOL, HANDLE, LPCVOID, DWORD, POINTER(DWORD), LPVOID) \
                        (("WriteFile", windll.kernel32))
GetLastError = WINFUNCTYPE(DWORD)(("GetLastError", windll.kernel32))
STD_OUTPUT_HANDLE = DWORD(-11)
INVALID_HANDLE_VALUE = DWORD(-1).value

hStdout = GetStdHandle(STD_OUTPUT_HANDLE)
assert hStdout is not None and hStdout != INVALID_HANDLE_VALUE

LENGTH = 20000 #bytes
THREADS = 10

data = b'\x08'*LENGTH

def run():
    n = DWORD(0)
    while True:
        ret = WriteFile(hStdout, data, LENGTH, byref(n), None)
        if ret == 0 or n.value != LENGTH:
            print("WriteFile returned %d, bytes written = %d, last error = %d"
                  % (ret, n.value, GetLastError()))
            sys.exit(1)

for i in range(THREADS):
    Thread(target=run).start()
Note: See TracTickets for help on using tickets.