[tahoe-lafs-trac-stream] [tahoe-lafs] #1395: error doing a check --verify on files bigger than 1Gb on win32
tahoe-lafs
trac at tahoe-lafs.org
Thu May 5 13:13:45 PDT 2011
#1395: error doing a check --verify on files bigger than 1Gb on win32
--------------------------+-----------------------
Reporter: sickness | Owner: nobody
Type: defect | Status: new
Priority: minor | Milestone: undecided
Component: unknown | Version: 1.8.2
Resolution: | Keywords:
Launchpad Bug: |
--------------------------+-----------------------
Description changed by sickness:
Old description:
> on a grid with 10 storage nodes, all on win32, and a gateway/client on
> win32, with all the nodes up and all the shares ok, I can download all
> the files ok, and check all of them ok.
> only thing that doesn't work is check --verify of files bigger than 1Gb
>
> here's the error log of the gateway, and after it of a storage node:
>
> ######################################################################
>
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] [Failure instance:
> Traceback: <type 'exceptions.IOError'>: [Errno 22] invalid mode ('wb') or
> filename: u'C:\\tahoeclient\\logs\\incidents\\incident-2011-05-05--18:33
> :34Z-w2qn32q.flog'
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\defer.py:106:maybeDeferred
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\src\allmydata\web\common.py:302:renderHTTP
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\src\allmydata\web\root.py:124:render_POST
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\support\Lib\site-
> packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\log.py:210:msg
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] --- <exception
> caught here> ---
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\support\Lib\site-
> packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\log.py:265:add_event
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\support\Lib\site-
> packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\incident.py:35:event
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\support\Lib\site-
> packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\log.py:278:declare_incident
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
> lafs\support\Lib\site-
> packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\incident.py:85:incident_declared
> 2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] ]
> 2011-05-05 20:42:33+0200 [HTTPChannel,43,10.100.9.71] Unhandled Error
> Traceback (most recent call last):
> File "c:\tahoe-lafs\support\Lib\site-
> packages\nevow-0.10.0-py2.6.egg\nevow\rend.py", line 569, in _renderHTTP
> return self.flattenFactory(doc, ctx, writer, finisher)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\nevow-0.10.0-py2.6.egg\nevow\rend.py", line 528, in <lambda>
> flattenFactory = lambda self, *args:
> flat.flattenFactory(*args)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\nevow-0.10.0-py2.6.egg\nevow\flat\__init__.py", line 14, in
> flattenFactory
> return deferflatten(stan, ctx, writer).addCallback(finisher)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\nevow-0.10.0-py2.6.egg\nevow\flat\twist.py", line 63, in
> deferflatten
> _drive(iterable, finished)
> --- <exception caught here> ---
> File "c:\tahoe-lafs\support\Lib\site-
> packages\nevow-0.10.0-py2.6.egg\nevow\flat\twist.py", line 24, in _drive
> next = iterable.next()
> File "c:\tahoe-lafs\support\Lib\site-
> packages\nevow-0.10.0-py2.6.egg\nevow\flat\ten.py", line 85, in
> iterflatten
> straccum.append(item)
> exceptions.MemoryError:
>
> ######################################################################
>
> now for the storage log:
>
> ######################################################################
>
> 2011-05-05 20:29:01+0200 [Negotiation,8,10.100.26.102] Unhandled Error
> Traceback (most recent call last):
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\log.py", line 84,
> in callWithLogger
> return callWithContext({"system": lp}, func, *args, **kw)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\log.py", line 69,
> in callWithContext
> return context.call({ILogContext: newCtx}, func, *args, **kw)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\context.py", line
> 59, in callWithContext
> return self.currentContext().callWithContext(ctx, func,
> *args, **kw)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\context.py", line
> 37, in callWithContext
> return func(*args,**kw)
> --- <exception caught here> ---
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\selectreactor.py",
> line 146, in _doReadOrWrite
> why = getattr(selectable, method)()
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\tcp.py", line
> 180, in doWrite
> return Connection.doWrite(self)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\tcp.py", line
> 431, in doWrite
> result = abstract.FileDescriptor.doWrite(self)
> File "c:\tahoe-lafs\support\Lib\site-
> packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\abstract.py",
> line 101, in doWrite
> self.dataBuffer = buffer(self.dataBuffer, self.offset) +
> "".join(self._tempDataBuffer)
> exceptions.MemoryError:
>
> the gateway/client process quickly grow over 1,5Gbyte in ram, then it
> shrunk to 180mb, the storage process grow to 300Mb and stayed there
New description:
on a grid with 10 storage nodes, all on win32, and a gateway/client on
win32, with all the nodes up and all the shares ok, I can download all the
files ok, and check all of them ok.
only thing that doesn't work is check --verify of files bigger than 1Gb
here's the error log of the gateway, and after it of a storage node:
######################################################################
{{{
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] [Failure instance:
Traceback: <type 'exceptions.IOError'>: [Errno 22] invalid mode ('wb') or
filename: u'C:\\tahoeclient\\logs\\incidents\\incident-2011-05-05--18:33
:34Z-w2qn32q.flog'
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\defer.py:106:maybeDeferred
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\src\allmydata\web\common.py:302:renderHTTP
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\src\allmydata\web\root.py:124:render_POST
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\support\Lib\site-
packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\log.py:210:msg
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] --- <exception
caught here> ---
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\support\Lib\site-
packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\log.py:265:add_event
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\support\Lib\site-
packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\incident.py:35:event
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\support\Lib\site-
packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\log.py:278:declare_incident
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] c:\tahoe-
lafs\support\Lib\site-
packages\foolscap-0.6.1-py2.6.egg\foolscap\logging\incident.py:85:incident_declared
2011-05-05 20:33:34+0200 [HTTPChannel,42,10.100.9.71] ]
2011-05-05 20:42:33+0200 [HTTPChannel,43,10.100.9.71] Unhandled Error
Traceback (most recent call last):
File "c:\tahoe-lafs\support\Lib\site-
packages\nevow-0.10.0-py2.6.egg\nevow\rend.py", line 569, in _renderHTTP
return self.flattenFactory(doc, ctx, writer, finisher)
File "c:\tahoe-lafs\support\Lib\site-
packages\nevow-0.10.0-py2.6.egg\nevow\rend.py", line 528, in <lambda>
flattenFactory = lambda self, *args:
flat.flattenFactory(*args)
File "c:\tahoe-lafs\support\Lib\site-
packages\nevow-0.10.0-py2.6.egg\nevow\flat\__init__.py", line 14, in
flattenFactory
return deferflatten(stan, ctx, writer).addCallback(finisher)
File "c:\tahoe-lafs\support\Lib\site-
packages\nevow-0.10.0-py2.6.egg\nevow\flat\twist.py", line 63, in
deferflatten
_drive(iterable, finished)
--- <exception caught here> ---
File "c:\tahoe-lafs\support\Lib\site-
packages\nevow-0.10.0-py2.6.egg\nevow\flat\twist.py", line 24, in _drive
next = iterable.next()
File "c:\tahoe-lafs\support\Lib\site-
packages\nevow-0.10.0-py2.6.egg\nevow\flat\ten.py", line 85, in
iterflatten
straccum.append(item)
exceptions.MemoryError:
}}}
######################################################################
now for the storage log:
######################################################################
{{{
2011-05-05 20:29:01+0200 [Negotiation,8,10.100.26.102] Unhandled Error
Traceback (most recent call last):
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\log.py", line 84, in
callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\log.py", line 69, in
callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\context.py", line
59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args,
**kw)
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\python\context.py", line
37, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\selectreactor.py",
line 146, in _doReadOrWrite
why = getattr(selectable, method)()
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\tcp.py", line 180,
in doWrite
return Connection.doWrite(self)
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\tcp.py", line 431,
in doWrite
result = abstract.FileDescriptor.doWrite(self)
File "c:\tahoe-lafs\support\Lib\site-
packages\twisted-8.2.0-py2.6-win32.egg\twisted\internet\abstract.py", line
101, in doWrite
self.dataBuffer = buffer(self.dataBuffer, self.offset) +
"".join(self._tempDataBuffer)
exceptions.MemoryError:
}}}
######################################################################
the gateway/client process quickly grow over 1,5Gbyte in ram, then it
shrunk to 180mb, the storage process grow to 300Mb and stayed there
--
--
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1395#comment:1>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage
More information about the tahoe-lafs-trac-stream
mailing list