[tahoe-lafs-trac-stream] [Tahoe-LAFS] #1201: 'rm' via sftp+sshfs may hang if previous operations on the file are "stuck"
Tahoe-LAFS
trac at tahoe-lafs.org
Tue Dec 2 19:44:17 UTC 2014
#1201: 'rm' via sftp+sshfs may hang if previous operations on the file are "stuck"
-------------------------------------+-------------------------------------
Reporter: bj0 | Owner: davidsarah
Type: defect | Status: assigned
Priority: major | Milestone: soon
Component: code-frontend-ftp- | Version: 1.8β
sftp | Keywords: hang sftp sshfs
Resolution: | reliability
Launchpad Bug: |
-------------------------------------+-------------------------------------
Changes (by warner):
* component: code-frontend => code-frontend-ftp-sftp
Old description:
> I was trying to figure out what part of the bonnie++ benchmark was
> failing on tahoe (via sshfs), so I mounted the sshfs interface and ran:
>
> {{{
> > bonnie -d ./mnt -s 100m -n 0 -r 50m
> Writing a byte at a time...done
> Writing intelligently...done
> Rewriting...done
> Reading a byte at a time...done
> Reading intelligently...done
> start 'em...
> ^C
> Can't read data from ITC pipe.
> Can't read from pipe, expected 32, got -1.
>
> }}}
>
> It hung for a while after "start 'em", so I checked the flog and there
> was no activity, so I hit Ctr-C, which gave me those 2 errors and then
> the interface hung.
>
> I had to kill -9 sshfs to get everything to free up, then I re-mounted
> the sshfs interface, did a:
>
> {{{
>
> > ls ./mnt
> Bonnie.3520
>
> > rm ./mnt/Bonnie.3520
>
> }}}
>
> And it hung on the 'rm'.
>
> I had flogtool running for the 'rm', and the output was (below).
>
> After restarting the tahoe client process and then reconnecting via
> sshfs, I was able to delete the file.
>
> {{{
>
> 17:48:43.537 L20 []#12940179 SUCCESS ".getAttrs('/.', followLinks=0)"
> {'permissions': 16895}
> 17:48:47.416 L20 []#12940180 <SFTPUserHandler #3>(bjp):
> .removeFile('/Bonnie.3520')
> 17:48:47.416 L10 []#12940181 <SFTPUserHandler #3>(bjp): CONVERT
> '/Bonnie.3520'
> 17:48:47.416 L10 []#12940182 <SFTPUserHandler #3>(bjp): PATH
> [u'Bonnie.3520']
> 17:48:47.416 L20 []#12940183 <SFTPUserHandler #3>(bjp):
> ._abandon_any_heisenfiles('/Bonnie.3520',
> 'URI:DIR2:unzmpoxlhyvhhhe37tenmmk5km:nfifb3ehimdhep3qbkxfimheznfgfrgc6hen5piwim3kda5zykca/Bonnie.3520')
> 17:48:47.416 L10 []#12940184 <SFTPUserHandler #3>(bjp): files =
> [<GeneralSFTPFile #7>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
> <GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
> <GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
> <GeneralSFTPFile #10>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
> <GeneralSFTPFile #9>, <GeneralSFTPFile #10>, <GeneralSFTPFile #11>,
> <GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
> <GeneralSFTPFile #10>, <GeneralSFTPFile #11>, <GeneralSFTPFile #7>,
> <GeneralSFTPFile #8>, <GeneralSFTPFile #9>, <GeneralSFTPFile #10>,
> <GeneralSFTPFile #11>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
> <GeneralSFTPFile #9>, <GeneralSFTPFile #10>, <GeneralSFTPFile #11>,
> <GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
> <GeneralSFTPFile #10>, <GeneralSFTPFile #11>, <GeneralSFTPFile #7>,
> <GeneralSFTPFile #8>, <GeneralSFTPFile #9>, <GeneralSFTPFile #10>,
> <GeneralSFTPFile #11>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
> <GeneralSFTPFile #9>, <GeneralSFTPFile #10>, <GeneralSFTPFile #11>] in
> "._abandon_any_heisenfiles('/Bonnie.3520',
> 'URI:DIR2:unzmpoxlhyvhhhe37tenmmk5km:nfifb3ehimdhep3qbkxfimheznfgfrgc6hen5piwim3kda5zykca/Bonnie.3520')"
> 17:48:47.416 L20 []#12940185 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940186 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940187 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940188 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940189 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940190 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940191 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940192 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940193 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940194 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940195 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.416 L20 []#12940196 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940197 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940198 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940199 <GeneralSFTPFile #11>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940200 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940201 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940202 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940203 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940204 <GeneralSFTPFile #11>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940205 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940206 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940207 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940208 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940209 <GeneralSFTPFile #11>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940210 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940211 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940212 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940213 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940214 <GeneralSFTPFile #11>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940215 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940216 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940217 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940218 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940219 <GeneralSFTPFile #11>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940220 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940221 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940222 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940223 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940224 <GeneralSFTPFile #11>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940225 <GeneralSFTPFile #7>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940226 <GeneralSFTPFile #8>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940227 <GeneralSFTPFile #9>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940228 <GeneralSFTPFile #10>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940229 <GeneralSFTPFile #11>(/Bonnie.3520):
> .abandon()
> 17:48:47.417 L20 []#12940230 <GeneralSFTPFile #7>(/Bonnie.3520): .sync()
>
> }}}
New description:
I was trying to figure out what part of the bonnie++ benchmark was failing
on tahoe (via sshfs), so I mounted the sshfs interface and ran:
{{{
> bonnie -d ./mnt -s 100m -n 0 -r 50m
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...
^C
Can't read data from ITC pipe.
Can't read from pipe, expected 32, got -1.
}}}
It hung for a while after "start 'em", so I checked the flog and there was
no activity, so I hit Ctr-C, which gave me those 2 errors and then the
interface hung.
I had to kill -9 sshfs to get everything to free up, then I re-mounted the
sshfs interface, did a:
{{{
> ls ./mnt
Bonnie.3520
> rm ./mnt/Bonnie.3520
}}}
And it hung on the 'rm'.
I had flogtool running for the 'rm', and the output was (below).
After restarting the tahoe client process and then reconnecting via sshfs,
I was able to delete the file.
{{{
17:48:43.537 L20 []#12940179 SUCCESS ".getAttrs('/.', followLinks=0)"
{'permissions': 16895}
17:48:47.416 L20 []#12940180 <SFTPUserHandler #3>(bjp):
.removeFile('/Bonnie.3520')
17:48:47.416 L10 []#12940181 <SFTPUserHandler #3>(bjp): CONVERT
'/Bonnie.3520'
17:48:47.416 L10 []#12940182 <SFTPUserHandler #3>(bjp): PATH
[u'Bonnie.3520']
17:48:47.416 L20 []#12940183 <SFTPUserHandler #3>(bjp):
._abandon_any_heisenfiles('/Bonnie.3520',
'URI:DIR2:unzmpoxlhyvhhhe37tenmmk5km:nfifb3ehimdhep3qbkxfimheznfgfrgc6hen5piwim3kda5zykca/Bonnie.3520')
17:48:47.416 L10 []#12940184 <SFTPUserHandler #3>(bjp): files =
[<GeneralSFTPFile #7>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
<GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
<GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
<GeneralSFTPFile #10>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
<GeneralSFTPFile #9>, <GeneralSFTPFile #10>, <GeneralSFTPFile #11>,
<GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
<GeneralSFTPFile #10>, <GeneralSFTPFile #11>, <GeneralSFTPFile #7>,
<GeneralSFTPFile #8>, <GeneralSFTPFile #9>, <GeneralSFTPFile #10>,
<GeneralSFTPFile #11>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
<GeneralSFTPFile #9>, <GeneralSFTPFile #10>, <GeneralSFTPFile #11>,
<GeneralSFTPFile #7>, <GeneralSFTPFile #8>, <GeneralSFTPFile #9>,
<GeneralSFTPFile #10>, <GeneralSFTPFile #11>, <GeneralSFTPFile #7>,
<GeneralSFTPFile #8>, <GeneralSFTPFile #9>, <GeneralSFTPFile #10>,
<GeneralSFTPFile #11>, <GeneralSFTPFile #7>, <GeneralSFTPFile #8>,
<GeneralSFTPFile #9>, <GeneralSFTPFile #10>, <GeneralSFTPFile #11>] in
"._abandon_any_heisenfiles('/Bonnie.3520',
'URI:DIR2:unzmpoxlhyvhhhe37tenmmk5km:nfifb3ehimdhep3qbkxfimheznfgfrgc6hen5piwim3kda5zykca/Bonnie.3520')"
17:48:47.416 L20 []#12940185 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940186 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940187 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940188 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940189 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940190 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940191 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940192 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940193 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940194 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940195 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.416 L20 []#12940196 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940197 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940198 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940199 <GeneralSFTPFile #11>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940200 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940201 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940202 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940203 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940204 <GeneralSFTPFile #11>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940205 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940206 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940207 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940208 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940209 <GeneralSFTPFile #11>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940210 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940211 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940212 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940213 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940214 <GeneralSFTPFile #11>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940215 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940216 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940217 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940218 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940219 <GeneralSFTPFile #11>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940220 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940221 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940222 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940223 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940224 <GeneralSFTPFile #11>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940225 <GeneralSFTPFile #7>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940226 <GeneralSFTPFile #8>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940227 <GeneralSFTPFile #9>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940228 <GeneralSFTPFile #10>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940229 <GeneralSFTPFile #11>(/Bonnie.3520):
.abandon()
17:48:47.417 L20 []#12940230 <GeneralSFTPFile #7>(/Bonnie.3520): .sync()
}}}
--
--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1201#comment:5>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage
More information about the tahoe-lafs-trac-stream
mailing list