<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:950629950;
mso-list-type:hybrid;
mso-list-template-ids:-1695749166 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Jean-Paul,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks for the feedback. I will try to isolate the problem.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I also have some additional information which may be useful in determining where the issue is.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When the Tahoe state machine hangs on a node, it is in one of 2 states:<o:p></o:p></p><ol style='margin-top:0in' start=1 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'>No transfer taking place. When the node is in this state, I can see that there are is no data being transferred at the TCP layer. The Tahoe transfer state machine hangs for a period of time until an upper layer timeout occurs.<o:p></o:p></li><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'>Active transfer taking place. When the node is in this state, I can see that data is being transferred from one node to another at the TCP layer. So TCP is in fact not hung. However, there must be some type of state machine in Tahoe above the TCP layer that breaks the transfer into chunks. The Tahoe transfer state machine displayed using “tahoe status” again does not progress. When Tahoe is in this state, it does not seem to time out. This state is much more problematic since the node can end up transferring many Gigabytes of data before the issue is addressed.<o:p></o:p></li></ol><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Not sure if this additional information may provide insight on where I should look for the issue.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> Bruce T<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Jean-Paul Calderone <jean-paul+tahoe-dev@leastauthority.com><br><b>Date: </b>Wednesday, February 12, 2020 at 9:35 AM<br><b>To: </b>Bruce Thompson <brucet.cisco@gmail.com><br><b>Cc: </b>"tahoe-dev@tahoe-lafs.org" <tahoe-dev@tahoe-lafs.org>, eduardo gonzalez <eduardogonzalez@ged-innovations.com>, Brian Thompson <brianbthompson@sbcglobal.net><br><b>Subject: </b>Re: Status update on Tahoe LAFS integration with Nextcloud<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>On Mon, Feb 10, 2020 at 6:53 PM brucet <<a href="mailto:brucet.cisco@gmail.com">brucet.cisco@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Jean-Paul et al,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I just wanted to give you a status update on the progress I have made on creating a Nextcloud filesystem plugin for Tahoe-LAFS.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The good news is that the plugin is fully functional and seems to work quite well with Tahoe LAFS. The plugin maps a either Tahoe URI or a Tahoe alias / path to a Nextcloud mount point. The plugin also detects Tahoe magic-folder configurations and maps the resulting Tahoe magic-folder directory to a Nextcloud mount point. I have tested the plugin using both the Nextcloud web interface and using Nextcloud clients (wabdav). Both interfaces seem to work reliably (with a caveat).<o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Heya Bruce,<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>That's quite cool.<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The bad news is that I am having intermittent trouble with Tahoe mutable files which I believe are used to hold file directories in the Tahoe filesystem. I have multiple Tahoe groups running right now each with its own introducer. Each Tahoe group has about 8 nodes in it. The nodes which are members of the Tahoe groups are always behind a NAT. I use the frp package (<a href="https://github.com/fatedier/frp" target="_blank">https://github.com/fatedier/frp</a>) to create tunnels to a server with a public IP to get around the NAT issue. The result is that bandwidth between nodes can be reasonable low (< 1Mbps) and RTT can be quite high.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What I am finding is that when I use the Tahoe filesystem heavily, the Tahoe client state machine will often hang for about 5 – 10 minutes before some type of timeout occurs and data transfers continue. Here is an example of a Tahoe status screen I see when the Tahoe client state machine is hung:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=730 height=219 style='width:7.6041in;height:2.2812in' id="gmail-m_6536735232356792865Picture_x0020_1" src="cid:image001.png@01D5E1B4.6A07B2F0" alt="cid:1703a067d214cff311"><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>All of the Storage Indexes above are mutable files which I believe hold Tahoe filesystem directories. When a client hangs up, it stays in this state for a significant period of time (5 – 10 minutes) before timing out and continuing on. I have found that once the Tahoe client is in this state, it will not operate properly until I restart the Tahoe node.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have found multiple nodes in my network which exhibit similar behavior.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I am looking at the Tahoe code right now to try to understand the details of the mutable files subsystem and what may be causing this issue. Any pointers / feedback would be greatly appreciated.<o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Hm. I have an intuition that network level timeouts are probably not handled very gracefully throughout Tahoe-LAFS but I can't put my finger on anything in particular. I don't think I have any concrete ideas about what might be going wrong here, unfortunately. It sounds like a reliability issue that would be great to resolve. I wonder if there are any simpler deployment configurations that could be made to replicate the misbehavior. My approach to tracking down the problem would probably involve setting up an environment where I can easily replicate the behavior and then heavily instrumenting the implementation with improved logging (probably using Eliot) until I had enough information available to make it clear what's happening.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Jean-Paul<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Bruce T<o:p></o:p></p></div></div></blockquote></div></div></div></body></html>