SFTP and FTP: support for non-UTF-8 charsets (error message "Path could not be decoded as UTF-8")

[slush]

I open new ticket, because I didn't found any reported problem with UTF-8 (except this one #704 which is afaik something different).

When I try to create file/directory over web frontend, everything goes well. But any attempt to create directory/file with national symbols over FTP/SFTP frontend fails.

WinSCP tell me "Path could not be decoded as UTF-8", Total Commander just tell me some general error.

Example directory name which fails: žluťoučký kůň úpěl ďábelské ódy (means: Yellow horse moaned devil odes), but fails also with any other reasonable name as "dovolená" (holiday).

Reproducibility: always Python version: 2.6

[slush]

And when I create directory thru web interface, I cannot enter them using FTP client (TC tell me Internal server error).

[davidsarah]

This error will occur if the file or directory name is not valid UTF-8. Polish systems often use ISO-Latin-2 locales -- is that what your filesystem uses?

If so, the SFTP specification implies that it is the responsibility of the client to convert names to UTF-8, and apparently the clients you tried aren't doing that.

Alternatives:

• have an option in tahoe.cfg or the FTP/SFTP accounts file to specify another encoding.
• declare this to be a client bug.

The FTP frontend does not support Unicode at all; that is ticket #682. However, if #682 were fixed by implementing RFC 2640, then the FTP frontend would also only support UTF-8.

We can also improve the error message for clients that display it.

(It is a bug in Total Commander that it doesn't display the message. Also, its description of FX_FAILURE as "Internal server error" is inaccurate -- it just means an error that has no more specific code. In this case we could arguably report FX_BAD_MESSAGE, though.)

[davidsarah]

wiki:SftpFrontend#Unicodefilenames already documented this problem, but I added a reference to this ticket.

[davidsarah]

Replying to slush:

And when I create directory thru web interface, I cannot enter them using FTP client (TC tell me Internal server error).

This problem (listing a directory containing non-ASCII names in FTP) is part of #682.

[slush]

@David:

You are right, I fixed problems with UTF-8 in WinSCP by setting correct charset. I think it should be mentioned in SFTP frontend docs (because many other SFTP servers Im using works without any settings changes).

[davidsarah]

Replying to slush:

@David:

You are right, I fixed problems with UTF-8 in WinSCP by setting correct charset.

Please change the 'Unicode filenames' section of wiki:SftpFrontend to explain how to do this.

[davidsarah]

Replying to davidsarah:

Replying to slush:

You are right, I fixed problems with UTF-8 in WinSCP by setting correct charset.

Please change the 'Unicode filenames' section of wiki:SftpFrontend to explain how to do this.

Done.

[davidsarah]

I'm unconvinced that supporting non-UTF-8 encodings is worth the hassle and complexity. Does anyone want to argue in favour of it?

Note that neither SFTP nor FTP have any standard by which a specific non-UTF-8 encoding could be automatically negotiated. So, this would have to be manually configured. But clients that do a reasonable job of supporting non-ASCII characters at all, usually have an option to select UTF-8. So I think that support for other encodings would benefit very few users.

[daira]

I'm wontfixing this, because SFTP and FTP simply have no support for negotiating non-UTF-8 encodings.