[patch] Make streaming from stdin / to stdout work.

[ericpollmann]

The documented feature of streaming a file in via stdin/stdout does not work.

$ cat foo.tar.gz | zfec -m5 -k3 - -ptest -v
Traceback (most recent call last):

File "/usr/local/bin/zfec", line 8, in <module>

load_entry_point('zfec==1.4.22', 'console_scripts', 'zfec')()

File "/Library/Python?/2.6/site-packages/zfec-1.4.22-py2.6-macosx-10.6-universal.egg/zfec/cmdline_zfec.py", line 60, in main

args.inputfile.seek(0, 2)

IOError: [Errno 29] Illegal seek

I fixed this exception and a couple of other issues (compute pad for each chunk after the stream is fully read) as well as updates for the unit tests to verify the behavior is correct. Finally, in verbose mode, this patch calls sys.stdin.flush() to make the status updates appear immediately rather than at the end.

It includes similar changes for zunfec to make streaming to stdout work as well, including moving verbose status messages from stdout to stderr so they do not corrupt the output file.

Now it works well:
$ cat foo.tar.gz | zfec -m5 -k3 - -ptest -v
Creating share file './test.0_5.fec'...
Creating share file './test.1_5.fec'...
Creating share file './test.2_5.fec'...
Creating share file './test.3_5.fec'...
Creating share file './test.4_5.fec'...
30MB ... 60MB ... 90MB ... 120MB ... 150MB ... 180MB ... 210MB ... 240MB ... 270MB ... 300MB ... 330MB ... 360MB ... 390MB ... 420MB ... 450MB ... 480MB ... 510MB ... 540MB ... 570MB ... 600MB ... 630MB ... 660MB ... 690MB ... 720MB ... 750MB ... 780MB ... 810MB ... 840MB ...
Done!
$ rm ./test.{0,4}_5.fec
$ zunfec -o - test.* -v > foo.tar.gz.new
30MB ... 60MB ... 90MB ... 120MB ... 150MB ... 180MB ... 210MB ... 240MB ... 270MB ... 300MB ... 330MB ... 360MB ... 390MB ... 420MB ... 450MB ... 480MB ... 510MB ... 540MB ... 570MB ... 600MB ... 630MB ... 660MB ... 690MB ... 720MB ... 750MB ... 780MB ... 810MB ... 840MB ...
Done!
$ diff foo.tar.gz foo.tar.gz.new
$

[ericpollmann]

Patch to fix stdin/stdout streaming issues in zfec / zunfec.

[davidsarah]

The patch seems to have been lost, perhaps due to a trac upgrade. ericpollmann, do you still have it?

[davidsarah]

ericpollmann wrote:

Here it is, not sure email is the way to re-upload, but I'm giving it a shot since I could not find a way to modify a ticket with the web interface. Also uploading an updated patch against 1.4.24

[davidsarah]

Make streaming from stdin / to stdout work - patch against 1.4.22

[davidsarah]

Make streaming from stdin / to stdout work - patch against 1.4.24

[davidsarah]

I reviewed zfec-1.4.24-streaming-new.patch​ (without testing it) and it looks good, including the tests.

Review question:

• Is sys.stdin.name documented to be "<stdin>", or is that a CPython standard library implementation detail? Perhaps use args.inputfile.fileno() == 0 (or args.inputfile.fileno() == sys.stdin.fileno() which should be equivalent)?

[davidsarah]

On 02/03/15 19:21, Eric Pollmann wrote:

Thanks for the improvement! I just simplified the patches to use args.inputfile == '-'
which implements exactly what the documentation says.

That has a type error; args.inputfile is a file object, not a string.

Now that I think about it, the behaviour should be the same for all non-seekable files, which includes devices. A non-seekable file will give an IOError with errno.ESPIPE (on Unix) or errno.EBADF (on Windows) from the attempt to seek.