[tahoe-dev] zfec: bug with m=16 k=13 parameters at certain file length

[Bodecs Bela]

I am on 64 bit.

Since my 1st post, I have found the root cause of the bug in filefec.py,
here is the problematic part in decode_from_files method:

    while True:
        chunks = [ inf.read(CHUNKSIZE) for inf in infs ]
        if [ch for ch in chunks if len(ch) != len(chunks[-1])]:
            raise CorruptedShareFilesError("Share files were corrupted
-- all share files are required to be the same length, but they weren't.")

        if len(chunks[-1]) == CHUNKSIZE:
            # Then this was a full read, so we're still in the sharefiles.
            resultdata = dec.decode(chunks, shnums, padlen=0)
            outf.write(resultdata)
            byteswritten += len(resultdata)
            if verbose:
                if ((byteswritten - len(resultdata)) /
(10*MILLION_BYTES)) != (byteswritten / (10*MILLION_BYTES)):
                    print str(byteswritten / MILLION_BYTES) + " MB ...",
        else:
            # Then this was a short read, so we've reached the end of
the sharefiles.
            resultdata = dec.decode(chunks, shnums, padlen)
            outf.write(resultdata)
            return # Done.


In certain cases the else branch never reach if the fec chunk file size
is exactly n times CHUNKSIZE.
(CHUNKSIZE variable value is always 4096)

I have altered the script and it works great now:

    while True:
        chunks = [ inf.read(CHUNKSIZE) for inf in infs ]
        if [ch for ch in chunks if len(ch) != len(chunks[-1])]:
            raise CorruptedShareFilesError("Share files were corrupted
-- all share files are required to be the same length, but they weren't.")

        #if len(chunks[-1]) == CHUNKSIZE:
        if len(chunks[-1]) > 0:
            # Then this was a full read, so we're still in the sharefiles.
            resultdata = dec.decode(chunks, shnums, padlen=0)
            outf.write(resultdata)
            byteswritten += len(resultdata)
            if verbose:
                if ((byteswritten - len(resultdata)) /
(10*MILLION_BYTES)) != (byteswritten / (10*MILLION_BYTES)):
                    print str(byteswritten / MILLION_BYTES) + " MB ...",
        else:
            # Then this was a short read, so we've reached the end of
the sharefiles.
            #resultdata = dec.decode(chunks, shnums, padlen)
            #outf.write(resultdata)
            if padlen > 0:
                outf.truncate(byteswritten - padlen)
            return # Done.


bb


2013.09.11. 20:23 keltezéssel, jg71 írta:
> * Bodecs Bela <bodecsb at vivanet.hu> wrote:
>
>> please confirm the following bug:
>>
>> zfec command line tool, version: 1.4.24
> same here (on Slackware64-13.37 - are you on a 32bit system?)
>
> Well.... I can reproduce the bug.
>
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
>