[tahoe-dev] timestamps on mutable files
Jody Harris
imhavoc at gmail.com
Mon Jan 4 10:57:05 PST 2010
IF the mutable file update problem is due to memory use problems with
downloading the file, keeping it in memory then re-uploading it, then (for
this use case) it is worth the overhead (to me) for now.
IF there are other issues that I need to be aware of, please let me know.
Question: Why was the decision made to download the file before uploading
the new version?
thanks,
jody
----
- Think carefully.
- Contra mundum - "Against the world" (St. Athanasius)
- Credo ut intelliga - "I believe that I may know" (St. Augustin of Hippo)
On Mon, Jan 4, 2010 at 11:33 AM, Peter Secor <secorp at allmydata.com> wrote:
> You are correct Kevan, the hard limit has been removed, sorry I didn't
> realize that had been fixed already!
>
> So, the limit is removed, but Jody you may want to read the ticket #694
> notes, there is a good discussion of why we originally had a hard-limit.
>
> Ps
>
> On 1/4/10 10:24 AM, Kevan Carstensen wrote:
> > I'm sure someone will correct me if I'm wrong, but I think #694 removed
> > the hard limit on mutable files. So your data should be backed up.
> >
> > (apologies for formatting - I'm on a phone)
> > --
> > Kevan Carstensen | <kevan at isnotajoke.com <mailto:kevan at isnotajoke.com>>
> >
> > On Jan 4, 2010, at 10:10 AM, Jody Harris <imhavoc at gmail.com
> > <mailto:imhavoc at gmail.com>> wrote:
> >
> >> Ack!
> >>
> >> So, I'm NOT backing up my data!? (I'm just backing up the first 2MB of
> >> each compressed table?)
> >>
> >> Okay, if immutable files are my only choice, how do I do garbage
> >> collection to insure that a minimum of dead space is taken up by
> >> "yesterday's" files?
> >>
> >> jody
> >> ----
> >> - Think carefully.
> >> - Contra mundum - "Against the world" (St. Athanasius)
> >> - Credo ut intelliga - "I believe that I may know" (St. Augustin of
> Hippo)
> >>
> >>
> >> On Mon, Jan 4, 2010 at 11:05 AM, Peter Secor <
> >> <mailto:secorp at allmydata.com>secorp at allmydata.com
> >> <mailto:secorp at allmydata.com>> wrote:
> >>
> >> Mutable files currently have a small (2MB?) limit and cannot be
> >> used for
> >> larger files. Immutable files (the default) have a much higher limit
> >> (12GB now?) and are much better suited for the purpose you
> >> described of
> >> large snapshots of data that won't change later.
> >>
> >> There are ideas behind larger mutable files (#393) but they are not
> >> implemented yet.
> >>
> >> Ps
> >>
> >> On 1/4/10 7:37 AM, Jody Harris wrote:
> >> > Oh, I forgot to answer the "how?"
> >> >
> >> > ... put --mutable ...
> >> >
> >> > jody
> >> > ----
> >> > - Think carefully.
> >> > - Contra mundum - "Against the world" (St. Athanasius)
> >> > - Credo ut intelliga - "I believe that I may know" (St. Augustin
> >> of Hippo)
> >> >
> >> >
> >> > On Sun, Jan 3, 2010 at 10:59 PM, Jody Harris <
> >> <mailto:imhavoc at gmail.com>imhavoc at gmail.com <mailto:
> imhavoc at gmail.com>
> >> > <mailto: <mailto:imhavoc at gmail.com>imhavoc at gmail.com
> >> <mailto:imhavoc at gmail.com>>> wrote:
> >> >
> >> > I have a script that:
> >> > - iterates though my MySQL tables
> >> > - dumps each table
> >> > - pipes the dump through gzip
> >> > - write the file to a temp file
> >> > - copies the file to [remote storage] (now Tahoe grid)
> >> > - deletes temp file
> >> >
> >> > This gives me a daily snapshot of all of the active databases on
> my
> >> > server every night.
> >> >
> >> > The current backup is ~ 300 MB/night, which isn't bad, but after
> one
> >> > year, that's 110+GB x 3.3 and counting of consumed tahoe space.
> >> > Unless tahoe has a better garbage collection system than I have
> come
> >> > to believe, that's going to be a problem on a 500 GB grid.
> >> >
> >> > In my reading of the tahoe docs, I have not come across any alarms
> >> > concerning mutable files. Is there something I should be made
> >> aware of?
> >> >
> >> > jody
> >> > ----
> >> > - Think carefully.
> >> > - Contra mundum - "Against the world" (St. Athanasius)
> >> > - Credo ut intelliga - "I believe that I may know" (St. Augustin
> of
> >> > Hippo)
> >> >
> >> >
> >> > On Sun, Jan 3, 2010 at 10:50 PM, Zooko O'Whielacronx
> >> > < <mailto:zookog at gmail.com>zookog at gmail.com
> >> <mailto:zookog at gmail.com> <mailto:
> >> <mailto:zookog at gmail.com>zookog at gmail.com
> >> <mailto:zookog at gmail.com>>> wrote:
> >> >
> >> > On Sun, Jan 3, 2010 at 10:33 PM, Jody Harris
> >> > < <mailto:havoc at harrisdev.com>havoc at harrisdev.com
> >> <mailto:havoc at harrisdev.com> <mailto:
> >> <mailto:havoc at harrisdev.com>havoc at harrisdev.com
> >> <mailto:havoc at harrisdev.com>>> wrote:
> >> > > I have set up my web server to store backup files on my tahoe
> >> > grid, shifting
> >> > > from the use of a server at my house. One thing that
> >> > initially alarmed me
> >> > > was that the timestamps of the backed up files is not updated
> >> > in the
> >> > > tahoe-lafs interface. I did confirm that the shares
> >> > timestamps are updated.
> >> >
> >> > Tahoe-LAFS doesn't store timestamps or other metadata on files,
> only
> >> > on links to files, which links are stored in directories. So if
> you
> >> > have a directory which contains a link named "README.txt" to
> >> > version 1
> >> > of your file, and then you upload version 2 of your file and you
> >> > change that directory to link to version 2, then Tahoe-LAFS will
> >> > update the timestamp on that link named "README.txt" when it
> updates
> >> > the link to point to the new file.
> >> >
> >> > Did you say "mutable files"? You should probably not be using
> >> > mutable
> >> > files for anything. Immutable files (plus directories) probably
> fit
> >> > all your needs for backups, and mutable files have significant
> >> > performance problems.
> >> >
> >> > How do you trigger these backups -- are you using "tahoe cp" or
> >> > "tahoe
> >> > backup" or sending HTTP requests to the WAPI or something else?
> >> >
> >> > Regards,
> >> >
> >> > Zooko
> >> > _______________________________________________
> >> > tahoe-dev mailing list
> >> > <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >> <mailto:tahoe-dev at allmydata.org> <mailto:
> >> <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >> <mailto:tahoe-dev at allmydata.org>>
> >> >
> >> <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev>
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > tahoe-dev mailing list
> >> > <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >> <mailto:tahoe-dev at allmydata.org>
> >> >
> >> <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev>
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >> _______________________________________________
> >> tahoe-dev mailing list
> >> <mailto:tahoe-dev at allmydata.org>tahoe-dev at allmydata.org
> >> <mailto:tahoe-dev at allmydata.org>
> >> <http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev>
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >>
> >>
> >> _______________________________________________
> >> tahoe-dev mailing list
> >> tahoe-dev at allmydata.org <mailto:tahoe-dev at allmydata.org>
> >> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> >
> >
> > _______________________________________________
> > tahoe-dev mailing list
> > tahoe-dev at allmydata.org
> > http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at allmydata.org
> http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://allmydata.org/pipermail/tahoe-dev/attachments/20100104/390070b8/attachment-0001.htm
More information about the tahoe-dev
mailing list