Trac regularly unavailable with no space left on device

[btlogy]

This unavailability has been observed already a few time before (at least since #4095 last year), but seems to happen on daily basis since a few last weeks (end of March 2025).

See this list of past ​incidents.

The error show by the Trac software seems to be explicit:

IOError: [Errno 28] No space left on device

In addition to the service being often unavailable (to create and manage this issue for instance), one could be concerned of the increased risk data of corruption.

Meejah has been made aware of this problem on IRC and will hopefully be able to at least free some disk space (maybe 18th or 19th of May).

But 2 questions remains:

• could the root cause be identified to increase the availability and reduce the risk of corruption (if any)?
• is there a backup to restore the data in case those would end being corrupted?

[btlogy]

It seems like Meejah has fixed something in regard with the disk space on Saturday 18th of May. It would be nice to know if this fix was just a one-time cleanup or a longer term solution.

Unfortunately, while the error message seems to be gone, I'm still experiencing some unavailability today (502) and so is ​Upptime!

Something is still not disrupting the services on this Linode server: any idea what could that be?

[meejah]

You must not have IRC scrollback, I said:

13:13 < meejah> i made a cron job to run "nuke-logs.sh" every day, so trac should be "as happy as it gets" now

[meejah]

The core problem here (that logs regularly fill up the disc, making logging in or editing tickets hard) is solved.

While I too have seen occasional 502's, a reload has always just worked -- so this isn't really "unavailability" from a user perspective.

Of course if anyone has insight or more debugging than "there are sometime 502's" please provide more details (in a new ticket).

[btlogy]

Thank you Meejah for the fix and the heads up here.

I've created #4178 to try to get to the bottom of those intermittent 502 replies.