Silly Problems: Out of Disk Space

This is a guide how to debug and solve silly problems, this time, we’ll look at running out of disk space.

I’ve refreshed FreshRSS and in the middle of the process, page died with error: Permission is denied for '/var/freshrss/data/log/freesh.log'.

I should have guessed it right away but I went down the silly path, which lead to the solution quite fast anyway.

I’ve check if something happened with permissions ls -la /var/freshrss/data/log, everything seemed fine there.

Next I’ve restarted web server and database:

# systemctl restart nginx

… worked fine.

# systemctl restart mysqld

… hanged! So that put me one step closer. Next, I’ve checked mysqld status:

# systemctl status mysqld

* mysqld.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled)
   Active: activating (start-post) since Thu 2014-09-25 15:10:34 CEST; 1s ago
 Main PID: 9943 (mysqld);         : 9944 (mysqld-post)
   CGroup: /system.slice/mysqld.service
           ├─9943 /usr/bin/mysqld --pid-file=/run/mysqld/mysqld.pid
           └─control
             ├─9944 /bin/sh /usr/bin/mysqld-post
             └─9970 sleep 1
Sep 25 15:10:35 mystic-server mysqld[9943]: 140925 15:10:35 [Note] InnoDB: 128 rollback segment(s) are active.
Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Note] InnoDB: Waiting for purge to start
Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.19-67.0 started; log sequence number 10221890
Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Note] Plugin 'FEEDBACK' is disabled.
Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Note] Recovering after a crash using mysql-bin
Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [ERROR] Error in Log_event::read_log_event(): 'read error', data_len: 256, event_type: 24
Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Note] Starting crash recovery...
Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Note] Crash recovery finished.

… and I’ve got my answer, here it is:

Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Warning] mysqld: Disk is full writing './mysql-bin.~rec~' (Errcode: 28 "No space left on device"). Waiting for someone to free space... (Expect up to 60 secs ...eeing disk space)

Sep 25 15:10:36 mystic-server mysqld[9943]: 140925 15:10:36 [Warning] mysqld: Retry in 60 secs. Message reprinted in 600 secs

Some time ago I wrote post about using Old Laptop as a File Server, and I’ve mentioned I’ve installed Arch on 4GB USB key.

On Arch, pacman stores its downloaded packages in /var/cache/pacman/pkg/ and does not remove the old or uninstalled versions automatically, therefore it is necessary to deliberately clean up that folder periodically to prevent such folder to grow indefinitely in size. I forgot about that!

The solution is simple, I’ve just cleared cache:

# pacman -Scc

Note, using -Scc is considered a bad practice, as it prevents from reinstalling a package directly from the cache folder in case of need, thus forcing to re-download it. You can read more about it on Pacman Wiki.

To avoid this problem in future, I’ve changed upgrade alias, from:

alias upme='sudo pacman -Syu'

… to:

alias upme='sudo pacman -Syu; paccache -r; paccache -ruk0'