Your root partition remaining free space is critically low

[Gregorean]

Hello, lately I have a problem that CumulusMX crashes on my Raspberry Pi 3B+.with the message:

Code: Select all

Disk Space Alert - Your root partition remaining free space is critically low. Used: 100%

<<<<<<<<< Cumulus Status >>>>>>>>

CumulusMX is running ()
CumulusMX is not listening on TCP Port: 8998 (http://192.168.1.56:8998)
ProcessID (PID) of CumulusMX is: 1445 476
CumulusMX Versie is:
System Uptime of raspberrypi is: 6 hours 57 minutes
CumulusMX is running since: 2021-10-05 02:53
Path to CumulusMX is: /home/pi/CumulusMX

Do you want to stop CumulusMX ? [Y/n]

If I then enter df -h in the terminal window I get this message

Code: Select all

Using username "root".
root@192.168.1.56's password:
Linux raspberrypi 5.10.63-v7+ #1457 SMP Tue Sep 28 11:25:31 BST 2021 armv7l
Last login: Mon Oct  4 21:35:37 2021 from 192.168.1.223
root@raspberrypi:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G   29G     0 100% /
devtmpfs        430M     0  430M   0% /dev
tmpfs           462M  8.0K  462M   1% /dev/shm
tmpfs           462M   12M  450M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   49M  204M  20% /boot
tmpfs            93M     0   93M   0% /run/user/0

And when I enter lsblk in the terminal window I get this message:

Code: Select all

root@raspberrypi:~# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 29.6G  0 disk
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0 29.3G  0 part /
root@raspberrypi:~#

A temporary solution is that if I empty /var/log folder and reboot the RPI everything works again with enough space left.
But what is the real reason that the root partition keeps filling up and CumulusMX stops ?

With kind regards Sjaak.

[water01]

My guess is the MXDiag log files are in root in /var/logs. These continue to build until they are 20MB and then roll onto a new one, so it is probably filling up /var/logs with old logs.

You could run a cron job to delete old ones via a script.

[freddie]

My guess is the MXDiag log files are in root in /var/logs. These continue to build until they are 20MB and then roll onto a new one, so it is probably filling up /var/logs with old logs.

I'm not sure that is correct, as the MXdiags logs are stored in the MXdiags folder in your MX distribution - not in /var/log. Having said that, it is all the same mount point anyway. These days MX rotates its logs and only keeps the latest 10 or so, which only adds up to <250MB - which is small potatoes in a 29GB partition. It's going to be something else on your Pi taking up the disk space.

[Gregorean]

Hello Freddy, you are right, the MXdiags directory fills up with 26.5 GB of logs, it helps then temporarily to empty the /var/logs directory but that is only for a while.
How can I limit the number of logs and size in MXdiags?

Sjaak.

[freddie]

How can I limit the number of logs and size in MXdiags?

Hi Sjaak, MX does this for you - unless you're running a very old build. What version are you running?

[Gregorean]

Freddie, i run Cumulus MX release 3.13.3 (build 3148)

[freddie]

i run Cumulus MX release 3.13.3 (build 3148)

Then MX will manage the log files for you. I have data logging and debug logging permanently switched on and my realtime interval is two seconds and MX doesn't let my logfile volume get to more than 250MB. Each logfile is truncated at 20MB and a new one is started, with MX only keeping the 15 recent logfiles. So if you are running 3148 then you won't get a logfile problem from MX, I don't think.

[Gregorean]

Freddie, I sometimes have logs of 48 MB and sometimes more than 307MB in MXdiags.
see screenshot

mxdiags.JPG

[freddie]

Oh wow! I don't think MX is meant to do that! I will leave it to @mcrossley to comment further.

[water01]

Have you got all of the CMX loggers enabled i.e. Program Settings > Logging Options as I believe (but I could be wrong) that the rollover options and maximum files are not used if a logging is set up as you need to preserve the data gathered by the logging.

Switch of all the logging if you haven't got a problem and MX should work as per Freddie's description, mine certainly does!!

[Gregorean]

Freddie, I think I already found the problem, I just looked in the MXdiags last log and it said this:

Code: Select all

2021-10-05 03:00:23,645 Realtime[29]: Additional realtime web file[8] not found - /home/pi/CumulusMX/web/recordsMXT.js 

In addition to CumulusMX, I am also running MXUI Template v 2.2.3, which uploads /home/pi/CumulusMX/web/recordsMXT.js to my web root.

The file name recordsMXT.js has been changed to records-saMXT.js since the last version, which I had not changed in Extra webfiles.

That's why MXdiags is full, I'm going to change that right now and that should solve the problem.

Strange though that CumulusMX doesn't limit it to the 250MB you described.

Many thanks for your help.

Sjaak

[water01]

Strange though that CumulusMX doesn't limit it to the 250MB you described.



Sjaak

What about my reply that might explain this?

[Gregorean]

David, I had already disabled all log options, that's the first thing I looked at.
After the change in Extra webfiles CumulusMX now works without problems, the MXdiags folder remains empty which was not the case before, after half an hour the MXdiags was already full.

[mcrossley]

Strange though that CumulusMX doesn't limit it to the 250MB you described.



Sjaak

What about my reply that might explain this?

MX checks the size of the current log file every 10 minutes, if it is larger than 20 MB then it is closed, a new log file opened and any logfiles that exceed the allowed count of 12 are deleted.

Your log file sizes would indicate something is going seriously wrong with the running of MX, have you looked at them to see what they contain?

[Gregorean]

Mark, I thought the problem was over but MXdiags keeps filling up with logs, there is nothing wrong in my setup but it gives all uploads to my webroot.
In no time I reach 26Gb of logs and the whole thing crashes.
I made a cronjob as you suggested which clears the MXdiags folder.
I made a script ''cleanupMXdiags.sh'' that is executed by the cronjob every half hour.

cleanupMXdiags.sh

Code: Select all

#!/bin/sh
rm /home/pi/CumulusMX/MXdiags/202*.txt

And the cronjob is:

Code: Select all

00 * * * * sudo /home/pi/CumulusMX/cleanup.sh 

I will attach the MXdiags of a half hour before the cronjob erased it, maybe you see something wrong with it.

Sjaak.

20211008-171303.zip