Page 1 of 2
Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 7:50 am
by Gregorean
Hello, lately I have a problem that CumulusMX crashes on my Raspberry Pi 3B+.with the message:
Code: Select all
Disk Space Alert - Your root partition remaining free space is critically low. Used: 100%
<<<<<<<<< Cumulus Status >>>>>>>>
CumulusMX is running ()
CumulusMX is not listening on TCP Port: 8998 (http://192.168.1.56:8998)
ProcessID (PID) of CumulusMX is: 1445 476
CumulusMX Versie is:
System Uptime of raspberrypi is: 6 hours 57 minutes
CumulusMX is running since: 2021-10-05 02:53
Path to CumulusMX is: /home/pi/CumulusMX
Do you want to stop CumulusMX ? [Y/n]
If I then enter df -h in the terminal window I get this message
Code: Select all
Using username "root".
root@192.168.1.56's password:
Linux raspberrypi 5.10.63-v7+ #1457 SMP Tue Sep 28 11:25:31 BST 2021 armv7l
Last login: Mon Oct 4 21:35:37 2021 from 192.168.1.223
root@raspberrypi:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 29G 0 100% /
devtmpfs 430M 0 430M 0% /dev
tmpfs 462M 8.0K 462M 1% /dev/shm
tmpfs 462M 12M 450M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 462M 0 462M 0% /sys/fs/cgroup
/dev/mmcblk0p1 253M 49M 204M 20% /boot
tmpfs 93M 0 93M 0% /run/user/0
And when I enter lsblk in the terminal window I get this message:
Code: Select all
root@raspberrypi:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 29.6G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 29.3G 0 part /
root@raspberrypi:~#
A temporary solution is that if I empty /var/log folder and reboot the RPI everything works again with enough space left.
But what is the real reason that the root partition keeps filling up and CumulusMX stops ?
With kind regards Sjaak.
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 8:20 am
by water01
My guess is the MXDiag log files are in root in /var/logs. These continue to build until they are 20MB and then roll onto a new one, so it is probably filling up /var/logs with old logs.
You could run a cron job to delete old ones via a script.
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 9:57 am
by freddie
water01 wrote: ↑Tue 05 Oct 2021 8:20 am
My guess is the MXDiag log files are in root in /var/logs. These continue to build until they are 20MB and then roll onto a new one, so it is probably filling up /var/logs with old logs.
I'm not sure that is correct, as the MXdiags logs are stored in the MXdiags folder in your MX distribution - not in /var/log. Having said that, it is all the same mount point anyway. These days MX rotates its logs and only keeps the latest 10 or so, which only adds up to <250MB - which is small potatoes in a 29GB partition. It's going to be something else on your Pi taking up the disk space.
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 10:25 am
by Gregorean
Hello Freddy, you are right, the MXdiags directory fills up with 26.5 GB
of logs, it helps then temporarily to empty the /var/logs directory but that is only for a while.
How can I limit the number of logs and size in MXdiags?
Sjaak.
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 10:30 am
by freddie
Gregorean wrote: ↑Tue 05 Oct 2021 10:25 am
How can I limit the number of logs and size in MXdiags?
Hi Sjaak, MX does this for you - unless you're running a very old build. What version are you running?
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 10:37 am
by Gregorean
Freddie, i run Cumulus MX release 3.13.3 (build 3148)
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 10:53 am
by freddie
Gregorean wrote: ↑Tue 05 Oct 2021 10:37 am
i run Cumulus MX release 3.13.3 (build 3148)
Then MX will manage the log files for you. I have data logging and debug logging permanently switched on and my realtime interval is two seconds and MX doesn't let my logfile volume get to more than 250MB. Each logfile is truncated at 20MB and a new one is started, with MX only keeping the 15 recent logfiles. So if you are running 3148 then you won't get a logfile problem from MX, I don't think.
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 11:16 am
by Gregorean
Freddie, I sometimes have logs of 48 MB and sometimes more than 307MB in MXdiags.
see screenshot
mxdiags.JPG
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 11:25 am
by freddie
Oh wow! I don't think MX is meant to do that! I will leave it to @mcrossley to comment further.
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 11:34 am
by water01
Have you got all of the CMX loggers enabled i.e. Program Settings > Logging Options as I believe (but I could be wrong) that the rollover options and maximum files are not used if a logging is set up as you need to preserve the data gathered by the logging.
Switch of all the logging if you haven't got a problem and MX should work as per Freddie's description, mine certainly does!!
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 11:51 am
by Gregorean
Freddie, I think I already found the problem, I just looked in the MXdiags last log and it said this:
Code: Select all
2021-10-05 03:00:23,645 Realtime[29]: Additional realtime web file[8] not found - /home/pi/CumulusMX/web/recordsMXT.js
In addition to CumulusMX, I am also running MXUI Template v 2.2.3, which uploads /home/pi/CumulusMX/web/
recordsMXT.js to my web root.
The file name
recordsMXT.js has been changed to
records-saMXT.js since the last version, which I had not changed in Extra webfiles.
That's why MXdiags is full, I'm going to change that right now and that should solve the problem.
Strange though that CumulusMX doesn't limit it to the 250MB you described.
Many thanks for your help.
Sjaak
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 12:55 pm
by water01
Gregorean wrote: ↑Tue 05 Oct 2021 11:51 am
Strange though that CumulusMX doesn't limit it to the 250MB you described.
Sjaak
What about my reply that might explain this?
Re: Your root partition remaining free space is critically low
Posted: Tue 05 Oct 2021 1:27 pm
by Gregorean
David, I had already disabled all log options, that's the first thing I looked at.
After the change in Extra webfiles CumulusMX now works without problems, the MXdiags folder remains empty which was not the case before, after half an hour the MXdiags was already full.
Re: Your root partition remaining free space is critically low
Posted: Fri 08 Oct 2021 4:06 pm
by mcrossley
water01 wrote: ↑Tue 05 Oct 2021 12:55 pm
Gregorean wrote: ↑Tue 05 Oct 2021 11:51 am
Strange though that CumulusMX doesn't limit it to the 250MB you described.
Sjaak
What about my reply that might explain this?
MX checks the size of the current log file every 10 minutes, if it is larger than 20 MB then it is closed, a new log file opened and any logfiles that exceed the allowed count of 12 are deleted.
Your log file sizes would indicate something is going seriously wrong with the running of MX, have you looked at them to see what they contain?
Re: Your root partition remaining free space is critically low
Posted: Fri 08 Oct 2021 5:05 pm
by Gregorean
Mark, I thought the problem was over but MXdiags keeps filling up with logs, there is nothing wrong in my setup but it gives all uploads to my webroot.
In no time I reach 26Gb of logs and the whole thing crashes.
I made a cronjob as you suggested which clears the MXdiags folder.
I made a script ''cleanupMXdiags.sh'' that is executed by the cronjob every half hour.
cleanupMXdiags.sh
Code: Select all
#!/bin/sh
rm /home/pi/CumulusMX/MXdiags/202*.txt
And the cronjob is:
Code: Select all
00 * * * * sudo /home/pi/CumulusMX/cleanup.sh
I will attach the MXdiags of a half hour before the cronjob erased it, maybe you see something wrong with it.
Sjaak.
20211008-171303.zip