Cumulus Support

My CMX 3183 running on Debian with mono 6.12.0.122 crashes every now and then - with a long list of debugging info (see attachment).
The MXdiags file says nothing related, just ends
2022-05-16 05:21:02.394 Reading live data
2022-05-16 05:21:02.417 DoCommand(CMD_GW1000_LIVEDATA): Valid response
2022-05-16 05:21:02.417 Received: FF-FF-27-00-94-01-00-F8-06-31-08-26-7B-09-27-9E-02-00-93-07-5E-0A-00-25-0B-00-05-0C-00-07-15-00-00-00-00-16-00-00-17-00-2A-00-46-4D-00-4C-51-00-E6-4E-00-70-2C-09-2E-13-30-19-32-19-1A-00-98-22-59-1B-00-BC-23-49-1C-00-92-24-5D-1D-00-A2-25-56-1E-FF-5E-1F-00-AC-27-2F-20-00-45-28-2F-59-00-62-00-00-00-04-61-62-81-A5-F3-60-18-19-00-26-0E-00-00-10-00-17-11-00-17-12-00-00-00-AD-13-00-00-0E-8B-0D-00-17-70-00-F5-2F-00-12-00-1C-00-12-00-19-02-82-02-41-06-21
2022-05-16 05:21:02.417 WH45 CO₂: Decoding...
2022-05-16 05:21:02.417 WH45 CO₂: temp=24.5, hum=47, pm10=1.8, pm10_24h=2.8, pm2.5=1.8, pm2.5_24h=2.5, CO₂=642, CO₂_24h=577
2022-05-16 05:21:02.417 LiveData: Wind Decode >> Last=1.8, LastDir=37, Gust=2.5, (MXAvg=0.6)

I never had such crashes before ...
Any ideas ?

A memory leak by the looks - possibly in Mono.

It looks as if Mono is being 'fingered' as the guilty party by the crash dump!

I am running a similar setup to you with one noticable exception - I am still on Debian 10.

I run two instances of CumulusMX with two Ecowitt weather stations, one using GW1000 and the other using an HP2551 console. The system also runs a MariaDb and serves three webservers using NginX. This is on an RPi4 with 4Gb memory, Debian 10 booted from USB3 connected SSD. Both instances of CumulusMX are run as Debian services. Other than upgrades to CumulusMX (and the odd power cut out here in the sticks) the system stays up. Currently at 57 days (last major power outage during one of the storms). I've never had issues with Mono, or noticed the oft reported memory leak.

What system are you on? Also you may want to check your SD card if you are booting/running from it - they do get corrupted from time to time and any issues may be very random in the crash report.

philpugh wrote: ↑Mon 16 May 2022 2:28 pm It looks as if Mono is being 'fingered' as the guilty party by the crash dump!

I am running a similar setup to you with one noticable exception - I am still on Debian 10.

I run two instances of CumulusMX with two Ecowitt weather stations, one using GW1000 and the other using an HP2551 console. The system also runs a MariaDb and serves three webservers using NginX. This is on an RPi4 with 4Gb memory, Debian 10 booted from USB3 connected SSD. Both instances of CumulusMX are run as Debian services. Other than upgrades to CumulusMX (and the odd power cut out here in the sticks) the system stays up. Currently at 57 days (last major power outage during one of the storms). I've never had issues with Mono, or noticed the oft reported memory leak.

What system are you on? Also you may want to check your SD card if you are booting/running from it - they do get corrupted from time to time and any issues may be very random in the crash report.

my (this) system runs not only CMX but also weewx. If my microSD card were corrupted, which is highly unlikely with an industrial SLC card of one year age only, weewx would also give up. CMX is usually more robust than weewx in case of "disturbances". And other services wouldn't run either. My other Debian installations (2 x RPi4B-8GB) run from a SSD and don't show any issues. Here, after a restart, everything works fine again. Therefore I doubt it's a corrupted card. It's a RPi4B-4GB by the way. And it didn't happen before with earlier versions. Who knows - maybe just a not yet discovered bug coming up in my constellation. The system is fed/feeds itself from a GW1000 with a WS80, WH40 and a bunch of extra sensors. I also run CMX on a RPi4B-4GB Win11 installation and no complaints either - running the "native" .NET environment. Meanwhile all my installs are on 3183.

Just strange. Luckily it's a test system and it has a mirror brother/sister system from which I can refill my gaps.
Or my RPi has some faulty RAM - also possible. Overheating can be excluded due to a double-fan and a metal "cage".

Could you monitor the memory usage by CMX periodically?

Unfortunately CMX does have some memory leaks in the network stack on Mono that do not occur when running on the .Net framework on Windows.

The only solution at the moment for network based stations under Mono is to restart CMX periodically - I do mine every couple of weeks, but some people have to reboot sooner.

Any ideas as to what memory usage is expected on Linux? I currently see 86,304k memory and 42,312k Shared memory on my openSUSE system running mono with CMX. OK mine does not run 24x7 but I can keep en eye on it, normally runs for about 12+ hours a day, this is a desktop PC with 32GB RAM.

Stuart

Around 90 - 110 MB is normal depending on what processes you have configured.

My "live" install I just restarted, it was up to 630 MB after 18 and a half days.