Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.5 (build 3282) - 23 February 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

CMX 3183 crashes

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
User avatar
Gyvate
Posts: 295
Joined: Wed 16 Dec 2020 2:14 pm
Weather Station: GW1x00/WH2650WiFi/HP2553/GW2000
Operating System: Win 11 (PC/RPi), Raspbian 11,WSL
Location: Saarbrücken, Germany
Contact:

CMX 3183 crashes

Post by Gyvate »

My CMX 3183 running on Debian with mono 6.12.0.122 crashes every now and then - with a long list of debugging info (see attachment).
The MXdiags file says nothing related, just ends
2022-05-16 05:21:02.394 Reading live data
2022-05-16 05:21:02.417 DoCommand(CMD_GW1000_LIVEDATA): Valid response
2022-05-16 05:21:02.417 Received: FF-FF-27-00-94-01-00-F8-06-31-08-26-7B-09-27-9E-02-00-93-07-5E-0A-00-25-0B-00-05-0C-00-07-15-00-00-00-00-16-00-00-17-00-2A-00-46-4D-00-4C-51-00-E6-4E-00-70-2C-09-2E-13-30-19-32-19-1A-00-98-22-59-1B-00-BC-23-49-1C-00-92-24-5D-1D-00-A2-25-56-1E-FF-5E-1F-00-AC-27-2F-20-00-45-28-2F-59-00-62-00-00-00-04-61-62-81-A5-F3-60-18-19-00-26-0E-00-00-10-00-17-11-00-17-12-00-00-00-AD-13-00-00-0E-8B-0D-00-17-70-00-F5-2F-00-12-00-1C-00-12-00-19-02-82-02-41-06-21
2022-05-16 05:21:02.417 WH45 CO₂: Decoding...
2022-05-16 05:21:02.417 WH45 CO₂: temp=24.5, hum=47, pm10=1.8, pm10_24h=2.8, pm2.5=1.8, pm2.5_24h=2.5, CO₂=642, CO₂_24h=577
2022-05-16 05:21:02.417 LiveData: Wind Decode >> Last=1.8, LastDir=37, Gust=2.5, (MXAvg=0.6)

I never had such crashes before ...
Any ideas ?
You do not have the required permissions to view the files attached to this post.
freddie
Posts: 2421
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: CMX 3183 crashes

Post by freddie »

A memory leak by the looks - possibly in Mono.
Freddie
Image
User avatar
philpugh
Posts: 377
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: CMX 3183 crashes

Post by philpugh »

It looks as if Mono is being 'fingered' as the guilty party by the crash dump!

I am running a similar setup to you with one noticable exception - I am still on Debian 10.

I run two instances of CumulusMX with two Ecowitt weather stations, one using GW1000 and the other using an HP2551 console. The system also runs a MariaDb and serves three webservers using NginX. This is on an RPi4 with 4Gb memory, Debian 10 booted from USB3 connected SSD. Both instances of CumulusMX are run as Debian services. Other than upgrades to CumulusMX (and the odd power cut out here in the sticks) the system stays up. Currently at 57 days (last major power outage during one of the storms). I've never had issues with Mono, or noticed the oft reported memory leak.

What system are you on? Also you may want to check your SD card if you are booting/running from it - they do get corrupted from time to time and any issues may be very random in the crash report.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
Gyvate
Posts: 295
Joined: Wed 16 Dec 2020 2:14 pm
Weather Station: GW1x00/WH2650WiFi/HP2553/GW2000
Operating System: Win 11 (PC/RPi), Raspbian 11,WSL
Location: Saarbrücken, Germany
Contact:

Re: CMX 3183 crashes

Post by Gyvate »

philpugh wrote: Mon 16 May 2022 2:28 pm It looks as if Mono is being 'fingered' as the guilty party by the crash dump!

I am running a similar setup to you with one noticable exception - I am still on Debian 10.

I run two instances of CumulusMX with two Ecowitt weather stations, one using GW1000 and the other using an HP2551 console. The system also runs a MariaDb and serves three webservers using NginX. This is on an RPi4 with 4Gb memory, Debian 10 booted from USB3 connected SSD. Both instances of CumulusMX are run as Debian services. Other than upgrades to CumulusMX (and the odd power cut out here in the sticks) the system stays up. Currently at 57 days (last major power outage during one of the storms). I've never had issues with Mono, or noticed the oft reported memory leak.

What system are you on? Also you may want to check your SD card if you are booting/running from it - they do get corrupted from time to time and any issues may be very random in the crash report.
my (this) system runs not only CMX but also weewx. If my microSD card were corrupted, which is highly unlikely with an industrial SLC card of one year age only, weewx would also give up. CMX is usually more robust than weewx in case of "disturbances". And other services wouldn't run either. My other Debian installations (2 x RPi4B-8GB) run from a SSD and don't show any issues. Here, after a restart, everything works fine again. Therefore I doubt it's a corrupted card. It's a RPi4B-4GB by the way. And it didn't happen before with earlier versions. Who knows - maybe just a not yet discovered bug coming up in my constellation. The system is fed/feeds itself from a GW1000 with a WS80, WH40 and a bunch of extra sensors. I also run CMX on a RPi4B-4GB Win11 installation and no complaints either - running the "native" .NET environment. Meanwhile all my installs are on 3183.

Just strange. Luckily it's a test system and it has a mirror brother/sister system from which I can refill my gaps. :ugeek:
Or my RPi has some faulty RAM - also possible. Overheating can be excluded due to a double-fan and a metal "cage".
User avatar
mcrossley
Posts: 12641
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: CMX 3183 crashes

Post by mcrossley »

Could you monitor the memory usage by CMX periodically?

Unfortunately CMX does have some memory leaks in the network stack on Mono that do not occur when running on the .Net framework on Windows.

The only solution at the moment for network based stations under Mono is to restart CMX periodically - I do mine every couple of weeks, but some people have to reboot sooner.
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: CMX 3183 crashes

Post by broadstairs »

Any ideas as to what memory usage is expected on Linux? I currently see 86,304k memory and 42,312k Shared memory on my openSUSE system running mono with CMX. OK mine does not run 24x7 but I can keep en eye on it, normally runs for about 12+ hours a day, this is a desktop PC with 32GB RAM.

Stuart
User avatar
mcrossley
Posts: 12641
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: CMX 3183 crashes

Post by mcrossley »

Around 90 - 110 MB is normal depending on what processes you have configured.

My "live" install I just restarted, it was up to 630 MB after 18 and a half days.
Post Reply