Page 1 of 2
First ever CumulusMX crash
Posted: Tue 30 Mar 2021 11:49 am
by Feek
I have monitoring enabled so I can see if CumulusMX crashes and I just had an alert.
When I look at the status of the service, this is what I saw.
Code: Select all
● cumulusmx.service - CumulusMX service
Loaded: loaded (/etc/systemd/system/cumulusmx.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-03-30 12:20:03 BST; 21min ago
Docs: https://cumuluswiki.org/a/Main_Page
Process: 22226 ExecStart=/usr/bin/mono-service -d:/opt/CumulusMX CumulusMX.exe -service (code=exited, status=0/SUCCESS)
Process: 13202 ExecStopPost=/bin/rm /tmp/CumulusMX.exe.lock (code=exited, status=0/SUCCESS)
Main PID: 22227 (code=exited, status=1/FAILURE)
Mar 30 12:20:03 cumulusmx systemd[1]: cumulusmx.service: Main process exited, code=exited, status=1/FAILURE
Mar 30 12:20:03 cumulusmx systemd[1]: cumulusmx.service: Failed with result 'exit-code'.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
I've restarted the service and it's all running again but I wonder if there's anything I should look at for this or is it just 'one of those things'?
Running Cumulus MX 3.10.4 b3121 on a Pi Zero.
Thanks.
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 12:43 pm
by freddie
You could attach the latest log file from MXdiags to a reply to this topic. Also, look in /var/log/syslog for any entries around the time of the crash.
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 1:11 pm
by Feek
Thanks,
When I look in syslog, this is all that's there for that time. The previous entry is a few minutes earlier.
Code: Select all
Mar 30 12:20:03 cumulusmx systemd[1]: cumulusmx.service: Main process exited, code=exited, status=1/FAILURE
Mar 30 12:20:03 cumulusmx systemd[1]: cumulusmx.service: Failed with result 'exit-code'.
Excuse my ignorance, but how do I run MXdiags?
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 1:50 pm
by freddie
Feek wrote: ↑Tue 30 Mar 2021 1:11 pmExcuse my ignorance, but how do I run MXdiags?
You don't. There is a directory/folder in your MX install that is called MXdiags. In this folder can be found the application's log files. Zip up the one that covers the period of your crash and attach it to a reply to this topic.
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 2:18 pm
by Feek
Thank you. I've attached it but I suspect these are the only lines of interest.
Code: Select all
2021-03-30 12:19:00.211 Writing today.ini, LastUpdateTime = 30/03/2021 12:19:00 raindaystart = 175.2 rain counter = 175.2
2021-03-30 12:19:46.518 Realtime[83]: Uploading - realtime.txt
2021-03-30 12:19:46.779 Realtime[83]: Uploading - realtimegauges.txt
2021-03-30 12:20:00.013 Reading sensor ids
2021-03-30 12:20:00.236 DoLogFile: Writing log entry for 30/03/2021 12:20:00
2021-03-30 12:20:00.240 DoLogFile: Written log entry for 30/03/2021 12:20:00
2021-03-30 12:20:00.255 Writing today.ini, LastUpdateTime = 30/03/2021 12:20:00 raindaystart = 175.2 rain counter = 175.2
2021-03-30 12:20:01.028 DoCommand(CMD_READ_SENSOR_ID_NEW): No response received
!!! Unhandled Exception !!!
System.NullReferenceException: Object reference not set to an instance of an object.
at CumulusMX.GW1000Station.GetSensorIdsNew () [0x00019] in <cb338b78ebaa485892728bab41fd242d>:0
at CumulusMX.GW1000Station.Start () [0x000e8] in <cb338b78ebaa485892728bab41fd242d>:0
at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00014] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0
at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 2:19 pm
by freddie
@mcrossley - one for you
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 2:58 pm
by mcrossley
Got it, and I can see where the exception occurred - I'll get that fixed. The root cause was the GW1000 not responding to a command - read sensor ids new.
Though looking at the full log that appears to happen semi-regularly, along with read live data commands failing.
Anyway, looks fine apart from that, I get the code fixed to handle rouge responses better.
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 3:03 pm
by Feek
mcrossley wrote: ↑Tue 30 Mar 2021 2:58 pm
I'll get that fixed. The root cause was the GW1000 not responding to a command - read sensor ids new.
Oh cool, it was worth reporting then.
Re: First ever CumulusMX crash
Posted: Tue 30 Mar 2021 3:18 pm
by freddie
Feek wrote: ↑Tue 30 Mar 2021 3:03 pm
mcrossley wrote: ↑Tue 30 Mar 2021 2:58 pm
I'll get that fixed. The root cause was the GW1000 not responding to a command - read sensor ids new.
Oh cool, it was worth reporting then.
Definitely, and thank you.
Re: First ever CumulusMX crash
Posted: Tue 06 Apr 2021 6:17 pm
by Feek
I see this has been fixed, thank you.
I very much like the new gauge layout in b3122
Re: First ever CumulusMX crash
Posted: Thu 08 Apr 2021 6:55 pm
by Feek
mcrossley wrote: ↑Tue 30 Mar 2021 2:58 pm
Got it, and I can see where the exception occurred - I'll get that fixed. The root cause was the GW1000 not responding to a command - read sensor ids new.
Though looking at the full log that appears to happen semi-regularly, along with read live data commands failing.
Anyway, looks fine apart from that, I get the code fixed to handle rouge responses better.
Just had another crash, here's the service status output and the latest MXdiags log is attached.
Code: Select all
● cumulusmx.service - CumulusMX service
Loaded: loaded (/etc/systemd/system/cumulusmx.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-04-08 19:30:01 BST; 19min ago
Docs: https://cumuluswiki.org/a/Main_Page
Process: 9320 ExecStart=/usr/bin/mono-service -d:/opt/CumulusMX CumulusMX.exe -service (code=exited, status=0/SUCCESS)
Process: 26048 ExecStopPost=/bin/rm /tmp/CumulusMX.exe.lock (code=exited, status=0/SUCCESS)
Main PID: 9321 (code=exited, status=1/FAILURE)
Apr 08 19:30:01 cumulusmx systemd[1]: cumulusmx.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 19:30:01 cumulusmx systemd[1]: cumulusmx.service: Failed with result 'exit-code'.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
Re: First ever CumulusMX crash
Posted: Thu 08 Apr 2021 8:41 pm
by mcrossley
Hmm, it's failing more subtly than I thought.
I can fix the crash, but I could really do with a log file capturing with debug and data logging enabled so I can see exactly why the response is invalid.
Re: First ever CumulusMX crash
Posted: Thu 08 Apr 2021 8:52 pm
by Feek
mcrossley wrote: ↑Thu 08 Apr 2021 8:41 pm
I can fix the crash, but I could really do with a log file capturing with debug and data logging enabled so I can see exactly why the response is invalid.
What exactly can I do to get that for you?
Re: First ever CumulusMX crash
Posted: Fri 09 Apr 2021 2:11 pm
by mcrossley
Just switch on debug and data logging from Program Settings | Program General Options.
Then if it crashes again before I get the next release out, please post the the MXdiags log file - it may get fairly large, I'll only really be interested in the last few pages before the crash.
Re: First ever CumulusMX crash
Posted: Fri 09 Apr 2021 3:37 pm
by Feek
mcrossley wrote: ↑Fri 09 Apr 2021 2:11 pm
Just switch on debug and data logging from Program Settings | Program General Options.
Then if it crashes again before I get the next release out, please post the the MXdiags log file - it may get fairly large, I'll only really be interested in the last few pages before the crash.
I've done that - Watch this space.