CumulusMX stopped reporting to WU after power outage

[VTHokie74]

I had an extended power failure last night. After the power came back on my RP 3 and CumulusMX restarted OK. I noticed the web site was updating so I knew it was connected to the Internet and FTP was working. This morning I noticed that both WU and PWS were showing my station off line since the power outage. I checked the settings for both and they were enabled and normal on CMX. I restarted CMX using the script and both WU and PWS started updating. Any ideas about why this would happen? Could just be a glitch from the less than normal shutdown I suppose.

This is the CumulusMX log file:

pi@raspberrypi:~/CumulusMX $ cat cumulusmxsh.log
2016-07-26 18:17:20 - CumulusMX Logfile reached 10000. Truncating old Logfile, creating a new Logfile
2016-07-26 18:17:20 - Network Failure - Unable to reach 8.8.8.8 - Waiting 60 seconds before retry. Max Attempts 5 x 60 seconds. 0 of 5 retries.
2016-07-26 18:18:20 - Network Failure - Unable to reach 8.8.8.8 - Waiting 60 seconds before retry. Max Attempts 5 x 60 seconds. 1 of 5 retries.
2016-07-26 19:59:25 - Network Failure - Unable to reach 8.8.8.8 - Waiting 60 seconds before retry. Max Attempts 5 x 60 seconds. 2 of 5 retries.
2016-07-26 20:00:35 - Network Failure - Unable to reach 8.8.8.8 - Waiting 60 seconds before retry. Max Attempts 5 x 60 seconds. 3 of 5 retries.
2016-07-26 20:00:46 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-26 20:00:47 - CumulusMX Script is being called with -s (Status)
2016-07-26 20:01:14 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-26 20:01:15 - CumulusMX Script is being called with -s (Status)
2016-07-26 20:01:36 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-26 20:01:37 - CumulusMX Script is being called with -s (Status)
2016-07-26 20:01:45 - Network Failure - Unable to reach 8.8.8.8 - Waiting 60 seconds before retry. Max Attempts 5 x 60 seconds. 4 of 5 retries.
2016-07-26 20:02:05 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-26 20:02:06 - CumulusMX Script is being called with -s (Status)
2016-07-26 20:02:45 - Network Failure - Unable to reach Internet/Network - Based on Configuration (NETWORKFAILOVER = 1) Continue starting CumulusMX without Network
2016-07-26 20:02:45 - CumulusMX trying to start with these parameters /home/pi/CumulusMX/CumulusMX.exe -port 8998
2016-07-26 20:02:57 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-26 20:02:58 - CumulusMX Script is being called with -s (Status)
2016-07-26 20:03:09 - System - CumulusMX is successfully being started
2016-07-26 20:03:24 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-26 20:03:25 - CumulusMX Script is being called with -s (Status)
2016-07-26 23:59:01 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-26 23:59:03 - ZIP Backup Create a zip backup from /home/pi/CumulusMX into folder /home/pi/routerdrive1/backup/daily
2016-07-27 00:00:21 - ZIP Backup - /home/pi/routerdrive1/backup/daily2016-07-27_0000_CumulusMX.zip successfully created
Now 1 of 5 backup files exist - remaining Backup files: 4 - nothing to delete
2016-07-27 09:53:18 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-27 09:53:25 - CumulusMX is being Stopped with Parameter -r (Restart)
2016-07-27 09:53:25 - CumulusMX trying to start with these parameters /home/pi/CumulusMX/CumulusMX.exe -port 8998
2016-07-27 09:53:35 - CumulusMX is being Started with Parameter -r (Restart)
2016-07-27 10:08:26 - CumulusMX 2.5.2.0 (2016-05-02) Script is being called with some Parameter
2016-07-27 10:08:29 - CumulusMX Script is being called with -s (Status)

[steve]

If you zip up the MXdiags folder and attach it, I'll take a look. Hopefully you are running with debug logging enabled?

[VTHokie74]

If you zip up the MXdiags folder and attach it, I'll take a look. Hopefully you are running with debug logging enabled?

Unfortunately no, I turned it on for the next time I get a power problem. I could always just pull the plug on the RP3 of course.

[steve]

The diags files may still be useful even without debug logging, if you'd like to attach them. My guess would be that as MX was apparently started when there was no network connection, this caused a problem with the WU and PWS threads, causing them to hang or crash.

[VTHokie74]

The diags files may still be useful even without debug logging, if you'd like to attach them. My guess would be that as MX was apparently started when there was no network connection, this caused a problem with the WU and PWS threads, causing them to hang or crash.

I suspect you are correct. The RP3 probably booted faster than my router and cable modem when the power came back. I saw several network failure entries where it was looking for the DNS server 8.8.8.8 (I use the Google DNS servers).

[steve]

The system HTTP code did get an exception when it tried to do the updates initially, but it handled the exception, so the thread didn't hang, and it continued to try to do the update. But it kept getting "The task was cancelled" back from the HTTP code, and I'm afraid I have no idea why that would be, or what if anything I can do about it.

[VTHokie74]

OK, thanks for looking. At least I can fix it with a manual restart.

[VTHokie74]

I also found that my Windows network drive had not mounted properly. I ran raspi-config and I think I had not checked "wait on network for boot". I believe this ,ight have bee the issue. Normally the network is already running on a Raspberry reboot but after a power failure the Raspberry will boot quicker than most routers. I rebooted and the network drive mounted properly.

[VTHokie74]

I had to change the battery in a UPS today so I took my router down. I shut down the RP3 first and then started it before powering the router. The "wait on network to boot" setting worked and the boot was delayed until the router was powered up. CumulusMX started normally. It updated the Weather Underground and the web site as expected.

The only thing that didn't work is that my Windows shared drive didn't mount so I had to mount it using "sudo mount -a". I will check into that. It seems to mount fine on just a reboot.

[philpugh]

I have a similar problem with my Pi/Davis/NAS and Pi/WH3083/NAS setups. I do the Wait for Network - which comes up quickly - but the NAS servers (2* BUFFALO 2TB Raid 1 systems) take up to 5 minutes before accepting connections. I store all my data in MySQL database on the NAS servers. Fortunately power outages are rare these days - one in last year - but I think my solution will be to start a process at boot that waits for the NAS/MySQL to be available - before starting CumulusMX. An additional 5 mins wait after power outage isn't really a problem.

[VTHokie74]

I added a script as a crontab entry to mount all drives a few minutes before the nightly backup script runs. That way if there has been a power failure the NAS drive will be mounted if it didn't mount correctly after a power failure.

[philpugh]

Post by VTHokie74 » Sat Aug 20, 2016 3:05 am
I added a script as a crontab entry to mount all drives a few minutes before the nightly backup script runs. That way if there has been a power failure the NAS drive will be mounted if it didn't mount correctly after a power failure.

I need the NAS RAID servers to be up before CumulusMX runs as I store all Weather data from two stations on these - most of it in MySQL database/tables. The main weather system is a Davis VantageVue and CMX is running on a BananaPi with an SSD. This systems also acts as my webserver and mailserver - and a few other things - and is only 5% utilised on average.