Potential issue with my Ecowitt gateways

[broadstairs]

I have seen this issue many times with my GW1100 but until yesterday it had not happened with my GW1000. The problem is that CMX says the gateway has not responded within 30 seconds and then repeats in this recent case several times from 2022-12-05 12:06 until 12:39, and again at 18:59. The errors showing from 21:12 are from a problem I caused by disconnecting wi-fi accidentally on that machine

Now this happened quite frequently on the GW1100 so a few days ago I swapped the gateways so the GW1100 was working with WD and CMX was using the GW1000. The GW1100 on WD has now been behaving with no apparent dropouts or missed data and now the GW1000 on CMX had this issue. There are no missing data log entries for this timeframe and the GW1000 posts a custom server which is my PWT on a tablet, during which time there were no alerts from PWT for missed data. I should add that WD is on a Windows PC and CMX is on Linux, both PCs are small ones using a Celeron chip.

I've attached the MXDiags log and a screenshot of the errors.

I'd just like to try to understand why this is happening?

Stuart

CMX-issue.zip

[mcrossley]

It appears to be your internet connection is going terribly slow/unreliable at times. I suspect the GW's are tied up waiting on responses from their various uploads and do not respond to the local API in a timely manner.

Take you interval FTP upload at 12:06. It never actually completes, because after it has been running for three failed cycles (three minutes in your case with one-minute uploads) it gets aborted by CMX. During this time the real time FTP connection also aborts and has to reconnect, and the interval FTP gets a failed DNS lookup for your FTP server.

Code: Select all

12:06:07.838 - CumulusMX Connecting to artemis.krystal.co.uk
12:06:12.192 - upload CU-tags.txt to CU-tags.phptmp
12:06:17.906 - rename CU-tags.phptmp
12:06:19.413 - renamed
12:06:19.439 - upload plaintext.txttmp to plaintext.txttmp
12:06:21.176 - Renaming plaintext.txttmp to plaintext.txt
12:06:21.237 - Renamed plaintext.txttmp
12:06:21.264 - Uploading web/websitedata.json to websitedata.jsontmp
12:06:22.242 - Renaming websitedata.jsontmp to websitedata.json
12:06:22.315 - Renamed websitedata.jsontmp
12:06:22.592 - Uploading wxnow.txt to wxnow.txttmp
12:06:23.046 - Renaming wxnow.txttmp to wxnow.txt
12:06:23.105 - Renamed wxnow.txttmp
12:06:23.141 - Uploading web/graphconfig.json to graphconfig.jsontmp
12:06:23.979 - Renaming graphconfig.jsontmp to graphconfig.json
12:06:24.018 - Renamed graphconfig.jsontmp
12:06:24.037 - Uploading web/availabledata.json to availabledata.jsontmp
12:06:25.449 - Renaming availabledata.jsontmp to availabledata.json
12:06:25.487 - Renamed availabledata.jsontmp
12:06:25.503 - Uploading web/tempdata.json to tempdata.jsontmp
12:06:29.717 - Renaming tempdata.jsontmp to tempdata.json
12:06:29.767 - Renamed tempdata.jsontmp
12:06:29.790 - Uploading web/pressdata.json to pressdata.jsontmp
12:06:32.075 - Renaming pressdata.jsontmp to pressdata.json
12:06:32.138 - Renamed pressdata.jsontmp
12:06:32.189 - Uploading web/winddata.json to winddata.jsontmp
12:06:40.046 - Renaming winddata.jsontmp to winddata.json
12:06:40.256 - Renamed winddata.jsontmp
12:06:42.294 - Uploading web/wdirdata.json to wdirdata.jsontmp
12:06:56.200 - Renaming wdirdata.jsontmp to wdirdata.json
12:07:03.050 - Renamed wdirdata.jsontmp
12:07:03.070 - Uploading web/humdata.json to humdata.jsontmp
12:07:10.215 - Renaming humdata.jsontmp to humdata.json
12:07:10.277 - Renamed humdata.jsontmp
12:07:10.305 - Uploading web/raindata.json to raindata.jsontmp
12:07:25.825 - Renaming raindata.jsontmp to raindata.json
12:07:28.132 - Renamed raindata.jsontmp
12:07:29.228 - Uploading web/dailyrain.json to dailyrain.jsontmp
12:07:31.394 - REALTIME FTP ERROR: Timed out trying to read data from the socket stream!
12:07:35.319 - Renaming dailyrain.jsontmp to dailyrain.json
12:07:37.387 - Renamed dailyrain.jsontmp
12:07:37.435 - Uploading web/dailytemp.json to dailytemp.jsontmp
12:07:40.759 - Renaming dailytemp.jsontmp to dailytemp.json
12:07:41.745 - Renamed dailytemp.jsontmp
12:07:42.091 - Uploading web/solardata.json to solardata.jsontmp
12:08:13.582 - Renaming solardata.jsontmp to solardata.json
12:08:53.621 - Error renaming solardata.jsontmp to solardata.json : Could not resolve host 'artemis.krystal.co.uk'
12:08:53.621 - Uploading graph data file: web/sunhours.json
12:09:00.433 - Warning, previous web update is still in progress, second chance, aborting connection
12:09:00.434 - Trying new web update

Compare that to a "normal" cycle earlier in the day that completes the whole upload in about 7 seconds.

Code: Select all

08:33:07.529 - CumulusMX Connecting to artemis.krystal.co.uk
08:33:07.963 - Uploading /home/stuart/CumulusMX/webfiles/CU-tags.txttmp to CU-tags.phptmp
08:33:08.145 - Renaming CU-tags.phptmp to CU-tags.php
08:33:08.188 - Renamed CU-tags.phptmp
08:33:08.212 - Uploading /home/stuart/.wine/drive_c/wxsim/plaintext.txttmp to plaintext.txttmp
08:33:08.382 - Renaming plaintext.txttmp to plaintext.txt
08:33:08.425 - Renamed plaintext.txttmp
08:33:08.483 - Uploading web/websitedata.json to websitedata.jsontmp
08:33:08.675 - Renaming websitedata.jsontmp to websitedata.json
08:33:08.712 - Renamed websitedata.jsontmp
08:33:08.729 - Uploading wxnow.txt to wxnow.txttmp
08:33:08.874 - Renaming wxnow.txttmp to wxnow.txt
08:33:08.930 - Renamed wxnow.txttmp
08:33:08.947 - Uploading web/graphconfig.json to graphconfig.jsontmp
08:33:09.117 - Renaming graphconfig.jsontmp to graphconfig.json
08:33:09.177 - Renamed graphconfig.jsontmp
08:33:09.194 - Uploading web/availabledata.json to availabledata.jsontmp
08:33:09.357 - Renaming availabledata.jsontmp to availabledata.json
08:33:09.397 - Renamed availabledata.jsontmp
08:33:09.417 - Uploading web/tempdata.json to tempdata.jsontmp
08:33:10.656 - Renaming tempdata.jsontmp to tempdata.json
08:33:10.690 - Renamed tempdata.jsontmp
08:33:10.717 - Uploading web/pressdata.json to pressdata.jsontmp
08:33:11.178 - Renaming pressdata.jsontmp to pressdata.json
08:33:11.211 - Renamed pressdata.jsontmp
08:33:11.224 - Uploading web/winddata.json to winddata.jsontmp
08:33:11.764 - Renaming winddata.jsontmp to winddata.json
08:33:11.829 - Renamed winddata.jsontmp
08:33:11.845 - Uploading web/wdirdata.json to wdirdata.jsontmp
08:33:12.279 - Renaming wdirdata.jsontmp to wdirdata.json
08:33:12.321 - Renamed wdirdata.jsontmp
08:33:12.344 - Uploading web/humdata.json to humdata.jsontmp
08:33:12.717 - Renaming humdata.jsontmp to humdata.json
08:33:12.761 - Renamed humdata.jsontmp
08:33:12.782 - Uploading web/raindata.json to raindata.jsontmp
08:33:13.262 - Renaming raindata.jsontmp to raindata.json
08:33:13.295 - Renamed raindata.jsontmp
08:33:13.313 - Uploading web/dailyrain.json to dailyrain.jsontmp
08:33:13.457 - Renaming dailyrain.jsontmp to dailyrain.json
08:33:13.515 - Renamed dailyrain.jsontmp
08:33:13.536 - Uploading web/dailytemp.json to dailytemp.jsontmp
08:33:13.684 - Renaming dailytemp.jsontmp to dailytemp.json
08:33:13.730 - Renamed dailytemp.jsontmp
08:33:13.751 - Uploading web/solardata.json to solardata.jsontmp
08:33:14.232 - Renaming solardata.jsontmp to solardata.json
08:33:14.283 - Renamed solardata.jsontmp
08:33:14.304 - Uploading web/sunhours.json to sunhours.jsontmp
08:33:14.458 - Renaming sunhours.jsontmp to sunhours.json
08:33:14.497 - Renamed sunhours.jsontmp
08:33:14.511 - Uploading web/airquality.json to airquality.jsontmp
08:33:14.775 - Renaming airquality.jsontmp to airquality.json
08:33:14.812 - Renamed airquality.jsontmp
08:33:14.830 - Disconnected from artemis.krystal.co.uk
08:33:14.831 - Process complete

It may be that CMX is polling your GW's at a faster rate and with shorter timeouts than WD - The update rate depends on the type of wind sensor attached and its update rate, too slow and you may miss some peak values as unlike most stations Ecowitt do not send a value like the highest gust value in last 10 minutes.

[broadstairs]

Interesting that my Internet access seems slow because I have full fibre (FTTP) so it should not be slow! If it happens again and I notice in time I'll check it. Yesterday we were out when it happened so only WD,CMX and gateways using the net, I suspect the current gateways are not equipped well enough to cope, I know they do not like a lot of sockets open somewhere it was suggested I think 6 is a max!

I know my hosting site can be slow but rarely and I'll check what DNS I have specified to make sure it is the best ones selected.

Stuart

[Nossie]

Stuart,

I think I'm on a different server group than you but I've not noticed any sluggishness with my krystal hosting with cmx.

Could not resolve host 'artemis.krystal.co.uk'

This does seem to suggest that it cant find the hosting, are you using pihole or any custom DNS resolution?

[mcrossley]

The upload speed is only the first link in the chain, it also depends on your contention ratio at whatever aggregates the links, the usage by all those people that share that aggregation, backbone congestion etc.

It could also be contention on your machine hosting CMX, was anything else hosting network, cpu etc.

If you are going direct to a ISP or public DNS, as if that is timing out then there are definitely issues.

[broadstairs]

I regularly check my DNS performance and have 3 DNS servers with the best performance and reliability configured on all my systems. It is very unusual for me to see problems with Krystal but the odd glitch is always possible. The PC running CMX only runs that 24x7 otherwise it runs WXSim under wine 4 times a day and not at the time this issue happened. Strange thing is that my GW1100 when it was accessed by CMX used to see these sort of problems with the gateway not responding sometimes several times a day and probably 3 or so days a week. The GW1000 which now is used by CMX has only had this one extended issue in the past week. I do suspect this may well have been several issues coming together which caused it. I should have said that the Linux system is not an RPi but a small format PC with a 1900mhz Celeron chip running openSUSE Leap 15.4 full up to date and my router is a BT Home Hub2.

Stuart

[mcrossley]

If your public IP is pingable it might be interesting to set up one of the online latency checkers to monitor it.

[philpugh]

I host my own website along with three Ecowitt stations (three instances of CMX) on the one Pi 4. I occasionally get an error like yours and haven't been able to determine what is causing it - but my 'problem' happens less than once a day across all three CMX instances so I don't really worry about it.

I have a mesh wifi network and 150Mbps FTTP external network but I also suspect that the ECOWITT servers occasionally cause a hang on my GW1000 and GX2000 Ecowitt gateways but can't really prove it!

[broadstairs]

I have been looking at what I have set up for FTPing and realised that because I use CU-tags and CU-defs I do not need 99% of what CMX actually creates and FTPs to my website so for now I've removed all the json stuff and only FTP my CU-tags and the two realtime files. I've left the plaintext from WXSim in for now while I sort out FTPing outside of CMX. That reduces considerably the amount of files to FTP. In the future I may need to FTP some other files as I move more things on my site over from WD to CMX. I have also ordered a new wi-fi card for the PC as it was only using 2.4ghz and slowly! I've altered the DNS servers it uses as well although the others were not bad they could be improved. I also think the DNS failures could be attributable to the slow wi-fi connection.

Stuart

[Phil23]

Slight Tangent but,
I feel my GW1100 is somewhat less stable than when I got it about 12 months back.

At times it appears unresponsive & can't get an interface or ping it.
Reboot often solves it, but not always immediately.

I does have a large number of sensors; 3 air, 5 Soil Moisture, 16 Temp & now feeds to both CMX and Home Assistant,
but don't think that any of that could be relevant.

Suspect it may have Flash wear, but have no idea how they write their stuff.
At any rate & have bought a Spare as I've seen enough issues recently to suspect something at a hardware level is going on.

[SamiS]

Slight Tangent but,
I feel my GW1100 is somewhat less stable than when I got it about 12 months back.

At times it appears unresponsive & can't get an interface or ping it.
Reboot often solves it, but not always immediately.

I does have a large number of sensors; 3 air, 5 Soil Moisture, 16 Temp & now feeds to both CMX and Home Assistant,
but don't think that any of that could be relevant.

Suspect it may have Flash wear, but have no idea how they write their stuff.
At any rate & have bought a Spare as I've seen enough issues recently to suspect something at a hardware level is going on.

Have you updated the firmware how recently? They had at least one bug that prevented it from reconnecting the wifi. Also the latest version has some optimizations regarding the webserver. My gw1100 has been running since June, and has not yet suffered from any problems except the wifi issue. For your case, if it does not even respond on ping, it must be either very totally stuck or for some reason dropped off the wifi. If your router gives enough information, you could check if the gw1100 is still associated or not when the problem occurs.

[Phil23]

Have you updated the firmware how recently?

Yeah firmware is all up to date on all devices.

Wifi Network consists of 7 managed AP's all on same SSID.
(Omada Network).

It could be an ESP/Omada issues relating to BSSID's.
Devices are supposed to scan the network, find know SSID's, and their BSSID's & then connect to the strongest BSSID.
The BSSID is the individual AP's MAC address, as opposed to the SSID which will refer to all 7 devices.

Spotted that particular GW1100 off the air yesterday & noticed the other was a bit slow in response.

Rebooted all the Omada AP's & all returned to normal with no other intervention, so may well have been a Network issue.
(The Omada Controller showed the uptime of most of the AP's to be sitting at 45 days).

[SamiS]

Multiple AP systems do tend to have their own issues, even on the ”high end” corporate systems (I do my daywork on a team that among other things hosts also a Cisco wifi environment with 1000+ AP:s, so nothing is a surprise anymore). Let’s hope that your gw is indeed fine, and the ap issue just an intermittent sw glitch.