Cumulus Support

Posted: **Tue 05 Sep 2023 7:27 am**

'I've now turned on the FTP upload diagnostic log for something that has only happened twice before.'

My stations last upload to the server was at 21:24 last night (4/9/23). Cumulus was still running this morning (on my RPi), but not uploading (I use php).

The diagnostic log I normally have switched on is attached. All seemed well until 21:24 (line 14031).

A shut down and restart this morning reset Cumulus, and were OK again.

Last time this happened was in April.

Thanks
Martyn

Posted: **Tue 05 Sep 2023 7:44 am**

As an aside to this issue the data for that period is missing from my 'Sep23log.txt' .txt file.

Tried the 'CreateMissing.exe' but no luck.

May be because I'm trying to get my 15 minute values back, and not the 'Day record'!

I'll probably try adding the missing records manually later, as Weatherlink has them.

Thanks

Martyn

Posted: **Tue 05 Sep 2023 10:20 am**

A network issue occurred at 21:11:07, and the real-time upload for that interval did not complete until 21:24. Meantime all the other real-time uploads queued up and started processing in parallel with the 21:11:07 upload. They shouldn't do that, so I'll look at that bit of code and fix it.

The main loops of MX stop processing after that with no error, so it is hard to understand what went wrong. I suspect the thread pool ran out of space, but it should have given an error. Your system logs at that point may give a clue. But I think fixing the above issue will prevent this happening.

You are misunderstanding CreateMissing - it uses the data from the monthly log files to (re)create missing entries in your day file. It cannot invent data to put into the monthly log files. It should create your missing day file entry from yesterday. But the data gap must be filled from your stations data logger using a catch-up process. So, you can rewind MX and use the 15 minute data from the logger to fill the gap.

Posted: **Tue 05 Sep 2023 1:32 pm**

I was about to add missing values manually into the Sept23log.txt

Is there a way of doing it automatically?

Can I delete everything in Sept23log.txt from the failed upload onwards? Would it then import the missing values ?

Thanks

Martyn

Posted: **Tue 05 Sep 2023 2:25 pm**

Not so easy, you would also have to set the last update time in today.ini, along with all the other values to reflect times before 21:34 yesterday.

You could...
1. Stop MX
2. Restore the data folder files from the Sunday midnight backup
3. Start MX

Job, done. But you will only have 15 minute data from Sunday midnight until present.

Personally, to preserve the maximum data, I would...
0. Have a coffee
1. Stop MX
2. Make a backup of the data folder
3. Copy over the files from the midnight backup of Sunday night.
4. Run MX, and let it catch up. That will get your missing data back.
5. Stop MX.
6. Make a copy of the new Sept23 log.txt file.
7. Restore the previous backup of the data folder.
8. Edit the restored Sept23 file, and the saved copy from step 6. Copy the missing time period from the saved copy into the current data folder copy.
9. Delete the dayfile entry for yesterday (if it is present)
10. Run CreateMissing
11. Start MX

Phew! Have a beer.

Posted: **Tue 05 Sep 2023 2:41 pm**

Ah, not so easy then.

I think I’ll manually update the Sept23log.txt file by adding in the missing time slots (copy entry from previous time slot) then updating with the correct time stamp and values from Weatherlink using the Data Log Viewer, Edit facility.

Thanks for responding Mark, and for the feedback on a possible update.

Martyn

Posted: **Tue 12 Sep 2023 5:45 am**

I have a similar issue with PHP uploads, see attachment.

Some remarks:

There seem to be multiple errors (network sending to the server - also third party; network reading the sensors...)
It starts complaining "No such host is known" so it can't read the nameserver?
Mail starts failing at 2h00 (also nameserver issue)
At 2h40 it starts failing reading the sensor IDs
Shutdown given at 6h39 giving additional info on the waiting tasks
Restart of CMX is OK

I first thought it was the OS having an issue with the sockets (with some additional error in CMX) and in the past I tried a reboot. This time I just restarted CMX after the successful shutdown. That did work so even when the issue originates in the OS or the network, it is the total reconnection which solves the issue. The issue happens just after a week (earliest) until 3 weeks (latest).

Posted: **Tue 12 Sep 2023 6:26 am**

Hi Hans

That looks like your internet server host calling foul in the middle of the night and then CMX getting stuck. At least your data will have been recorded locally and you will have a consistent data set.

Posted: **Tue 12 Sep 2023 7:47 am**

AndyKF650 wrote: ↑Tue 12 Sep 2023 6:26 am That looks like your internet server host calling foul in the middle of the night and then CMX getting stuck. At least your data will have been recorded locally and you will have a consistent data set.

Not sure it is a server fail as it says: No such host is known so it seems more like a network fail.
Wherever it originates, CMX apparently does not handle it well.
Indeed, data is stored at Ecowitt.net and catch-up is OK.

Posted: **Tue 12 Sep 2023 10:32 am**

And btw... a selfinduced network error - I switched off the wifi - did recover when restored.
So it may be something in the OS or the server after all.
Log fragment below.

Code: Select all

2023-09-12 12:22:00.437 *** Data input appears to have stopped
2023-09-12 12:22:05.210 Not Connected
2023-09-12 12:22:05.210 Not Connected
2023-09-12 12:22:07.267 ERROR: No data received from the GW1000 for 30 seconds
2023-09-12 12:22:07.268 Using IP address = 192.168.178.24 Port = 45000
2023-09-12 12:22:07.810 Error opening TCP port: No route to host
2023-09-12 12:22:07.810 Error opening TCP port: No route to host
2023-09-12 12:22:10.929 Error opening TCP port: No route to host
2023-09-12 12:22:15.970 Error opening TCP port: No route to host
2023-09-12 12:22:15.970 Error opening TCP port: No route to host
2023-09-12 12:22:25.971 Error opening TCP port: Network is unreachable
2023-09-12 12:22:30.930 Error opening TCP port: Network is unreachable
2023-09-12 12:22:32.810 Not Connected
2023-09-12 12:22:37.267 ERROR: No data received from the GW1000 for 30 seconds
2023-09-12 12:22:37.268 Using IP address = 192.168.178.24 Port = 45000
2023-09-12 12:22:37.269 Error opening TCP port: Network is unreachable
2023-09-12 12:22:40.970 Not Connected
2023-09-12 12:22:44.130 Error opening TCP port: No route to host
2023-09-12 12:22:44.130 Error opening TCP port: No route to host
2023-09-12 12:22:47.252 Error opening TCP port: No route to host
2023-09-12 12:22:55.931 Not Connected
2023-09-12 12:22:57.250 Error opening TCP port: No route to host
2023-09-12 12:23:00.444 *** Data input appears to have stopped
2023-09-12 12:23:05.410 Error opening TCP port: No route to host
2023-09-12 12:23:05.410 Error opening TCP port: No route to host
2023-09-12 12:23:07.268 ERROR: No data received from the GW1000 for 30 seconds
2023-09-12 12:23:07.268 Using IP address = 192.168.178.24 Port = 45000
2023-09-12 12:23:08.529 Error opening TCP port: No route to host
2023-09-12 12:23:11.649 Error opening TCP port: No route to host
2023-09-12 12:23:15.410 Error opening TCP port: No route to host
2023-09-12 12:23:15.410 Error opening TCP port: No route to host
2023-09-12 12:23:25.420 Connected OK
2023-09-12 12:23:25.421 Reading firmware version
2023-09-12 12:23:25.422 Reconnected to Ecowitt device
2023-09-12 12:23:25.457 Ecowitt firmware version: GW1100A_V2.2.6
2023-09-12 12:23:25.457 Reading Ecowitt system info
2023-09-12 12:23:25.477 DoCommand(CMD_READ_SSSS): Invalid response
2023-09-12 12:23:25.478 Checksum: Error - Bad checksum
2023-09-12 12:23:25.478 Nothing returned from System Info!
2023-09-12 12:23:25.478 Reading sensor ids
2023-09-12 12:23:25.495 DoCommand(CMD_READ_SENSOR_ID_NEW): Invalid response
2023-09-12 12:23:25.497 Ckecksum: Error - Calculated data length [2817] exceeds the buffer size!
2023-09-12 12:23:25.514 DoCommand(CMD_GW1000_LIVEDATA): Invalid response
2023-09-12 12:23:25.514 GetLiveData: Invalid response
2023-09-12 12:23:26.650 Connected OK
2023-09-12 12:23:26.650 Reading firmware version
2023-09-12 12:23:26.672 Ecowitt firmware version: GW1100A_V2.2.6
2023-09-12 12:23:26.672 Reading Ecowitt system info
2023-09-12 12:23:26.689 Gateway Info: frequency: 868MHz, main sensor: Other than WH24, date/time: dinsdag 12 september 2023 12:23:25, Automatic DST adjustment: True
2023-09-12 12:23:26.689 Gateway Info: Gateway clock is 1 secs slow compared to Cumulus
2023-09-12 12:23:26.690 Reading sensor ids
2023-09-12 12:23:30.410 Connected OK
2023-09-12 12:23:30.410 Reading firmware version
2023-09-12 12:23:30.428 Ecowitt firmware version: GW1100A_V2.2.6
2023-09-12 12:23:30.430 Reading Ecowitt system info
2023-09-12 12:23:30.660 Gateway Info: frequency: 868MHz, main sensor: Other than WH24, date/time: dinsdag 12 september 2023 12:23:29, Automatic DST adjustment: True
2023-09-12 12:23:30.660 Gateway Info: Gateway clock is 1 secs slow compared to Cumulus
2023-09-12 12:23:30.660 Reading sensor ids
2023-09-12 12:23:35.410 Connected OK
2023-09-12 12:23:35.411 Reading firmware version
2023-09-12 12:23:35.430 Ecowitt firmware version: GW1100A_V2.2.6
2023-09-12 12:23:35.431 Reading Ecowitt system info
2023-09-12 12:23:35.454 Gateway Info: frequency: 868MHz, main sensor: Other than WH24, date/time: dinsdag 12 september 2023 12:23:34, Automatic DST adjustment: True
2023-09-12 12:23:35.454 Gateway Info: Gateway clock is 1 secs slow compared to Cumulus
2023-09-12 12:23:35.454 Reading sensor ids
2023-09-12 12:25:00.470 DoLogFile: Writing log entry for 12-9-2023 12:25:00
2023-09-12 12:25:00.472 DoLogFile: log entry for 12-9-2023 12:25:00 written
2023-09-12 12:25:00.478 Writing today.ini, LastUpdateTime = 12-9-2023 12:25:00 raindaystart = 628,3 rain counter = 629,1
2023-09-12 12:30:00.023 DoLogFile: Writing log entry for 12-9-2023 12:30:00
2023-09-12 12:30:00.025 DoLogFile: log entry for 12-9-2023 12:30:00 written
2023-09-12 12:30:00.031 Writing today.ini, LastUpdateTime = 12-9-2023 12:30:00 raindaystart = 628,3 rain counter = 629,1
2023-09-12 12:30:00.199 Alarm (HTTP Uploads): Trigger cleared

Posted: **Tue 12 Sep 2023 1:49 pm**

Hi Hans

Just a thought but have you reserved the ip address and mac address of the ws-gateway on your router DCHP listing.

I found to my cost that not doing this when everything went pear shaped the ip address of the ws gateway provided by the router was changed. Reserving the address sorted that issue but you need to remember that CMX and the gateway need to be checked to ensure the correct connectivity.

Posted: **Tue 12 Sep 2023 2:03 pm**

Yes, I did.

Posted: **Tue 19 Sep 2023 4:02 pm**

The network stack seems to get "stuck" and does not clear down in timely manner when MX closes and reopens the connections.

All the reports of this nature that I recall are on Linux, so I'm wondering if Mono is the problem.

Stopping and restarting MX also stops and restarts the mono stack.

Posted: **Tue 19 Sep 2023 5:18 pm**

mcrossley wrote: ↑Tue 19 Sep 2023 4:02 pm The network stack seems to get "stuck" and does not clear down in timely manner when MX closes and reopens the connections.

All the reports of this nature that I recall are on Linux, so I'm wondering if Mono is the problem.

Stopping and restarting MX also stops and restarts the mono stack.

So would be an idea to have CMX restart itself when it detects such situation?
(Possibly a user config?)

Posted: **Tue 19 Sep 2023 8:18 pm**

I looked into auto-restart, and quickly realised why Steve never implemented it - it's pretty hard to do!

Cumulus Support

Upload Fail

Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail

Re: Upload Fail