Missing Data puzzle

[boss281]

I have Cumulus MX installed and running on a dedicated Windows 10 laptop. I have a Davis Vantage Pro 2 and Davis WeatherLink Live hooked up to my WiFi in the house. The version is 3141. It works, I leave it alone and don't play with it. It feeds WeatherUnderground, the PWS and Windy.

Yesterday, all my laptops seemed to have a major update available, so I manually updated those except the rig with CMX. I got distracted and didn't do the CMX machine and left it running with CMX running along. At 6pm yesterday that machine did the WIndows update install itself, rebooted and I didn't know it until 12pm today. So CMX was offline all that time. I restarted CMX and walked away. Two hours later the data is showing as missing on all my CMX consumers but the Davis Android app data is complete. So it's clear the data was stored and available.

I could be wrong, but in the past I could swear that WU and the others filled in the data when CMX was restarted after being down. Usually after that long the history files load up and all is well. But only ONE history file loaded at startup after yesterday's downtime. So I have the data gap.

So:
1. Is my memory correct? Does CMX obtain the history during the down time and fill in the missing data points.
2. If so, is there a way to get it back so all my consumers have the data?
3. Is there a way to troubleshoot the problem on why it happened in the past. I'll update to the new version anyhow later today.

Thanks in advance.

[Mapantz]

Do you have a pro subscription with the WeatherLink Live?

[PaulMy]

Hi,

So:
1. Is my memory correct? Does CMX obtain the history during the down time and fill in the missing data points.
2. If so, is there a way to get it back so all my consumers have the data?
3. Is there a way to troubleshoot the problem on why it happened in the past. I'll update to the new version anyhow later today.

1. Yes, if CMX was stopped in the normal way and you have a WL.com Pro subscription. But in your case with the Windows update and PC stop and restart it may not have been shut down properly. As best as I know and as indicated in the Third Party uploads settings all of those 3 do have an Enable catch up option.
2. Depends on your reply to Mapantz question regarding your WL.com subscription type.
3. Check the MXdiags file for the period including the Windows update PC shut down and restart, does it show anything just prior to shut down?

Enjoy,
Paul

[boss281]

Yes, I do have the WeatherLink Pro subscription. Here's the log. I see line items as it appears it couldn't retrieve historical info. So maybe the system wasn't updating at this time after all but the time it ran into trouble.

20230709-161633.txt

[Mapantz]

Is the "archive rate" on weatherlink.com the same as in the Cumulus MX settings?

Mine for example, are both set at 5 minutes.

[PaulMy]

Yes, I do have the WeatherLink Pro subscription. Here's the log. I see line items as it appears it couldn't retrieve historical info. So maybe the system wasn't updating at this time after all but the time it ran into trouble.

Code: Select all

2023-07-12 17:46:50.782 WLL Health: Get WL.com Historic Data
2023-07-12 17:46:50.939 WLL Health: Invalid historic message received
2023-07-12 17:46:50.939 WLL: Percentage good packets received from WLL 93.48% - (6801,97527)
2023-07-12 17:50:00.192 DoLogFile: Writing log entry for 7/12/2023 5:50:00 PM
2023-07-12 17:50:00.192 DoLogFile: log entry for 7/12/2023 5:50:00 PM written
2023-07-12 17:50:00.192 Writing today.ini, LastUpdateTime = 7/12/2023 5:50:00 PM raindaystart = 13.8 rain counter = 13.8
2023-07-12 17:54:00.497 Updating CWOP
2023-07-12 17:55:00.467 DoLogFile: Writing log entry for 7/12/2023 5:55:00 PM
2023-07-12 17:55:00.467 DoLogFile: log entry for 7/12/2023 5:55:00 PM written
2023-07-12 17:55:00.467 Writing today.ini, LastUpdateTime = 7/12/2023 5:55:00 PM raindaystart = 13.8 rain counter = 13.8

From this last log entry WLL seems ok. But that also shows that CMX was stopped abruptly (no proper closing message) and that could be the cause of no catch-up data when CMX restarted.

If you have missing data in CMX and that missing data is in your WL.com account you should be able to get that archive data by:
With CMX stopped, from a good CMX backup from before the issue occurred - i.e. \CumulusMX\backup\daily\20230712000000\ copy all the files and paste them in your working \CumulusMX\data folder.
At a restart CMX will catch up data from WL.com from the time that last backup occurred.
Take a backup of your current \CumulusMX\data for quick restore just-in-case.

Enjoy,
Paul

[mcrossley]

Code: Select all

GetWlHistoricData: Invalid historic message received

This was fixed in the latest release - you need to update your copy of MX...

3.25.2 - b3245
——————————————
Fixed
- Davis WLL not doing historic catch-up or downloading stats from weatherlink.com

Davis have changed the returned JSON format slightly.

[Mapantz]

Of course, I forgot about that! doh