Missing Data - WLL

[Phil23]

Hi All,

I've been running a test install on my laptop with WLL since December.
Had a power outage yesterday & now seem to have a missing data period in CMX.

Last recording before the outage is 9:50 & then the next is 19:50. The outage was around an hour.
When I look in the history on WLL, I see all the data has been uploaded at the 5 minute interval, but seems Like MX is not retrieving it & filling in the gap.

My main MX Win 10 install is fine, it read all the data from the console & think the laptop it runs on did not loose battery power.
Both the WLL & the console read the same VP2.

Any thoughts?

Code: Select all

16/01/22,09:00,22.7,76,18.3,0.0,0.0,0,0.0,0.0,1009.0,18.8,26.4,56,0.0,22.7,22.7,1.3,271,0.00,92.94,25.6,548,0.3,360,0.0,0.0,25.1,28.8
16/01/22,09:10,23.2,77,19.0,1.4,6.4,13,0.0,0.0,1009.1,18.8,26.4,55,3.2,23.2,23.2,1.4,272,0.15,93.09,26.2,589,0.3,13,0.0,0.0,25.6,29.8
16/01/22,09:20,23.7,74,18.8,3.6,8.0,59,0.0,0.0,1009.1,18.8,26.5,55,4.8,23.7,23.7,1.3,251,0.15,93.09,26.2,629,0.3,71,0.0,0.0,25.7,30.1
16/01/22,09:30,23.7,73,18.5,5.4,14.5,16,0.0,0.0,1009.2,18.8,26.5,56,4.8,23.7,23.7,1.3,234,0.15,93.09,25.7,668,0.3,5,0.0,0.0,25.3,29.9
16/01/22,09:40,23.7,73,18.6,4.5,14.5,15,0.0,0.0,1009.2,18.8,26.6,56,4.8,23.7,23.7,1.6,283,0.15,93.09,25.8,706,0.3,22,0.0,0.0,25.4,30.0
16/01/22,09:50,23.6,73,18.5,3.7,9.7,31,0.0,0.0,1009.3,18.8,26.7,55,4.8,23.6,23.6,1.4,211,0.15,93.09,25.9,743,0.3,351,0.0,0.0,25.4,29.8
16/01/22,19:50,28.3,62,20.4,2.3,4.8,338,0.0,0.0,1006.6,18.8,29.2,44,1.6,28.3,30.2,0.0,7,0.15,93.09,31.8,0,0.3,338,0.0,0.0,31.1,36.0
16/01/22,20:00,27.9,63,20.2,1.9,4.8,338,0.0,0.0,1006.9,18.8,28.8,40,0.0,27.9,29.7,0.0,0,0.15,93.09,31.4,0,0.4,360,0.0,0.0,30.7,35.5
16/01/22,20:10,27.4,65,20.3,2.0,4.8,338,0.0,0.0,1007.2,18.8,28.6,43,1.6,27.4,29.1,0.0,0,0.20,93.14,30.9,0,0.4,338,0.0,0.0,30.3,35.1
16/01/22,20:20,27.1,67,20.4,2.5,4.8,338,0.0,0.0,1007.3,18.8,28.4,44,3.2,27.1,28.7,0.0,0,0.20,93.14,30.5,0,0.4,338,0.0,0.0,29.8,34.8
16/01/22,20:30,26.6,70,20.6,6.4,17.7,75,0.0,0.0,1007.7,18.8,28.1,40,8.0,26.6,26.6,0.0,0,0.20,93.14,29.5,0,0.4,128,0.0,0.0,28.8,34.5

[mcrossley]

Hi Phil, have you correctly input all the WLL API information into the CMX configuration? As you have two stations you will also have to tell CMX which one is the WLL (if you only have one it just uses that one). If you look in the MXdiags log file it will show if it is connecting to wl.com OK and finding your station id.

[Phil23]

As far as I know....

I'm sure I've left this Laptop off for a few days back in Dec & on starting MX I made a point of watching for the download & it appeared all OK.

I don't suppose there is a way to force a re-sync for a given period at this point?

It's only test data so of no great issue other than the reason why.

My two stations on WLL are the Airlink & the VP2. API Key used is the V2.

20211225-163047.txt in the Diags covers Dec thru to 10pm after the outage & does show a few errors after the power dropped.
Around line 25,596 & onwards.

Diags in first post.

Cheers.

[mcrossley]

You had a network outage that prevented CMX from talking to your WLL, so there is a gap in the data for this period.

You could restore CMX from the daily backup files at 9am on 16th. CMX will then recover the data from wl.com up to the present time.

[Phil23]

Repeated the same scenario last night.
Power outage @ 6:16pm. Resumed around 8:10pm.

Things fail at different rates due to mix of network hardware that is & isn't on UPS/Hybrid Power.
Test Laptop Sleeps after 5 minutes on Battery. Was woken up after power restored & Network back up & running.

Again WLL has complete data uploaded and so does my main CMX install that's connected to the console.

Seems that the functionality of the check with WLL may only look at the latest records at both ends & download missing data based on that.

Wondering is there's avenue to improve this?
Maybe with a flag of some sort in CMX for "Last update before error/anomaly", then a process of comparing WLL data from that point to current to find out if there's a gap.

Obviously this would need a bit of reworking as I assume currently that the WLL catch-up is added to the end of the month file & for above would need to be inserted in between existing data in the above case & other relevant records files updated appropriately.

Given that Power Outages are only becoming more frequent in my world, at this point I'm pretty hesitant to abandon my console based install & move completely to the WLL.

Final thought - Pros & Cons weighed up; rather than CMX trying to check for this as a matter of course, would another alternative be to have a variation on the Find Missing?
As I know damn well that the Bloody Power went out AGAIN, and know where the gap is, it may be easier from a programming point of view, & acceptable from the users point of view to have a utility where you could ask to check for missing data between say 6:00 & 9:00 on Feb 6th in the above case.

Food for thought as at this point the WLL's data integrity is not the same as that of a Console based system.

Cheers

Phil.

Code: Select all

06/02/22,18:00,24.4,46,12.1,14.5,25.7,133,0.0,0.0,1015.3,124.4,25.5,40,12.9,23.7,24.4,0.8,367,4.98,193.07,22.3,302,6.4,89,0.0,0.0,23.0,26.7
06/02/22,18:10,24.1,48,12.4,15.9,30.6,132,0.0,0.0,1015.4,124.4,25.4,40,11.3,23.1,24.1,0.0,53,5.38,193.47,21.7,259,6.5,161,0.0,0.0,22.4,26.5
06/02/22,20:20,19.2,63,12.0,11.2,29.0,132,0.0,0.0,1017.3,124.4,23.3,46,8.0,19.0,19.2,0.0,0,5.38,193.47,17.4,0,6.6,130,0.0,0.0,17.9,21.4
06/02/22,20:30,19.0,63,11.8,9.8,19.3,136,0.0,0.0,1017.3,124.4,23.7,45,3.2,19.0,19.0,0.0,0,5.38,193.47,17.4,0,6.6,138,0.0,0.0,17.9,21.1
06/02/22,20:40,18.6,65,11.8,7.3,16.1,146,0.0,0.0,1017.4,124.4,23.9,44,16.1,18.6,18.6,0.0,0,5.38,193.47,17.7,0,6.6,119,0.0,0.0,18.1,20.7

[mcrossley]

There are two mitigations already in place for this.

1. Disable No Sensor check if it is enabled. Then MX will stop logging data when the data flow from the station stops. On restart it will pick up from the time the last data was received.
2. Do not allow your computer to sleep/hibernate. This is not supported by CMX, never has and will not be for the foreseeable future. Configure your computer to shutdown on low battery. If MX is running as a service then it too will perform an orderly shutdown.

Yes, there are ways MX could cope with situations like this. But remember MX supports lots of different stations not just the WLL. Going down the route that every station has to be handled differently in "N" number of situations is not a path I am prepared to go down.

as at this point the WLL's data integrity is not the same as that of a Console based system.

I'm not sure that is quite true. The WLL does not lose data, it buffers it until it has a connection to wl.com then uploads it.
On start-up MX downloads the data from either wl.com for the WLL or the console logger in the case of the VP2.

I do not see what would be different if the computer running MX connected to VP console was put into standby and then restarted, or one connected to a WLL. They would both have gaps in their data as standby is not supported.

The only difference is you are dependent on the network being powered for the WLL (it would the same for a VP IP logger of course). But MX can stop logging in that situation and will catch-up as normal on restart. It would be same if you had a powered USB hub to your USB logger and that lost power, or the cat pulled the cable out!

Bottom line, there are many things MX *could* do given enough development resource and time, but both of those are limited.

[Phil23]

Bottom line, there are many things MX *could* do given enough development resource and time, but both of those are limited.

First up, that's fully understood & the efforts greatly appreciated. I'm sure all users feel the same way.
I think everyone felt a loss when Steve retired,
Followed by a huge sigh of relief when you picked up the ball & ran with it.

My comments are to be taken as thinking out loud more so than suggesting things need a change.
More thrown out for comment than anything else.

1. Disable No Sensor check if it is enabled. Then MX will stop logging data when the data flow from the station stops.

Found that now. Ini file only setting? Mine is "NoSensorCheck=0"
So I assume if data flow stopped CMX should shit down; seeming not to happen, can easily test by blocking the IP of the WLL on my Omada Network for 15 minutes or so.
Will actually test it like that now, IP address is currently blocked for a bit over 10 minutes so the 8:00am reading should be missed & I CMX should stop?


In these two instances I'm certain it hasn't shut down.

2. Do not allow your computer to sleep/hibernate.

Grrrr,
One of my absolute pet hates; I've always set my Power Plans to my needs, till Microsoft comes along & thinks I need to do better with my current footprint.....
Very frustrating to choose powerful hardware & have them decide to tame it down a bit.

The WLL does not lose data, it buffers it until it has a connection to wl.com then uploads it.

Understand that side of it completely & the data uploaded to WLL is always totally complete.
I should note that mine is uploading at 5 minute intervals, while CMX logs every 10.
Not sure if that has any data compatibility implications or not.
Should the two match or does it not really matter?

I think I am correct is saying that if the PC/Laptop/rPi where to have gone down at the time of the outages, on resumption & restarting CMX the data recorded by the WLL would be uploaded & the subsequently downloaded by CMX.
What I see as the situation I'm seeing is because,
a) - CMX continues to run once the network is lost,
b) - CMX is resuming when the laptop comes out of sleep.

I do wonder is there's any lag in when data is uploaded; probably based on the random times I see activity on the WLL.
Those times when the Blue LED decides to flash for a few minutes....

As mentioned originally, this is just an experimental install, so there's no real consequence it the data loss.
Just a case of learning more about who WLL interacts & putting what I'm seeing out there for discussion.

Test Result from Blocking the IP @ 8:14:30 ish:-

WLL site updated in a matter of seconds & included data records for 8:00, 8:05 & 8:10am.
Shutdown & restarted at 8:16 & it added an 8:15 record.

Proves the point to me that it adds records that don't match my normal log interval so I should fix that.
Just didn't see CMX shut down as expected, assuming I understand what the "NoSensorCheck=0" setting should work.

Cheers

Phil.

PS. Was expecting LED activity on the WLL when I unblocked, but either missed it or it wasn't present.
Was trying to take note.

[Phil23]

MS Decided to Sleep this particular Laptop again last night, following installing some pending updates.
To be expected, as this is my morning/evening browsing PC; not a weather dedicated unit.

Woke it up, MX not running & Feb22log.txt showed last record as 9:40pm & the DB file stamped 9:42.
Mx not running due to update reboot; exactly as expected.

Started & updated & 5 minuted records added as expected.

What caught my eye & I thought worthy of posting is the messages in the console.
Some I haven't seen before, referencing wl.com.

Screenshot 2022-02-10 072409.jpg

[RayProudfoot]

You can turn off sleep mode on Windows 10. See if this helps.
https://www.hellotech.com/guide/for/how ... windows-10

You’ll still need to be wary of updates but they can be controlled better with Pro.

[PaulMy]

@Phil23,
Sometimes my Windows 10 PC shuts down and says it needs to restart (actually quite often). I have StartWatch installed so my programs restarts automatically usually without any issue. The PC shut down at about 2:20 this morning and restarted as usual. However when I looked at the PC this morning I noticed in the DOS start screen a message the same as your attached. I didn't do a screen capture and just copied the text:

Code: Select all

Cumulus MX v.3.15.0 build 3169
Working Dir: C:\KomokaWeather\CumulusMXwll\
Cumulus is already running - but 'Stop second instance' is disabled
Delaying start for 10 seconds
Start-up delay complete, continuing...

Cumulus running at: http://localhost:8999/
                    http://192.168.0.212:8999/

  Open the admin interface by entering one of the above URLs into a web browser.

Today.ini = data\today.ini
Last update=2022-02-09T02:10:00
Weatherlink.com overall System Status: 'Degraded Performance', Updated: 09/02/22 2:23:21 AMError:
   wl.com system: WeatherLink Website, status: Operational, updated: 09/02/22 2:23:21 AM
   wl.com system: Mobile Applications, status: Operational, updated: 09/02/22 2:23:21 AM
   wl.com system: APIs and Data Feeds, status: Degraded Performance, updated: 09/02/22 2:23:21 AM
   wl.com system: Data Ingestion, status: Operational, updated: 09/02/22 2:23:21 AM
   wl.com system: System Communication, status: Operational, updated: 09/02/22 2:23:21 AM
Today.ini = data\today.ini
Last update=2022-02-09T02:10:00
09/02/22 2:23:24 AM
Type Ctrl-C to terminate
Downloading Historic Data from WL.com from: 2022-02-09T02:10:00 to: 2022-02-09T02:23:24
 - processed 100%
Connecting real time FTP
Normal running

Everything seems to be running ok though.

I have a second CumulusMX running a test version and that is also using the WLL and it has the same message when it restarted.
My CumulusMX on WiFiLogger was fine at restart.

So looks like WL.com issue. The first time I have seen that status message since using the WLL in October 2020.

Enjoy,
Paul

[Phil23]

You can turn off sleep mode on Windows 10. See if this helps.
https://www.hellotech.com/guide/for/how ... windows-10

You’ll still need to be wary of updates but they can be controlled better with Pro.

Thanks Ray,

Yes aware of that & all my systems here run on Win10 Pro.
My main Weather PC is a re-birthed Win10 Pro Tecra i7 & have it configured so it gives me no issues running it's Console based USB install.

It never sleeps, but I do allow it to now reboot on updates & have Scheduled Tasks to take care of the Start/Stop of CMX.

The one being mentioned here is at his point just experimental, on my daily use Tecra; Coffee Browsing in the morning & after dinner at night.

It's only running MX as means of learning all the if's & buts about a WLL install Vs my current console based installation.

One thing I still have a question over is the update interval miss-match.
My CMX has for years (2011 start with C1, MX since 16), has always operated at a 10 minute log interval.
I plan to keep it at that....

But the WLL has only to options close to that, 5 or 15 minutes. Currently using the 5 minute option, but not sure what implications that has if any with a data catch-up.

Cheers

Phil.

[Mapantz]

Ah, so all of us WLL minions got the same message?!

I would guess that Davis have updated something on their servers, so nothing to worry about.

[SJG1976]

That is interesting.

Just seen I had the same message as well.

[boss281]

This message? About the APIs and data feed degradation? Been happening for about a week off and on for me.

[SJG1976]

That's the one.

I noticed it on my Test Install, but thought it was where I was having WiFi issues.

Oh well.