New feature - Davis WLL historic data catch-up

[mcrossley]

Build 3064 - version 3.4.0 of Cumulus MX now allows users to Davis WeatherLink Live devices to download catch-up data for periods when Cumulus is not running.

It does this by fetching the missing data from WeatherLink.com.

There are a few caveats around this though...

1. You must have at least a Pro subscription for your device on WL.com to grant access to the historic data.
2. The new APIv2 used to access the data.
3. The APIv2 keys are available via WL.com.

If you meet the criteria and are willing to accept that the API is not yet "production" ready, then these are the steps you need to follow...

1. Sign on to the Davis weatherlink.com with your normal username/password
2. On wl.com open your user profile, you will see a section titled "API Key v2"
3. If you do not already have a key and secret displayed, click "Generate v2 key" to create them. Copy these strings somewhere safe so you can later paste them into Cumulus
4. Ensure you are running Cumulus MX v3.4.0
5. If this your first run of CumulusMX.
   1. Go into the Station setup and select Davis WLL as the station type
   2. In the WLL section enter your API key and secret. LEAVE THE STATION ID BLANK
   3. Set up your sensor to transmitter ID mappings if different from the defaults
   4. It is not necessary to enter the WLL IP address, Cumulus should just find it on your network. But if you know it it does not harm to enter it, Cumulus will change it anyway if your WLL ever changes address.
   5. Save the setup
   6. Stop/Restart Cumulus
6. If you only have a single station registered with WL.com, then Cumulus should discover its ID automatically and start using it
7. But if you have more than one, the Cumulus will list their API IDs and names in the console window.
8. Note the ID you want to use. Go back into the Station setup and enter the ID there. You do not need to restart Cumulus after that.
9. Enjoy! Cumulus should now download missing data for periods it is stopped.

There is effectively no limit on how much historic data you can download**, I have tested using a four day catch-up without issues.

Davis impose rate and data limits on the APIv2 - currently they are...
- 300 API calls per hour, or 2 API calls per second
- 24 hours of data in any one call
Cumulus honours these restrictions.

The APIv2 uses both a key and a secret that encrypts part of the URL making playback or emulating the API calls by another user impossible. A nice feature other web APIs could emulate.

**Cumulus MX will get the max 24 hours data per call if required, which will give a limit of 300 days of historic data catch-up.

[saratogaWX]

This is great, Mark! Will you also do a Wiki page for this too?

[mcrossley]

Maybe - at some point! Just fixing/working around some bugs at the mo - some mine, some Davis!

[mcrossley]

Here's a start... https://cumuluswiki.org/a/WeatherLink_Live

I have linked it off the Supported Devices page

[mcrossley]

Davis have announced that the WeatherLink.com API v2 that this solution uses will go live on 11th March 2020.

[mcrossley]

And live it has gone. I have updated the Wiki page above to reflect the process for now obtaining your key and secret

[hermite]

WLL_history.png

And live it has gone. I have updated the Wiki page above to reflect the process for now obtaining your key and secret

Hello, I'm trying this feature and all connections work fine, However, the downloaded files do not contain valid data (all values are set to '0'), yet I can confirm on WLL that data is populated with valid values... example from downloaded historical data;

28-08-18;22:57;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;22:58;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;22:59;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:00;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:01;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:02;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:03;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:04;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:05;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:06;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:07;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:08;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0
28-08-18;23:09;0,0;0;0,0;0,0;0,0;0;0,0;0,0;0,0;320,4;0,0;0;0,0;0,0;0,0;0,0;0;0,00;0,00;-4,0;0;0,0;0;0,0;0,0;0,0;0,0

Any thoughts?

[mcrossley]

Please stop cumulus and reset your data so that it will perform a catch up again.
The start cumulus using the -debug command line parameter.
Then post the MXdiags log file.

[hermite]

Please stop cumulus and reset your data so that it will perform a catch up again.
The start cumulus using the -debug command line parameter.
Then post the MXdiags log file.

I have 1.4K archive days, so I halted the process pretty early. I shrunk the debug file as it is too big and repetitive for this site to handle.

What stands out is 'WL.com historic: found an unknown data structure type [4]!'.

[freddie]

I shrunk the debug file as it is too big and repetitive for this site to handle.

For future reference, if you compress/zip the file first then you will comfortably be within the site's max file size limit.

[mcrossley]

A data structure type = 4 means that the archive data is coming from a "WeatherLink IP/Vantage Connect" rather than a WeatherLink Live device.

Sensor type = 33 means the sensor data is from a Vantage Pro.

Cumulus is only set up to work with WeatherLink Live devices using the Davis v2 API.

[hermite]

A data structure type = 4 means that the archive data is coming from a "WeatherLink IP/Vantage Connect" rather than a WeatherLink Live device.

Cumulus is only set up to work with WeatherLink Live devices using the Davis v2 API.

Perfect and thank you. I didn't enough reading with respect to limitation to WLL and not VP.

[mcrossley]

I have only added the ability to decode the various WLL records. The WL API catalog of sensor definitions is currently 2.5 MB is size, they support an awful lot of different sensor types, it would be a big task just to support the common ones.

If you want to import old history from the VP Pro into Cumulus, then I'm afraid you will either have to download the data and manually manipulate it, or knock up a script/program to do that.

[hermite]

I have only added the ability to decode the various WLL records. The WL API catalog of sensor definitions is currently 2.5 MB is size, they support an awful lot of different sensor types, it would be a big task just to support the common ones.

If you want to import old history from the VP Pro into Cumulus, then I'm afraid you will either have to download the data and manually manipulate it, or knock up a script/program to do that.

Tks again. I have downloaded all my history as .csv. I'll see how I can play with that.