Davis WLL - missing data on computer restart

Vegit8 · Post by **Vegit8** » Thu 12 Mar 2020 12:59 pm

I have been helping a friend set up a new weather site using a Davis Vantage Pro2 ISS and a WLL device, and Cumulus MX.
This is the first time I have set up a WLL.

Everything seems to be working fine
The website is running as expected www.charlesdowding.co.uk/weather/index.htm
The Cumulus Console updates very frequently, as I am led to beleive it should.

The other day he had to shut down his computer overnight.
When he rebooted in the morning there did not appear to be any catch up from Weatherlink.
I thought that this was now a new feature in MX.

I have sourced the key and the secret word from the Davis man Jeremy Wall.

I have checked the Weatherlink site and in the data tables - all the missing data is visible. It just not seem to have made its way down to the Cumulus system.

Perhaps I have missed something in the setup

Is there anyway to force a catch up?

Screenshot of MX local console showing gap in data.

http://www.geoffwebber.co.uk/forum_pics/Lostdata.jpg

Text from MX Diags file

2020-03-09 05:56:47.721 ========================== Cumulus MX starting ==========================
2020-03-09 05:56:47.723 Command line: /Users/charlesdowding/CumulusMX/CumulusMX.exe
2020-03-09 05:56:47.723 Cumulus MX v.3.4.0 build 3064
2020-03-09 05:56:47.724 Platform: Mac OS X
2020-03-09 05:56:47.725 OS version: Unix 19.3.0.0
2020-03-09 05:56:47.780 Mono version: 6.8.0.105 (2019-10/82fffe07497 Tue Feb 4 10:09:51 EST 2020)
2020-03-09 05:56:47.780 Current culture: English (United Kingdom)
2020-03-09 05:56:47.781 Directory separator=[/] Decimal separator=[.] List separator=[,]
2020-03-09 05:56:47.781 Date separator=[/] Time separator=[:]
2020-03-09 05:56:47.781 Standard time zone name: GMT
2020-03-09 05:56:47.781 Daylight saving time name: BST
2020-03-09 05:56:47.781 Daylight saving time? False
2020-03-09 05:56:47.781 09/03/2020 05:56:47
2020-03-09 05:56:47.781 Data path = data/
2020-03-09 05:56:47.820 Creating backup folder backup/20200309055647/
2020-03-09 05:56:47.929 Created backup folder backup/20200309055647/
2020-03-09 05:56:47.935 Reading Cumulus.ini file
2020-03-09 05:56:47.938 ASM=10 AST=00:10:00
2020-03-09 05:56:47.938 Cumulus start date: 02 March 2020
2020-03-09 05:56:47.939 Debug logging is disabled
2020-03-09 05:56:47.939 Data logging is disabled
2020-03-09 05:56:47.939 Logging interval = 5
2020-03-09 05:56:47.939 NoSensorCheck = 0
2020-03-09 05:56:47.991 Calculating sunrise and sunset times
2020-03-09 05:56:47.993 Sunrise: 06:36:23
2020-03-09 05:56:47.993 Sunset : 18:05:19
2020-03-09 05:56:47.993 Tomorrow sunrise: 06:34:10
2020-03-09 05:56:47.993 Tomorrow sunset : 18:06:59
2020-03-09 05:56:47.996 Station type: Davis WLL
2020-03-09 05:56:47.996 WindUnit=mph RainUnit=mm TempUnit=°C PressureUnit=mb
2020-03-09 05:56:47.996 YTDRain=0.000 Year=0
2020-03-09 05:56:47.996 RainDayThreshold=-1.000
2020-03-09 05:56:47.996 Offsets and Multipliers:
2020-03-09 05:56:47.996 PO=0.000 TO=0.000 HO=0 WDO=0 ITO=0.000 SO=0.000 UVO=0.000
2020-03-09 05:56:47.996 PM=1.000 WSM=1.000 WGM=1.000 TM=1.000 TM2=0.000 HM=1.000 HM2=0.000 RM=1.000 SM=1.000 UVM=1.000
2020-03-09 05:56:47.996 Spike removal:
2020-03-09 05:56:47.996 TD=999.000 GD=999.000 WD=999.000 HD=999.000 PD=999.000
2020-03-09 05:56:47.996 MR=999.000 MH=999.000
2020-03-09 05:56:47.996 Cumulus Starting
2020-03-09 05:56:47.997 Opening station
2020-03-09 05:56:48.004 Last update time from today.ini: 08/03/2020 20:45:00
2020-03-09 05:56:48.004 Read today file: Date = 08/03/2020, LastUpdateTime = 08/03/2020 20:45:00, Month = 3
2020-03-09 05:56:48.004 Sensor clock 00:00:00
2020-03-09 05:56:48.004 Station clock 00:00:00
2020-03-09 05:56:48.004 ReadTodayfile: set notraininit false
2020-03-09 05:56:48.005 /Users/charlesdowding/CumulusMX/data/alltime.ini
2020-03-09 05:56:48.006 Alltime.ini file read
2020-03-09 05:56:48.009 MonthlyAlltime.ini file read
2020-03-09 05:56:48.009 Month.ini file read
2020-03-09 05:56:48.010 Year.ini file read
2020-03-09 05:56:48.011 Finding raintoday from logfile data/Mar20log.txt
2020-03-09 05:56:48.011 Expecting listsep=, decimal=.
2020-03-09 05:56:48.022 Midnight rain found in the following entry:
2020-03-09 05:56:48.022 08/03/20,00:00,10.4,86,8.2,9.5,22.0,209,0.0,0.0,1008.0,0.0,14.8,66,14.0,10.4,10.4,0.0,0,0.00,0.00,7.2,0,0.0,205,0.0,0.0
2020-03-09 05:56:48.024 Midnight rain found, setting midnight rain count = 0
2020-03-09 05:56:48.024 Checking rain counter = 1.6
2020-03-09 05:56:48.025 Rain counter set to = 1.6
2020-03-09 05:56:48.025 Getting rain totals, rain season start = 1
2020-03-09 05:56:48.025 Today = 09/03/20
2020-03-09 05:56:48.026 Rainthismonth from dayfile.txt: 0
2020-03-09 05:56:48.026 Rainthisyear from dayfile.txt: 0
2020-03-09 05:56:48.027 Station type = Davis WLL
2020-03-09 05:56:48.034 Get WL.com Current Data
2020-03-09 05:56:48.054 Attempting to find WLL via zero-config...
2020-03-09 05:56:48.350 WLL found at IP address 192.168.1.183
2020-03-09 05:56:48.350 WLL found at IP address 192.168.1.183
2020-03-09 05:56:48.350 WLL found at IP address 192.168.1.183
2020-03-09 05:56:48.623 You are not running the latest version of CumulusMX, build 3068 is available.
2020-03-09 05:56:49.058 Reading history data from log files
2020-03-09 05:56:49.059 Loading last N hour data from data logs: 08/03/2020 20:45:00
2020-03-09 05:56:49.073 Loaded 13 entries to last hour data list
2020-03-09 05:56:49.085 Loaded 37 entries to last 3 hour data list
2020-03-09 05:56:49.098 Loaded 270 entries to graph data list
2020-03-09 05:56:49.132 Loaded 884 entries to recent data list
2020-03-09 05:56:49.133 Loaded 3 entries to daily data list
2020-03-09 05:56:49.133 Reading archive data from WeatherLink API
2020-03-09 05:56:49.155 671 web tags initialised
2020-03-09 05:56:49.168 HTML root path = /Users/charlesdowding/CumulusMX/interface
2020-03-09 05:56:49.225 Get WL.com Historic Data
2020-03-09 05:56:49.225 Rollover hour = 0
2020-03-09 05:56:49.458 WeatherLink API archive reading thread completed

mcrossley · Post by **mcrossley** » Thu 12 Mar 2020 1:41 pm

I would first update to the latest version of Cumulus - there have been fixes for WLL devices.

After you have upgraded, then next time you start Cumulus run it with the -debug parameter. I need to see a log with full debugging data in it.

Vegit8 · Post by **Vegit8** » Thu 12 Mar 2020 1:50 pm

Ok -
Have already upgraded to 3068.
Does the logging add greatly to the overhead?
As I am not at the location often and the user is not IT literate, MX will be running in debug mode for some time.
MX is being run on a mac via a shell script.
Is this the correct syntax?

#!/bin/zsh
cd CumulusMX
Mono CumulusMX.exe -debug

mcrossley · Post by **mcrossley** » Thu 12 Mar 2020 2:04 pm

The log file will grow over time, depends how long you leave it, but its the order of megabytes, not gigabytes.

If you have remote access the admin interface, then you can switch off the debug logging via the admin interface - Settings|Station Settings|Options - Debug logging & Data logging

You can do this after say 10 minutes. Or if you like, stop Cumulus and restart with the -debug option, then you can leave it.

Your syntax is correct.

Vegit8 · Post by **Vegit8** » Thu 12 Mar 2020 3:19 pm

Hi Mark
Thanks for that.
My problem with updating the System options via the menu is that the debug logging is only for the current session.
This would mean that MX will have already loaded and (I guess) gone through the WLL data recall before I could change the setting in the screen.

I'm going to create a new launcher that has the debug switch hard coded, that way it will be enabled on startup.
My user can then switch between the two versions as necessary.

One other question (to which you may not have an answer). MX is installed on an iMac (which has not complained about any firewall issues).
I would have thought that a UDP connection should be just allowed through the users home broadband router, is it possible that there is an issue there?

mcrossley · Post by **mcrossley** » Thu 12 Mar 2020 10:59 pm

I was thinking that you could start CMX with the -debug parameter, then switch it off via the interface to prevent the log files growing too large. It will revert to no debug logging next time it was started.

I would think that the local firewall would have to opened for UDP port 22222 as this is purely in-bound traffic. Routers will normally pass broadcast/multicast traffic on the local network.

Vegit8 · Post by **Vegit8** » Fri 13 Mar 2020 8:35 am

Thanks Mark
I have passed alternate launcher to my friend with instructions on how to run it as an alternate if he has had to turn off his machine for any period. Then shut down the debug version after an hour or so, revert to normal and send me the log.

One point that did cross my mind...
I understand that the WLL integration will pull down data from weather cloud for the period that has elapsed since last known Cumulus data and restart.
Is there a possibility to manually roll back the pick up date so that gaps in data (such as the situation with my friend) can be recovered?

mcrossley · Post by **mcrossley** » Fri 13 Mar 2020 10:55 am

Yes - Just stop Cumulus, restore the data files from one of the backup folders, and restart. Cumulus will then catch-up from the time of that backup.

It keeps the last 10 days of backups (taken at day rollover), and the state of the data at the last 10 startups.

If you want to preserve a backup - take a copy of that folder now before it is overwritten!

Vegit8 · Post by **Vegit8** » Fri 20 Mar 2020 10:02 am

Hi Mark
I am back on the users machine.
I have updated MX.exe to build 3069
I have opened up the router to pass port 22222 on UDP to the machine that is running MX
The iMac has its own firewall turned off
I moved the oldest set of backup files to a copy of the data folder (just in case the download didnt work) - renamed the current data file so that it was preserved and renamed the copy to data.

I restarted MX and the log says this

2020-03-20 09:22:01.211 Get WL.com Current Data
2020-03-20 09:22:01.230 Attempting to find WLL via zero-config...
2020-03-20 09:22:01.449 WLL found, reporting its IP address as: 192.168.1.183
2020-03-20 09:22:01.449 WLL found, reporting its IP address as: 192.168.1.183
2020-03-20 09:22:01.449 WLL found, reporting its IP address as: 192.168.1.183
2020-03-20 09:22:01.729 WeatherLink API Current: No data was returned. Check your Device Id.

I have checked the device ID - which is showing as a 5 digit number 86nn6
Having no way of verifying this as it was auto completed on setup the only other place I could see a device ID was on the devices page on weatherlink.com this figure is completely different being 12 chars long mixed alphanumeric.
I tried entering that but the page shouted at me as it was looking for a numeric value.
I have to assume that the 5 digit numeric is correct.

I have checked the API key and secret as supplied to me by Jeremy with the values now visible on the WL site and they are identical.
Also I would have assumed that some other error would have been displayed if the key / secret combination was incorrect

Running in debug mode gives me additional information

2020-03-20 09:57:41.909 Get WL.com Current Data
2020-03-20 09:57:41.914 WeatherLink URL = https://api.weatherlink.com/v2/current/ ... pi-key=THE CORRECT KEY IS HERE&api-signature=I HAVE CUT THE SIG FROM HERE available by PM/email if necessary
2020-03-20 09:57:41.927 Attempting to find WLL via zero-config...
2020-03-20 09:57:42.045 ZeroConf Service: + 'weatherlinklive-711ad0' on en1
2020-03-20 09:57:42.045 ZeroConf Service: + 'weatherlinklive-711ad0' on en0
2020-03-20 09:57:42.045 ZeroConf Service: + 'weatherlinklive-711ad0' on lo0
2020-03-20 09:57:42.045 Host: weatherlinklive-711ad0 (192.168.1.183)
2020-03-20 09:57:42.045 Host: weatherlinklive-711ad0 (192.168.1.183)
2020-03-20 09:57:42.045 Host: weatherlinklive-711ad0 (192.168.1.183)
2020-03-20 09:57:42.045 WLL found, reporting its IP address as: 192.168.1.183
2020-03-20 09:57:42.045 WLL found, reporting its IP address as: 192.168.1.183
2020-03-20 09:57:42.045 WLL found, reporting its IP address as: 192.168.1.183
2020-03-20 09:57:42.454 WeatherLink API Current Response: OK: {}
2020-03-20 09:57:42.459 WeatherLink API Current: No data was returned. Check your Device Id.

Hope this helps
I have the log files at home if you need the full item

mcrossley · Post by **mcrossley** » Fri 20 Mar 2020 10:48 am

Hmm, OK. A step at a time...

Yes, the Station ID is just a simple numeric, helpfully it is not viewable by the user on wl.com.

It's strange how we are received three responses to the zero-config (mDNS) query. It shouldn't matter as all I do is extract the IP from the response, but it could be symptomatic of another underlying issue - I have no idea what. The mDNS protocol says it will send a single response to each request, and that is what I see on my network.

The API key/secret look OK, as you get a success response. BTW, there is no need to obfuscate the URL, the Davis API is better than most in that encrypts part of the URL using the API secret so that it cannot be replayed, nor can it be used to construct future URLs. Other web API implementations could learn from this!
Cumulus does obfuscate parts of the URLs in the logs where required for other web APIs so the logs can be freely posted (unlike the Cumulus.ini file), but maybe I will also obfusticate the API key just to be sure.

But the API is returning an empty JSON string - "{}"

If you look on wl.com for this device is the data current? And is the historic data all there?

Could you post the whole debug log for me please, there are other things I can check.

Vegit8 · Post by **Vegit8** » Fri 20 Mar 2020 11:03 am

Hi
Yes all data is up to date on the WL site

It wont let me post the entire log file the max chars is 60k and my file is 260k!

mcrossley · Post by **mcrossley** » Fri 20 Mar 2020 11:14 am

mcrossley · Post by **mcrossley** » Fri 20 Mar 2020 11:48 am

Thanks for the log file.

The good news, you are now getting the live broadcast data from the WLL.

Other news, the computer seems to have multiple interfaces connected to the same network? Which is why the mDNS is seeing multiple responses, it's the same response seen over the different network connections - en0, en1, lo0 - I don't know enough macOS to say what they correspond to, but a guess is it is connected to both the local LAN and Wifi?

The same is happening with the WLL broadcast data, each packet is being received twice. This is inefficient, and also means that the wind array will have duplicate entries in it, not a fatal condition, but the array is fixed size so in effect will only cover half the normal time interval.

The bad news, I cannot see an issue with the API, it is just not returning anything, which is normally indicative of an incorrect station Id as the API key and secret are working.

Could you do the following please. Stop Cumulus, edit Cumulus.ini, in the [WLL] section remove the Station Id.
ie...
WLStationId=

Restart Cumulus with the -debug option and send me the log file. The Station Id *should* repopulate, but I'd like to see the messages sent/received.

mcrossley · Post by **mcrossley** » Fri 20 Mar 2020 11:51 am

Cumulus.ini instructions clarified above!

Vegit8 · Post by **Vegit8** » Fri 20 Mar 2020 11:59 am

sadly, this will have to be another day.
The guy is a fairly famous market gardener, and is often on his machine doing webinars etc, so I'll need to find a slot!

Cumulus Support

Davis WLL - missing data on computer restart

Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart

Re: Davis WLL - missing data on computer restart