Missing data

[malone]

I have a Vantage Pro 2 and am using it with both Cumulus (v1.8.8 (881)) and WeatherLink through VirtualVP. Data log interval is set to 10min for both Cumulus and WeatherLink. This configuration seems to be working OK except for one thing.

If I close down Cumulus for a period, say an hour, and then restart it, the data stored on the weather station is not downloaded to Cumulus. It's permanently lost to Cumulus. If I close down WeatherLink, I find that the weather data IS downloaded when I restart WeatherLink. Surely Cumulus, when restarted, should automatically download data stored on the weather station, or am I misunderstanding how Cumulus is intended to operate? I assumed that all data on the weather station later than the TimeStamp value in the today.ini file should be downloaded? It comes up with the "Please wait ..... Downloading data" message but nothing gets downloaded.

I thought it might be associated with VirtualVP somehow but having sent them various .log and .dat files they cannot suggest any reason why this is happening.

Anyone got any ideas? Perhaps I've got a setting wrong somewhere?

[steve]

There's a setting to prevent use of the logger, but it's unlikely you have that set. Could you zip up the diags folder and attach it, please?

[malone]

Thanks Steve, zipped folder is attached

Just to let you know:-

- this morning I have been turning Cumulus on and off a couple of times in order to try to resolve the issues, and data at the following times are missing - 09:20, 09:30, 11:20 even though they appear in WeatherLink
- this morning I upgraded to the v1.8.8
- we are GTM +13 here in New Zealand. This unusual time zone has presented problems with other programs in the past.

[steve]

First of all, I know that there is a bug somewhere which means that one less than the expected number of logger entries is received. I think this must be in the Davis DLL, because Cumulus appears to be doing the correct thing (the logs confirm this), and Weatherlink (which doesn't use the DLL) doesn't seem to suffer from the problem. It's always the most recent (rather than the earliest) entry which is missing, so it's hard to see what I can do about it. I have looked at the (confusing) documentation for the DLL and it looks like I am doing the correct thing. I'll try to find some time to do some more investigating of this. So, that's an explanation for a single entry going missing on some start ups.

But you also seem to have a problem where VirtualVP isn't returning all of the entries that should be available. Possibly this is something to do with your time zone. Unfortunately very few of your latest logs are for situations where there would have been any data to download anyway, so it's hard to find a pattern. Here's an example. At about 0926 on the 12th Jan, Cumulus asks for all of the logger entries since 0830:

2010-01-12 09:26:12 : VP2: Downloading history from 12/01/2010 8:20:00 a.m.
2010-01-12 09:26:17 : VP2: Number of history records = 1
2010-01-12 09:26:17 : Logger entry zero: Y = 2010, M = 1, D = 12, H = 8, M = 30
2010-01-12 09:26:17 : VP2: Earliest timestamp 12/01/2010 8:30:00 a.m.
2010-01-12 09:26:17 : VP2: Actual number of valid history records = 1
2010-01-12 09:26:17 : VP2: Setting raindaystart from todayfile = 20.3999996185303
2010-01-12 09:26:17 : VP2: Finding raintoday from logfile C:\Program Files\Cumulus\data\Jan10log.txt
2010-01-12 09:26:17 : VP2: Setting raintoday from logfile = 0
2010-01-12 09:26:17 : VP2: Setting raintotal = 20.3999996185303
2010-01-12 09:26:17 : VP2: processing logger data entry 0 for 12/01/2010 8:30:00 a.m.

VirtualVP only returns one entry, for 0830. It does look suspiciously like an 'off-by-one' on the time zone, but I don't know where the problem is; Cumulus uses the PC clock and appears to be doign the right thing. I assume VirtualVP and the station are set to exactly the same time and zone as the PC that Cumulus is running on? Could it be a daylight savings issue?

What is your VP2 firmware version?

Perhaps one way to get a better feel for the problem would be to stop Cumulus for two or three hours, then we can see if it is indeed getting one hour's less data than it should.

[malone]

Thanks Steve.

In answer to your questions:-
- I am using console version rev 1.6. I see the most recent version is 1.8. Perhaps I should upgrade?
- The (XP Pro) computer's time is set at GMT+12 with "Automatically adjust clock for daylight saving changes" enabled.
- VirtualVP doesn't have a time setting - presumably it is linked to the computer's time. But the times in the activity logs are all correct.
- The Vantage Pro station has its time set through WeatherLink and is set to GMT+12 with Daylight Saving set to autodetect.

- I turned off Cumulus for three hours between 15:03 and 18:12 yesterday 23 Jan. The data acquired by the weather station during the off period is lost. I have attached the new file created in the Diags folder "copy of cumulus1.009".
- Today I connected Cumulus directly to the weather station, not using VirtualVP at all, and then turned Cumulus off for another two hours between 8:15 and 10:15. In this case, the data acquired during the off period is downloaded to Cumulus after restart, except for the final 10:10 reading (which I assume is associated with the "Davis DLL bug" you refer to?). I have attached the new files created in the Diags folder "copy of cumulus2.009".

From this it seems that VirtualVP must be something to do with the problem. I'd welcome your views on these results before I go back to the VirtualVP support.

Are you aware of anyone using Cumulus with VirtualVP who knows for sure that all the weather data gets downloaded after a break?

[steve]

except for the final 10:10 reading (which I assume is associated with the "Davis DLL bug" you refer to?).

Yes, the last logger entry is always missing, for some reason.

From this it seems that VirtualVP must be something to do with the problem. I'd welcome your views on these results before I go back to the VirtualVP support.

It does look that way. There must be some logging we can switch on in VirtualVP that would show us what's going on.

Are you aware of anyone using Cumulus with VirtualVP who knows for sure that all the weather data gets downloaded after a break?

There are a number of people using Cumulus with VirtualVP, and I don't recall anyone else having this problem. I use VirtualVP from time to time and I don't think I've had the problem. I'll try to do some experimenting tomorrow.

[steve]

I've been experimenting with VirtualVP, and I seem to be able to download logger data succesfully. So I could see what VirtualVP was doing I clicked on View | Activity Log, and then selected Info and Verbose. I struggled a bit to get a copy of the output, but the best way seemed to be just to untick Info and verbose before doing the copy. This is what I got:

14:28:33:687 Info - Virtual Console5: Received Get Archive Data Since
14:28:33:687 Info - Virtual Console5: Sent response for Get Archive Data Since
14:28:33:906 Info - Virtual Console5: DMPAFT request for 2010/01/24 13:10 - found start
14:28:33:906 Info - Virtual Console5: Sending DMPAFT info, pages= 4, startRec= 2
14:28:33:906 Info - Virtual Console5: Sent response for Get Archive Data Since
14:28:33:968 Info - Virtual Console5: Sending archive page# 251, times: 2010/01/24 13:05, 2010/01/24 13:10, 2010/01/24 13:15, 2010/01/24 13:20, 2010/01/24 13:25
14:28:33:968 Info - Virtual Console5: Sent response for Get Archive Data Since
14:28:33:968 Info - Virtual Console5: Sending archive page# 252, times: 2010/01/24 13:30, 2010/01/24 13:35, 2010/01/24 13:40, 2010/01/24 13:45, 2010/01/24 13:50
14:28:33:968 Info - Virtual Console5: Sent response for Get Archive Data Since
14:28:33:968 Info - Virtual Console5: Sending archive page# 253, times: 2010/01/24 13:55, 2010/01/24 14:00, 2010/01/24 14:05, 2010/01/24 14:10, 2010/01/24 14:15
14:28:33:968 Info - Virtual Console5: Sent response for Get Archive Data Since
14:28:33:984 Info - Virtual Console5: Sending archive page# 254, times: 0000/00/00 00:00, 0000/00/00 00:00, 0000/00/00 00:00, 0000/00/00 00:00, 0000/00/00 00:00
14:28:33:984 Info - Virtual Console5: Sent response for Get Archive Data Since

Basically, Cumulus asked for the data since 13:10 and got what it asked for (the data is sent in full pages, that's why the 13:05 and 13:10 entries are also sent). Interestingly, the entries for 14:20 and 14:25 are missing, so this suggests that the 'last entry missing' problem is not actually a bug in the Davis DLL, but a bug in the station (and VVP?). I think I'll send this log to Steve Hatchett to see what he says.

Perhaps you could try getting the Activity Log so we can see what yours is doing?

[malone]

Thanks Steve. Yes, I'd actually had a look at that log several days ago, but hadn't been able to make much sense of it. I've attached a text file of the verbose information starting from immediately before I restarted Cumulus after a 20min break. It looks rather different from yours and, in particular, I can't find any "Received Get Archive Data".

[steve]

Do you have the Cumulus diags file to go with that?

[malone]

yes

[steve]

The VVP log suggests that the DLL didn't even ask it for the data. Could you get both logs again, please, but with a longer period (over an hour) between runs?

[malone]

Cumulus turned off 09:01
Cumulus turned on 10:04

[steve]

It's very odd. You might say it was a bug in the Davis DLL, but then why does it work with a direct connection, and why does it apparently work for everyone else (with or without VVP)?

VVP does seem to be logging an error at exactly the time that Cumulus is asking for the logger data:

10:04:07:671 Info - VpConsole: Sending Get EEPROM Data binary
10:04:08:000 Info - VpConsole: No response received for Get EEPROM Data binary

I don't know if that is relevant. I've been in touch with Steve Hatchett regarding the missing 'latest' entries at the end of the download and he's investigating a possible bug in VVP. But it's not clear whether that is anything to do with the problem you're seeing.

I'll add some more tracing to Cumulus to make absolutely sure that it's sending the correct command to the DLL.

[steve]

Actually, I think that EEPROM error is relevant. Here's the corresponding bit of trace from my machine:

21:28:42:562 Info - VpConsole: Sending Get EEPROM Data binary
21:28:42:578 Info - VpConsole: Received Get EEPROM Data binary response
21:28:42:578 Info - Virtual Console5: Sent response for Get EEPROM Data binary

Shortly after that the DLL sends the DMPAFT command to fetch the archive data. I think the DLL needs the EEPROM data before it can send the DMPAFT command - the EEPROM data contains the console archive period and various other details.

[malone]

I don't know whether this will shed any light on the problem, but I turned off Cumulus, I then turned off Weatherlink for a few minutes then turned it on again and monitored what VVP was doing (VirtualVP log - WL start.txt). Stored data is normally only downloaded on the hour, so I then pressed Ctrl - L to download values since the beginning of the hour and monitored it again (VirtualVP - WL download.txt). Everything goes smoothly. I wonder if details of the log files give any clues?