2 lines, same date, bogus data in dayfile file...

[Intheswamp]

I'm a newbie and probably just made a newbie mistake, but need to know what to do and maybe "what happened".

Here's my setup:
Davis VP2 (wireless) running v3.12 firmware
Davis USB datalogger
Old Dell D610 laptop (wifi connection)
Windows XP Pro SP3
WeatherLink v6.0.3
Cumulus v1.94 beta Build 1079
Wunderground v1.15

Ok, last night I was trying to get StartWatch working. I rebooted both WeatherLink and Cumulus a time or three and thought I had everything "OK". Then I looked at the rainfall section of the Cumulus screen and saw that we had .73" rain for the last 24 hours (correct) but also that the monthly rainfall showed 1.46" (wrong).

Don't get confused that the 1.46 is 2x.73...it just ended up that way but made me pull my hair out for a while trying to figure it out. I was piddling around with all of this around midnight (roll-over or whatever is set for midnight). Anyhow, I dug around some and find that I have two entries for 02/06/13 with a time for what I believe is the bogus data line being 00:03...but dated the same as the previous entry...02/06/13. The rainfall amount entered on that bogus line is exactly the amount of the error.

I also see on the "Recent Extremes" section of the main screen that most of "yesterday's" data is dated at 00:03 and much of that is wrong...high wind gust shows as 1mph whereas I know it was 24mph, max temp at 71F (I wish!) ....etc.,.

Should I simply delete that line of the dayfile? Is there another data file that I need to give attention to? Did I simply mess up by messing with things during the roll over event?

Thanks,
Ed

[steve]

Are you running Weatherlink and Cumulus at the same time? Using Virtual VP, presumably?

It's possible that restarting a number of times around midnight caused the problem; it's not recommended to restart Cumulus around midnight, the code tries to cater for many different situations, but it's not too hard to confuse it if you try. Please zip up the diags folder and attach it.

[Intheswamp]

Thanks for the fast response, Steve!

Ok, I hope that I did this correctly...I had to go back and read the "Read This First" message to figure out the "diags" part....it's all good learning! If the zipfile didn't work I'll try it again.

Yes, I am running WeatherLink and Cumulus via VirtualVP on Davis firmware v3.12. The only uploading that I've done so far is from Wlink to WU. That has worked well since the initial installation a few days ago. Only last night while tinkering with StartWatch and restarting the wx programs a few times did I run into this problem. I think it has to do with me monkeying with things during the roll over time.

Thanks again,
Ed

[steve]

You started Cumulus up at 3 minutes past midnight, and it was unable to get any logger data from Virtual VP (there are known issues with Virtual VP and the Davis DLL which Cumulus uses). So it went ahead and did the midnight rollover. You then closed it down before it had written the today.ini file; normally it would do this when it writes the data away, but VVP didn't give it any. So the next time it started up it thought it had to do the rollover again.

I did a change in 1.9.4 which I think may have prevented the spurious rollover occurring.

Note that you are still hitting a problem getting logger data from VVP. The number of logger entries that it says are available for the period requested is clearly nonsense, and when it provides the logger data, the timestamp is apparently garbage anyway:

6/3/2013 09:51:46.234 : VP2: Downloading history from 6/3/2013 9:50:00 AM
6/3/2013 09:51:46.234 : y=2013 m=6 d=3 h=9 m=50
6/3/2013 09:51:51.187 : VP2: DownloadData_V = 339
6/3/2013 09:51:51.187 : VP2: GetNumberOfArchiveRecords_V = 339
6/3/2013 09:51:51.187 : VP2: Number of history records = 339
6/3/2013 09:51:51.187 : Error in earliest timestamp, unable to process logger data

[steve]

I realised that I didn't answer your question about correcting it. Just remove that second entry. You might also find that some of the daily records have been erroneously broken; if so, the easiest fix is to retrieve copies of the corresponding ini file(s) from a backup folder (e.g. alltime.ini is the all-time records file).

[Intheswamp]

Thanks for the reply, Steve. I'm still pretty much in the dark but at least I know how to look at my diags now.

The data that I had been seeing on the Cumulus screen has been paralleling what I've been seeing on the WeatherLink screen...seems both were recording the same data. Until last night when I installed StartWatch (now uninstalled for the time being) both *appeared* to be working fine via VVP...which surprised me as I had read reports that the Davis v3.12 did not work well with VVP. It all looked good until the midnight roll over last night and me tinkiering with StartWatch about that time (including the restarting and closing of Cumulus). Just seems odd that I did not detect a problem with the screen data until after that point...the "maximums" and "minimums", etc., seemed to all be reporting good info until then.

I see where it notes the error in the timestamp (I figure that is bad). I take it that 339 records are *way* too many to have in 1 minute and 51 seconds...is that the length of time that the file is saying those records come from?...from 9:50:50 to 9:51:51?

It looks like my history stored in Cumulus is compromised. If I uninstall and reinstall Cumulus and run Cumulus without VVP in the way will that give me a fresh start where Cumulus will download all the data stored in the logger making my historical data "good"? I could remove VVP from the setup and use WeatherLink for just console maintenance or whatever and use Cumulus for my internet reporting and for my wx computer display screen.

Thanks again,
Ed

[steve]

I see where it notes the error in the timestamp (I figure that is bad). I take it that 339 records are *way* too many to have in 1 minute and 51 seconds...is that the length of time that the file is saying those records come from?...from 9:50:50 to 9:51:51?

Yes, that's right.

If I uninstall and reinstall Cumulus and run Cumulus without VVP in the way will that give me a fresh start where Cumulus will download all the data stored in the logger making my historical data "good"?

No, because it will retain the data. And if you delete the data it will only start from the point where you next run it, without intervention. See http://wiki.sandaysoft.com/a/FAQ#I.E2.8 ... er_station

I could remove VVP from the setup and use WeatherLink for just console maintenance or whatever and use Cumulus for my internet reporting and for my wx computer display screen.

Some people are using Cumulus with VVP and will tell you that they never have a problem. Perhaps they don't need the logger data. If you intend to run Cumulus 24/7 then it's possible you won't have a problem with VVP either.

[Intheswamp]

Steve, thanks for bearing with this newbie.

In regards to using a backup file there appears to be several subfolders in the backup folder. And within those, more folders. I'm a bit intimidated on figuring out which one I should use. Oldest? Newest? ?? Do I simply overwrite the current files? I'm sorry to be such a bother.

Ed

[steve]

A backup is created each time Cumulus starts up. Those are the folders in the backup folder. In addition, there's a daily backup taken just after midnight; those are the folders in the backup\daily folder. Use the latest files from before the problem occurred. Either pick and choose individual files, or copy the whole set into the data folder, which effectively 'rewinds' you to the state at that point, so the data since then will be downloaded from the logger to catch up to the present time. That's assuming that whatever Cumulus is talking to will actually let it have the logger data it needs...

[Intheswamp]

I see where it notes the error in the timestamp (I figure that is bad). I take it that 339 records are *way* too many to have in 1 minute and 51 seconds...is that the length of time that the file is saying those records come from?...from 9:50:50 to 9:51:51?

Yes, that's right.

Hey! I got something right!

If I uninstall and reinstall Cumulus and run Cumulus without VVP in the way will that give me a fresh start where Cumulus will download all the data stored in the logger making my historical data "good"?

No, because it will retain the data. And if you delete the data it will only start from the point where you next run it, without intervention. See http://wiki.sandaysoft.com/a/FAQ#I.E2.8 ... er_station

I apologize that you have to educate me on this. I thought that the data logger would retain "so many days" of data. If I completely remove Cumulus from my system and reinstall it it will only start storing the current data streaming into it and not access the "logged" data within the data logger? Is this correct? When I start up Cumulus a "download" window opens and my thinking has been that it was downloading data from the logger...which I think is correct. What I'm understanding here is that a new installation of Cumulus will not access the logged/archived data on the logger but rather will create a starting point from the initial live stream of data it receives and then will download any subsequent data from the logger?

Hypothetical scenario here... I have a console and logger that have been operating for five days while connected to an unpowered computer. After the five days I power up the computer, install Cumulus and run it for the first time. From what I *think* I'm understanding, Cumulus will not download the archived data stored in the logger but will instead create a starting point at the time of it's initial connection to the logger... maybe from the first feed of live data(?). After that starting point is created Cumulus will then continue to download logged data and "catch up" if the program is shutdown or otherwise interrupted. But...it will not do a mass download of data to start with.

I'm *really* having a problem wording what I'm trying to understand but I'm really trying to understand this.

I could remove VVP from the setup and use WeatherLink for just console maintenance or whatever and use Cumulus for my internet reporting and for my wx computer display screen.

Some people are using Cumulus with VVP and will tell you that they never have a problem. Perhaps they don't need the logger data. If you intend to run Cumulus 24/7 then it's possible you won't have a problem with VVP either.

[/quote]

I'm planning on running whichever software I decide on 24/7 as it will be uploading data to the usual online reporting sites. BUT...I know that I will be shutting the program down from time to time and that "things" happen where the station will be offline and need to catch up when it goes back online so I need the logger data. Looks like I need to figure out if I want to run multiple programs or not.

Ed

[steve]

I thought that the data logger would retain "so many days" of data. If I completely remove Cumulus from my system and reinstall it it will only start storing the current data streaming into it and not access the "logged" data within the data logger? Is this correct? When I start up Cumulus a "download" window opens and my thinking has been that it was downloading data from the logger...which I think is correct. What I'm understanding here is that a new installation of Cumulus will not access the logged/archived data on the logger but rather will create a starting point from the initial live stream of data it receives and then will download any subsequent data from the logger?

When Cumulus has been stopped for while, when it starts up again, it downloads the data from the logger to catch up from when it was stopped to the current time.

A completely new installation deliberately doesn't download the logger data when it starts up. This is an insurance against things going wrong for whatever reason. It's a simple matter to create a dummy today.ini file as described in the FAQ to override this behaviour. However, this is not now an issue for you anyway, as you have already been running Cumulus. You can use the last set of backup data created just before the problem, to recover.

Hypothetical scenario here... I have a console and logger that have been operating for five days while connected to an unpowered computer. After the five days I power up the computer, install Cumulus and run it for the first time. From what I *think* I'm understanding, Cumulus will not download the archived data stored in the logger but will instead create a starting point at the time of it's initial connection to the logger... maybe from the first feed of live data(?). After that starting point is created Cumulus will then continue to download logged data and "catch up" if the program is shutdown or otherwise interrupted. But...it will not do a mass download of data to start with.

That's correct, unless you override as described previously. So the worst case is that you 'lose' your first five days of data from all of the years that you will presumably be operating your station.

I'm planning on running whichever software I decide on 24/7 as it will be uploading data to the usual online reporting sites. BUT...I know that I will be shutting the program down from time to time and that "things" happen where the station will be offline and need to catch up when it goes back online so I need the logger data. Looks like I need to figure out if I want to run multiple programs or not.

Cumulus works perfectly well downloading logger data when talking to a real station directly. There are issues with Virtual VP which mean that the logger download can be problematic; I hit this problem when I bought VVP some years ago, and you have clearly hit the problem yourself. Some people are using VVP and apparently don't hit the problem. From my discussion with the VVP author about this several years ago, it seems that the problem is timing related in some way. I understand that with later versions of console firmware, there are other issues with VVP which affect a wider range of software.

[Intheswamp]

Thanks very much for all of your effort and for not giving up on me! Your last post was gold to me as it has cleared a good bit of confusion up for me and I'm perpetually confused so that's saying a lot! It seems like VVP was wishful thinking for me. I will go in and see if I can tell the best backup file to use and overwrite the bad data. If I fail I can then go back and do the reinstall. I may have a few more questions.

You've been tons of help and your patience has been great!
Thanks!
Ed

[Intheswamp]

YES YES YES!!!!!! SUCCESS!!!!!!!

I decided to give replacing the data files a shot. First thing I did was create a folder in the data folder and copied all the files found there into the new folder. Then I looked in the backup folder and there were several files in there, all with dates in their names of June 3rd which was a bad day. I then went into the "Daily" folder and found other folders with dates incorporated in their names. I went into the one that preceded the time the problem happened and found older copies of data files. There were a two or three files in the data folder that were not in the backup folder. I then copied and pasted the files in the backup folder over the files in the data folder. This left those two or three files (mostly log files) in the data folder unchanged. I held my breath and restarted Cumulus. It started up, but...the download window didn't open, the weather screen started up quickly, no apparent history data, it looked like it's "start date" was when I just started up Cumulus. My face was shortlived, though!

Since I had backed up all the files in the data folder previously I decided to delete everything in the data folder and then paste what files were in the chosen backup folder into the data folder. My weak mind was hoping that Cumulus was reading the old, bogus log files and thus not downloading the data from the logger. My feeble mind was telling me that Cumulus would create it's own log files if it didn't not find any in the folder...and it would have to create them from "new" data. I did the delete, copy, paste sequence and restarted Cumulus. The download window opens...and suddenly I'm seeing the last several days of weather life pass before my eyes!!! I looked at my All Time Records and they look about right. It worked slicker'n owl...poop! \:D/ \:D/

IT WORKED!!!!

I appreciate you sticking with me through this adventure(?), Steve. You definitely are very supportive of Cumulus!!!

Now to figure out how to seamlessly(?) convert over from WeatherLink to Cumulus for my WU uploads...

Ed

[steve]

Now to figure out how to seamlessly(?) convert over from WeatherLink to Cumulus for my WU uploads...

Turn off WU uploads in WL, and turn them on in Cumulus. Or am I missing something?

[Intheswamp]

Got it done. I was more concerned with gaps in the data on WU but that looks like it's an inevitable. It's working fine, even tinkering with the Rapid Fire feature. Now to move on to CWOP and ???? ...and eventually my own website.

Cumulus seems like a very good program, Steve, you're doing a great job!

Ed