PWS catchup

[Tau Bootis]

I've installed 1.9.2 beta and setup the WOW site OK,
I then decided to overwrite my data files to induce the catch up facility, but its scrambled the data for my PWS site: http://www.pwsweather.com/obs/AUDWXDATA.html#

It maybe because I accidentally set the main Cumulus update interval to 1 minute instead of 15; I also noticed PWS was being updated every minute even though that setting was still at 15 minutes! So not sure if there’s a bug or its just my fault for setting the interval to 1 minute, it seems to be back to normal now.
Wonderground had no side effects and WOW has now got some backdated data in there WOW!
http://wow.metoffice.gov.uk/graphdata?r ... eID=214622

Great work Steve

[steve]

The 'catch up' works by uploading the data from each entry from the logger that Cumulus downloads at start up. So, if you were getting PWS data at 1-minute intervals from catch up, you must have your station logger set to 1-minute intervals.

[Tau Bootis]

The station logger is set to 5 minute intervals, it was when the Web Settings interval was set to 1 minute and all other upload intervals were set to 15 minutes.
Wonderground remained at 5 min intervals during the episode.
Cheers

[steve]

The station logger is set to 5 minute intervals, it was when the Web Settings interval was set to 1 minute and all other upload intervals were set to 15 minutes.

Yes, indeed, the Fine Offset has a minimum interval of 5 minutes! Perhaps the bug is that I'm using the wrong timer for PWS uploads - the web update timer, most likely. I'll investigate.

[steve]

I've looked at the code and I can't see anything obviously wrong. I can't reproduce the problem; I tried setting my web update interval to 1 minute, and it didn't affect my PWS uploads. My 'catch up' uploads are correctly timestamped. So I'm afraid I don't know where those 1-minute updates came from.

[Tau Bootis]

Just a quick one as I am going out, I did another catchup and the same thing happend again, if its any help this is what I did just in case I'm doing something wrong;
I copied all the files from a backup folder from 3 days ago and over wrote the existing ones in the Data folder then restarted Cumulus, while the catch up was in opperation I did a manual upload.
It seems to be the temperature that affected.

Cheers
Mark

[steve]

By 'rewinding' Cumulus like that, you are effectively sending data to PWS Weather (and the other sites) for periods for which you have already sent data. The code in Cumulus for PWS, WU, and WOW is virtually identical, so if WU and WOW are OK, then I think the problem must be at PWS; it looks like it can't cope with getting repeated data - it looks like it's timestamping the new data incorrectly, perhaps ignoring the timestamp being sent and using the current time instead. I believe WU used to do this, which is why I hadn't previously implemented a 'catch up' facility; when I tested it a long time ago, the 'historical' uploads were all getting timestamped with the current time.

A 'normal' catch up - where it's just uploading data from the logger for the period where it wasn't running - appears to work OK, which does tend to confirm that the problem is not in Cumulus. If you look in the appropriate diags file you'll be able to see exactly what Cumulus is sending, including the timestamps, so you can check the timestamps and compare the data with what appears at PWS to see what PWS is doing with it.

[Tau Bootis]

I have had a look at the rather large Diags file and I think you are right, PWS can not handle it; I’m guessing a bit here, but it appears that as all the old data is being uploaded again, once a minute it grabs a line that is currently being uploaded and adds it until the upload finishes, this can be seen in the pressure trend data below.

If I need to do this again I can just disable the catch-up.

Thanks for your time Steve.