Dodgy data !

[adrian5750]

Hi Folks
I've been out of the forum for a while - setup here is w8681 and an elderly PC <g> - running a fairly old version of Cumulus... live pages at http://www.ambquality.co.uk/weather/

There's some duff data crept into the records (if you look under 'records' then you may see that temperatures in the region of -13c aren't common in the far south-west of Ireland, and, even though it does rain a bit from time to time, we don;t tend to get 4.44in/hr <g>.

So - given that this is an old version of Cumulus, what's the best way forwards to untangle this....

a) Upgrade, then sanitise the data (how? - is there a utility for doing this nowadays)
b) Leave all 'as is' in terms of Cumulus versions and then find some manual way of
sanitising the data ?

c Another answer ? <g>

I'm not blaming Cumulus, I'm sure it's the weather station that's fed it some duff data - but I'd welcome suggestions as to how to sort it.

Many thanks
Adrian

[steve]

I'd recommend you upgrade to 1.8.9 or 1.9.0, because that creates an 'audit trail' of all-time record changes, so it's fairly easy to look through it to correct your all-time records to what they were before something silly happened.

Regarding editing the all-time records, you can use Edit -> All time records, this assumes you know what the correct values should be.

Regarding editing the data logs around the times the glitches occured, the toolbox is good for this - http://wiki.sandaysoft.com/a/Toolbox

Editing is still a manual process, though.

A rain rate of 4.4 in/hr for a very brief period isn't unrealistic, by the way. I'm not saying it isn't wrong, but it could be right. Your 6.42 inches in an hour is probably suspect, though.

[adrian5750]

HI Steve
Many thanks for the quick reply - I'll take a look at the updated versions of Cumulus, and the toolbox..

Cumulus lives on a (slightly) remote PC - which isn't at all keen on letting me edit 'stuff' that's in the Windows folder from this machine... I don't suppose there's an easy way round this ? Both machines (local & remote) are running XP - just the remote one's got a rotten old CRT on it and nowhere to sit & type <g>.

Thanks again
Adrian

[nking]

There are a number of options to control remotely

Remote Desktop built into XP, I believe you will need the Pro version of XP http://www.microsoft.com/windowsxp/usin ... intro.mspx
Logmein which is free https://secure.logmein.com/UK/
Team Viewer which is my preferred choice http://www.teamviewer.com/index.aspx

Ther are others but some ideas for you

[adrian5750]

There are a number of options to control remotely

Remote Desktop built into XP, I believe you will need the Pro version of XP http://www.microsoft.com/windowsxp/usin ... intro.mspx
Logmein which is free https://secure.logmein.com/UK/
Team Viewer which is my preferred choice http://www.teamviewer.com/index.aspx

Ther are others but some ideas for you

Hi Nking

Thanks so much for the suggestions - have just installed the (free!) Team Viewer -
and I can now see the remote desktop on this PC - absolute magic!
Thanks for that....

Now off to download the latest copy of Cumulus...

You've no idea how much easier that makes things <g>

Thanks
Adrian

[adrian5750]

HI Folks

OK - have installed 1.8.9 - and all looks good.
Think I've also edited the dayfaile with the duff info in it -
at least the -13c temperature!

Is there any way to ask Cumulus to re-calculate the all-time
data - so that it picks up on the new corrected data ?

Alternatively, is there a utility that scans through the data files
and generates a new set of all-time figures that can be poured into
Cumulus's all-time data to sort things out ?

Thanks -
Adrian

[steve]

Is there any way to ask Cumulus to re-calculate the all-time
data - so that it picks up on the new corrected data ?

No; admittedly that would be useful, but it's just something I haven't had chance to do. It's one of the reasons for the alltime log, to make back-tracking easier. In theory, Cumulus 2 removes the need for this, as there is no separate all-time record data.

Alternatively, is there a utility that scans through the data files
and generates a new set of all-time figures that can be poured into
Cumulus's all-time data to sort things out ?

Yes, it's called Excel

Load dayfile.txt into a spreadsheet, and sort on the various columns, and you should be able to get most of your all-time records from that.

[Gina]

I've used Open Office spreadsheet to do the same - import and export csv.

[adrian5750]

HI Folks
Well - this has been a useful exercise.
Not only did I learn how to do 'virtual desktop' remote access on the PC that's linked to the weather station (which is a great asset) - I've also _nearly_ sorted out Cumulus's off max / min reports.

The one that's defeating me is the max windchill temperature - I can see where it's stored in alltime.ini - but I can't see it in the dayfile.txt - what am I missing ? It's showing a fairly unlikely -18.7 at the moment!

Thanks for all the help so far
Adrian

[steve]

The one that's defeating me is the max windchill temperature - I can see where it's stored in alltime.ini - but I can't see it in the dayfile.txt - what am I missing ?

It's not in there; this is one reason a 'regenerate all time records' facility isn't straightforward at the moment.

Wind chill is logged to the monthly log files, though. You would need to either merge all of your monthly files and then sort them, or do each one individually and find the max from your results. A slight complication is that wind chill wasn't logged to the monthly logs either, until some time early in 2009, so your earliest logs may not have it.

[adrian5750]

HI Steve

Before I could get round to sorting the windchill data, I seem to have had another hiccup....

All-time wind gust is now showing 101mph and all-time high rain rate is also 3.48in/hr - both about the same time on the same day.

Presubably I need to knobble the offending data in the dayfile.txt (which shouldn't be that difficult to spot <g>) -
but did you say that there's now an easy way of 'rolling back' the all-time data ?
Can you tell me how to do this, please ?

Thanks
Adrian

[steve]

but did you say that there's now an easy way of 'rolling back' the all-time data ?

It's not really easier to actually do, but at least the data is now available so you know what to roll back to. If you look at the alltimelog.txt file, it will show you which record changed, when, and from what to what. So you can retrieve your old, correct record and reinstate it. I guess at some point I could look at making it a bit more automatic.

http://wiki.sandaysoft.com/a/Alltimelog.txt

The all-time file itself is now a text 'ini' file, rather than the binary file it once was so you now have the option of editing the file directly (with Cumulus stopped) rather than using the built-in editor.

http://wiki.sandaysoft.com/a/Alltime.ini

[adrian5750]

HI Steve

Grand =- many thanks!
Being _really_ picky <g> - it'd be nice if the date / time formats were the same in both files - then
it'd be a simple copy/paste frmo one file to the other to restore the values....

...but hopefully it's not going to be necessary all that often....

Thanks again
Adrian

[steve]

Being _really_ picky <g> - it'd be nice if the date / time formats were the same in both files - then
it'd be a simple copy/paste frmo one file to the other to restore the values....

They don't need to be the same format, it works anyway - you can use the format from alltimelog.txt in alltime.ini.