Reindex Alltime Records?

[mcpaton]

Is it possible to get Cumulus to parse back through all the old monthly history files and recalculate the Alltime records? I have had some bad data come through (not sure what happens with my station somtimes) that has removed legitimate alltime records, and put in stupid amounts like 2000mm of rain in an hour etc.

I wouldnt mind cleaning up with the real data again?

Cheers

[steve]

Not currently. It wouldn't come up with exactly the same figures, but admittedly it would be better than nothing. If you still have a backup (in the Cumulus backups folder) from before the problem, you could restore the alltime.rec file from there. Other than that, I guess you would have to emulate what Cumulus would do if it had the facility - merge all of your monthly files, load them into Excel (etc) and sort on the various columns.

[steve]

As a side note - something I would like to implement when I get around to it, is a 'change log' for the alltime records, so you would know which record changed from what, to what, and when. It would still be a manual process to reinstate an old record, but it would be much easier.

[daj]

Hi

I have a little tool that will run through and give you all the necessary figures -- it's not quite ready for public release but will be soon. If you zip up your log files I could run them through it.

As Steve notes above, the figures will differ slightly as the log files are storing data at the logging interval (10 minutes in my case) so if there was a higher figure during that 10 minutes it would not be logged; however the data will be close to the truth

[TNETWeather]

As a side note - something I would like to implement when I get around to it, is a 'change log' for the alltime records, so you would know which record changed from what, to what, and when. It would still be a manual process to reinstate an old record, but it would be much easier.

WD has something like that. It is very helpful since you have a log of when something hit a target as it happened with history of the previous target.

[steve]

It is very helpful since you have a log of when something hit a target as it happened with history of the previous target.

Any suggestions for a good format, to be human and machine readable? I was thinking just a space-separated list of date/time/item/old/new. For example, as it happens I had a new low temperature overnight of -7.6C, so the entry in the log would look like this:

24/02/10 05:19 "Lowest temperature" -6.5 -7.6

[nitrx]

24/02/10 05:19 "Lowest temperature" -6.5 -7.6

I should prefer the time and date of the 'old' record too
someting like this
"Lowest temperature" record 23/02/10 23:11 -6,5 | 24/02/10 05:19 -6.7

With the old data it's easier to locate where to change the wrong data the '|' is a random separator the word(s) 'changed to' are more human

[steve]

I should prefer the time and date of the 'old' record too
someting like this
"Lowest temperature" record 23/02/10 23:11 -6,5 | 24/02/10 05:19 -6.7

Yes, I suppose having the old date and time as well makes it easier for a human to correct. But the 'pipe' separator is superfluous and makes it harder for a machine to parse. The word 'record' is also superfluous.

[TNETWeather]

WD uses this format:

Code: Select all

10:26:07 AM  02/23/10   All time high baro month value 1026.1
10:27:06 AM  02/23/10   All time high baro month value 1026.2
10:33:09 AM  02/23/10   All time high baro month value 1026.2
11:10:11 AM  02/23/10   All time high baro month value 1026.2
11:16:09 AM  02/23/10   All time high baro month value 1026.3
11:18:07 AM  02/23/10   All time high baro month value 1026.3
11:19:46 AM  02/23/10   All time high baro month value 1026.4

I would have preferred ISO dates with 24 times:

Code: Select all

2010-02-23 10:26:07 All time high baro month value 1026.1
2010-02-23 10:27:06 All time high baro month value 1026.2
2010-02-23 10:33:09 All time high baro month value 1026.2
2010-02-23 11:10:11 All time high baro month value 1026.2
2010-02-23 11:16:09 All time high baro month value 1026.3
2010-02-23 11:18:07 All time high baro month value 1026.3
2010-02-23 11:19:46 All time high baro month value 1026.4

You can quickly parse the previous record by simply searching for the setting, but that is coming from a Unix background.

fgrep "All time high baro month" recordsset.txt

[nitrx]

You're right I really can't find another word for record also the date notation is confusing

[steve]

I would have preferred ISO dates with 24 times

Yes - I was thinking that as Cumulus already uses local date format (well, even worse, a mixture) I should carry on the same. But there's no reason why this file shouldn't use ISO dates. I don't think the seconds are necessary, and I think it is a good idea to include the old value and date/time, so I propose: date/time/value/item/old date/old time/old value so the item name nicely separates the new and old values.

Example:

2010-02-24 05:19 -7.6 "Lowest temperature" 2009-02-09 04:50 -6.5

[nitrx]

2010-02-24 05:19 -7.6 "Lowest temperature" 2009-02-09 04:50 -6.5

this is readable for me

[beteljuice]

Is "Lowest Temperature" fixed, or will it become a strings.ini setting ?

[steve]

Is "Lowest Temperature" fixed, or will it become a strings.ini setting ?

I suppose I could make the all-time record descriptions into strings.ini settings while I was doing it.

[steve]

Is "Lowest Temperature" fixed, or will it become a strings.ini setting ?

Were you aksing this because you thought it ought to be, or did you just want to know either way? I'm now thinking that it will be fixed, in English, and I won't do anything to strings.ini at the moment (and even if I do at some point, the records log will still use the English). OK?