Best way to clean up bad "high/low" records spikes?

[scott_thewspot]

Greetings all,

I've been running a Fine Offset for about a year now and have only just got around to configuring the "spike removal" settings. However now I want to clean up the old "High/Low" data records and I'm wondering what the best way to do that is.

Do I need to go into each of the "All Time", "Monthly", "This year" records and remove the bad values manually? Is there a better way to then "reset" the values or do I have to then work out the next best "proper" value and put that in manually?

Any suggestions or tips most welcome.
Scott.

[PaulMy]

Would using [Edit] [All time records] [All time records editor] be helpful for you?

Paul

[scott_thewspot]

Would using [Edit] [All time records] [All time records editor] be helpful for you?

Paul

No, not that I can tell. That will let me edit the value, but how do I work out what to change it to?

The only way that would seem to work would be to go through all dayfile/monthly logs and remove anything that looks like a spike, then try fetching the data again, seeing if a new spike appears that I missed and repeating until things look right. So... kinda hoping someone may have a better suggestion.

[steve]

I can't think of an easier way than to get your dayfile.txt 'correct' and then use the editors to fetch the values from that. I still have some bad monthly temperature records myself from the last time my sensor failed that I still haven't got around to fixing.

If you load a copy of dayfile.txt into a spreadsheet, you can sort on the offending columns to see where the invalid entries are.

[uncle_bob]

If you find the field names for the dayfile, then importing it into excel and doing a custom sort could bring the highs to the top and lows to the bottom, edit those out and then resave the file as csv"dayfile.txt". That should work.

[steve]

and then resave the file as csv"dayfile.txt". That should work.

Be very careful if you use Excel to do the edits that it doesn't change the date format or change the data formatting in any other way.

[scott_thewspot]

Are the high/lows only taken from the dayfile, or does it use the monthlog files as well?

Steps I've found work so far:

1. Find the bad line in the monthlog file that matches the dodgy record and delete the whole line. So far I've found when I get bad records the rest of the values in that line are bad as well. If there is just one bad value then I'll zero it
2. Edit the dayfile.txt and delete the whole line for the day that was recorded with that bad high/low record
3. Use Cumulus dayfile editor to "create missing" and re-create "good" high/lows for that day again
4. Go back into the alltime and monthly records and fetch log and dayfile data and "copy" good values, or identify more bad ones and repeat the process

This is slow and tedious... and prone to error if proper attention isn't paid.

I was going to try an Excel approach next, but just thought it worth asking in case there was a spike cleanup tool I hadn't managed to find or notice.

[steve]

Are the high/lows only taken from the dayfile, or does it use the monthlog files as well?

The editors allow you to use either - but you've already seen that, from your "step 4"?

[scott_thewspot]

Yes, I was cleaning the data in both daily and month logs so it wouldn't pop up again at some later time.

When Cumulus calculates the high/low though, is the day or month log the main source it uses?

[steve]

When Cumulus calculates the high/low though, is the day or month log the main source it uses?

Neither, it uses the current data readings. The highs and lows are logged to separate ".ini" files, see http://wiki.sandaysoft.com/a/Category:Log_Files

[scott_thewspot]

Thanks again Steve.

I had assumed everything was written to the log files first, then the INI files were updated from those values... That explains why I sometimes see bad records that seem to match the day file and some that match the month log.

[THX]

I would love to see a feature whereby we can revert to the last known good value to quickly fix a spike in temperature/wind/etc. Then have it somehow fix the erroneous max recorded figure.

I too find myself editing the .txt files in Excel and removing the spurious readings that my Vantage Vue sometimes creates. Sometimes this doesn't work so I will have to manually edit the .ini files via the Cumulus menu.

I don't know how much work this would be but it seems like a lot of us are having problems with our weather stations ruining our monthly and/or all-time highs. Having an "easy button" to fix that would be just awesome.

Keep up the great work with Cumulus!

[steve]

Lots of people have problems if they have Fine Offset stations, and the 'spike removal' settings in Cumulus are there to stop the problem happening in the first place as far as possible, which is much easier than cleaning up afterwards. These settings only apply to Fine Offset and La Crosse stations, as these are the two 'cheap' station types which typically have this kind of problem as a matter of course.

The code doesn't apply to Davis stations as these are typically reliable in normal use; as far as I'm aware, any bad data is most often caused when the sensors fail, which is relatively rare. The only times I have had false readings from my VP2 is when the anemometer and temperature sensors died.

What kind of problems are you having with your Vue, and how often do they occur?

I think I've done pretty much all I can regarding correcting errors once they've occurred. Pretty much all of the high/low data can be recreated semi-automatically from the log file data, so there is only one place that corrections have to be made. In addition, backups of all of the files are taken every time Cumulus starts up, and also at midnight, so often the easiest way to correct the high/low data is to stop Cumulus and retrieve the 'good' files from the appropriate backup. But I can perhaps have a look to see if there's more that I can do in this area at some future point.

[scott_thewspot]

For the most part it isn't an issue and the built in spike removal seems to handle most of it, but when they do get through its a pain to clean particular if they don't get noticed for days or weeks.

I was thinking that a separate clean up editor might be easier to have than trying to write more control into the main program. I guess a bulk data editor that has this particular type of cleanup in mind. As stated before, something like excel can't be used as it can corrupt the format, but the built in editor seems more focused on single record editing.

I had completely forgotten about all this, but I think I might look into creating something myself as well.

[uncle_bob]

Hi Steve,
I think it would be handy to have a sort function in the built-in editors, then the risk of Excel screwing up the data schema would be eliminated.

Just throwing the idea out there

Cheers Bob