Simple Data Repair

[Charlie]

Needed. (sorry if you were hoping for a solution)

I'm convinced the FO units will always find a way to mess up - I think that bad packets get sent from the remote sensor array occasionally (electrical noise or whatever) and there is no detection or correction in the display unit. So once again, somewhere in the middle of the night, I hit 44.3 degrees C with 160 km wind gusts and a rain fall of 1.1794E5 mm/hr instantly accumulating 9.83 METERS of rain. Quite surprising for March in Canada, where negative temperatures and snow are more the norm.

So I'm resolved to the fact that once or twice a week I'll need to fix a bit of bad data.

Are there any thoughts about simpifying the file structure is some future release so that when this occurs we have fewer places to correct the error? Right now we need to tweak the month file, the day file, the alltime records, at least one and maybe two .ini files. Toolbox helps immensely with the readability, but if the records were in only one place, this would be a trivial exercise.

Or maybe we need a routine (manually run or automatically run) to detect wildly different values, suggest a replacement value, and if acceptable, update all the appropriate files.

I'm just exploring thoughts to see if there are any bright ideas out there before trying to write a "toolbox 2.0" or something. Thoughts?

[steve]

This was one of the reasons for the SQL database format in Cumulus 2, where the data is recorded just the once and everything is worked out from that by SQL query. Although some people didn't seem to see the benefits of this and were asking for the plain text format to be continued

I will get back to Cumulus 2 at some point, until then I think you're stuck with the way it is, unless somebody clever like David comes up with something to make it easier. When I designed (perhaps that's not the right word) the data file structure, I hadn't considered the possibility of supporting a weather station that regularly decided to do its own thing regardless of the actual conditions.

[Gina]

I've been thinking about this myself. I've rarely had a catastrophic error but do occasionally get a glitch lasting one sample from the external sensors. Cumulus already has a mechanism to correct this, with adjustable sensitivity.

[Charlie]

No rush on Cumulus 2, and certainly no negative comments from me on Cumulus 1! It's as close to ideal as I've ever seen, and you've done a fantastic job on it. While a database is the right answer, a data file fixer will be good enough for now. I might take a crack at writing one if nobody else has started yet. My code is not in the same league as yours or David's, but this is a simple problem calling for brute force and ignorance - describing my skills precicesly

Still, if someone else has something nearly finished, I could certainly use my time elsewhere.

Gina, tell us more about what you're thinking!

[Gina]

Single sample glitches I'm proposing to correct by taking 3 or more probably 5 samples, averaging the outer 4 (ie. samples 1,2,4,5) and if the middle one exceeds a certain difference from the average, to change it to the average. Having looked into ways of doing this, I've concluded that this is best done on the raw data rather than later. But we'll see.

Regarding Cumulus data, I expect to be writing code to convert the Cumulus data to the same format as I'm using in my software (based on the pywws Python coded modules for Linux). As an offshoot of this I may write software to aid in correcting errors in the Cumulus data itself.. This will also be written in Python and as Python is available on the Windows platform, it should be useful to Cumulus users.

Edit... My own reasons for doing this is that I have over 9 months of Cumulus data acquired before I started using the Linux software. My current data starts on 3rd January of this year whereas the Cumulus data dates back to February last year.

[daj]

I've been experimenting with offering Data Fixing via 'Toolbox' -- initially rainfall correction. It's difficult as there are a number of issues that can go wrong and they all need their own way of working with the data. I am sure I will get there!

The first step was the Editor facility in 'Toolbox', next a search and replace I think

[steve]

It's difficult as there are a number of issues that can go wrong

Yes, this is a big problem, particularly with the rainfall. It's suddenly jumped up - but wait, it might go back down again. If it doesn't, what value do you use? It's gone down - has it been reset? Will it go back up again? In many ways, it's easier to deal with retrospectively, when you can look at the sequence of values and make a better judgement.

[Gina]

I haven't looked at the Toolbox as I'm not currently running Windows so can only put my ideas on how I personally would go about writing a data editor that was as user friendly as I could manage.

My thought is to display the data in graphical form. With this form of data display, the error areas should be easier to see. Data correction would be by dragging the graph to a better looking level. Once corrected the data could be saved back. A backup file could be used as a "safety net". Alternatively, an error correcting algorithm could be applied to the data in the graph and the user could see if this worked.

I have not yet looked into coding this and don't know how difficult it would be but my thinking is that - if you can imagine it, it should be possible!

TBH I have not as yet done any GUI coding in Python though I've done plenty in other languages. So far it seems to me that Python can do anything other languages can do (that I've used) except for speed, which in this case, is not critical.

[daj]

I haven't looked at the Toolbox as I'm not currently running Windows

LOL -- me neither......well sort of. I run Windows for Cumulus and for development for the masses. Mac as my laptop and Ubuntu for my dekstop PC (from where this message comes to you now!)