Extreme Problems...

[RichK]

Hi. I have had the odd erroneous reading get into the system, and this wrecks the graphs and records.

For example, today, it erroneously recorded a 376mph wind gust. I cant find the entry in the month record, which seems to only record the data at 2minute intervals.

However, the data is stored somewhere. Where do I go to eliminate it? (I have also had this happen on temperatures - one SNAFU reading, and the system goes haywire).

Is there any chance in future versions to pass some commonsense over the incoming data, and reject records if they look daft;

eg. in an 8 second gap, temperature can only change by at most 10C, gust windspeed can only go to a max of 150 or (avge windspeed * 3), whichever is greater. (maybe set a configurable upper limit - Cheshire isnt known for regularly passing hurricane eye-walls).

This would stop some of the mega-weird readings getting saved as kosher data.

Any help?
Cheers, RichK

[steve]

For example, today, it erroneously recorded a 376mph wind gust. I cant find the entry in the dayfile, which seems to only record the data at 2minute intervals.

Entries in dayfile.txt are logged daily; Do you mean the monthly log file (the current one is called Aug10log.txt on an English system)? That logs at the interval you configure yourself.

However, the data is stored somewhere. Where do I go to eliminate it? (I have also had this happen on temperatures - one SNAFU reading, and the system goes haywire).

It depends where the erroneous reading is. If it's today's high, then stop Cumulus and edit today.ini.

Is there any chance in future versions to pass some commonsense over the incoming data, and reject records if they look daft;

Cumulus already does quite a bit of 'static' filtering of garbage, for those systems that have a tendency to make things up from time to time. I hope to put some more 'dynamic' filtering into Cumulus 2. There is already a 'maximum temperature change' setting in Cumulus 1; see the 'EWtempdiff' setting in cumulus.ini: http://wiki.sandaysoft.com/a/Cumulus.ini

How is your station connected to your PC? To a real serial port, or via a serial/USB converter?

[RichK]

Hi. I literally cannot find any of the data files that contain the erroneous windgust. However, the data must be somewhere, as it is used to create the windgust graph.

I have had this issue before - where a single temperature reading has gone mad, and I can graph it in gory detail down to a 2 minute slot. But can I find the bad data at that time? Nope. Really confused. Its gotta be somewhere to be able to use it.

I have had no problems correcting the records, month, day or 2 minute entries. But sometimes the errored data just aint there. Very odd. It always seems to be the transient data (my 8second data). Cant find where this is stored.

I connect via serial link - never could get the USB talk to my WS2300. At the moment, Ive upgraded my PC to one that doesnt have a serial onboard, so I have to keep my old PC running just to download the weather!! LOL - progress... hmm. (not your fault... just one of those things)

Ah - didnt know about maxtempdiff - I'll configure that tonite - had one cold snap where it dropped 60C in 2mins, then came back... from a nice balmy 25C to a bitter -35!

LOL!

My site is: http://weather.rkhosting.co.uk/index.htm

Really appreciate Cumulus - many thanks for it.

Cheers, RichK

[steve]

I have had this issue before - where a single temperature reading has gone mad, and I can graph it in gory detail down to a 2 minute slot. But can I find the bad data at that time? Nope. Really confused. Its gotta be somewhere to be able to use it.

The graphs are plotted at 1-minute intervals from live data; they just keep their own data. So if you're seeing a glitch in there that isn't in the data files, restarting would get rid of it. When you restart Cumulus, the graphs are initialised from the data in the monthly log files.

Ah - didnt know about maxtempdiff - I'll configure that tonite - had one cold snap where it dropped 60C in 2mins, then came back... from a nice balmy 25C to a bitter -35!

I've just remembered that there's also 'LCMaxWind' which sets the maximum wind speed that you want to accept.

[nking]

I've just remembered that there's also 'LCMaxWind' which sets the maximum wind speed that you want to accept.

Hi Steve,

I use EWTempdiff to stop erroneous temperature data, I didn't know there was one for wind (sounds a little rude ), are there any others that can be used for out of pattern data records? For example, I had a pressure reading which dropped by 10 hPa (my alarms alerted me), it was clearly an error which I then had to edit out of the data file.

[steve]

I use EWTempdiff to stop erroneous temperature data, I didn't know there was one for wind (sounds a little rude ),

It's only configurable for La Crosse stations. The cut-off for Fine Offset stations is fixed at 60 m/s (arbitrary, I know, and probably could be lower).

are there any others that can be used for out of pattern data records? For example, I had a pressure reading which dropped by 10 hPa (my alarms alerted me), it was clearly an error which I then had to edit out of the data file.

There aren't any at the moment, but there will hopefully be more in Cumulus 2.

The trouble is that every time I add a horrible kludge for bad data from La Crosse and Fine Offset stations, I die a little inside...

[Charlie]

Hi Steve,

Maybe the answer (for the next version) is not to keep adding kludges to filter bad data automatically, but simply make it easy to find and change a piece of data.

I've never been a database guy, but the principle of saving a piece of data only once and driving all uses from that one location dynamically has always felt like an elegant solution to me. Also the ability to create relatively unstructured queries to find the needle in the haystack (QUERY ALL where Temp=nn AND Date = xx/xx/xxxx) gives the database solutions a huge advantage.

You may already be doing this in Cumulus 2 - or perhaps this should go on the list for Cumulus 3 ?

[steve]

Maybe the answer (for the next version) is not to keep adding kludges to filter bad data automatically, but simply make it easy to find and change a piece of data.

I'm not sure that would suit everyone, as many people seem to think that the software should somehow magically be able to detect and correct all erroneous readings

I've never been a database guy, but the principle of saving a piece of data only once and driving all uses from that one location dynamically has always felt like an elegant solution to me.

Yes, that's exactly what I'm trying to achieve in Cumulus 2. I haven't quite got there with the rainfall yet; I've noticed that there are one or two places where I'm using the start of day rain counter where I could be adding up individual totals, so correcting some of rainfall records isn't as straightforward and intuitive as it could be, but hopefully it will all be much simpler eventually.