Page 1 of 1

Need to remove or correct rogue entries

Posted: Wed 18 Jan 2017 2:31 pm
by sstmjoy
I need to clean some data for Temp (have a reading of -250 !), rainfall, and wind. I got some rogue readings for 2 or 3 times over the past 24 hours (and in fact they repeat 100s of times in the data file).

I know I can clean the ini and txt files for the various day / yesterday / month / alltime settings (although its a long and painful process), and clear out the bad data from the dayfiles, but I am missing something... when I reload the site the max / min temperatures are re-calculated; also, the gauges for temp go wild because -250 is clearly a problem... and they just "loop" (running from -250 to +30).
This loop could be linked to the 100s of bad data points I suppose ?

Anyway, thoughts ? What files can I supply that will help anyone ? Is it a case of doing a restore from a couple of days ago and see if it re-occurs ?

Have an Aercus 3083 (i.e. Fine Offset with UV clone), and am running version 3042...

Re: Need to remove or correct rogue entries

Posted: Wed 18 Jan 2017 3:43 pm
by steve
Today's max and min so far are loaded from today.ini at startup. Are they they the max and min that you're referring to?

You could try 'rewinding' using the Cumulus backup files - if the station didn't leave those values in its memory, they shouldn't reoccur.

Are you using the 'spike removal' settings? They normally help to significantly reduce the garbage that Fine Offset stations are prone to generating. You have selected the correct station type in Cumulus?

Re: Need to remove or correct rogue entries

Posted: Wed 18 Jan 2017 5:27 pm
by sstmjoy
I have stopped file uploads to my website, so you can see the values I am struggling with (last uploaded yesterday).

You can see what the gauges are doing at : http://steve-joy.com/weather/gauges.htm
The low apparent temperature can be seen at http://steve-joy.com/weather/today.htm
(you can see some other really silly values on that page too)

I have cleaned up some of the data manually now, so the other pages look relatively OK - at least they are within the realms of reality - (apart from wind run),

Yes I have the right station selected - I thought I had spike settings turned on, but I will check that.

I will disable FTP and try with a backup, and see what happens

Thanks a lot.

Re: Need to remove or correct rogue entries

Posted: Wed 18 Jan 2017 8:06 pm
by sstmjoy
I have kind of got it working now... still got a couple of rogue data points but its working, restored from a backup and got rid of the bad data.
Thanks