Best way to clean up bad "high/low" records spikes?

[steve]

The best solution is to use an external editor, with Cumulus stopped. The data logs can't be edited from within Cumulus anyway (it's too hard to co-ordinate editing with logging). Something like CSVed doesn't mangle the dates, and allows sorting on any column. It will also correct date formats which have already been mangled.

[uncle_bob]

Something like CSVed doesn't mangle the dates, and allows sorting on any column. It will also correct date formats which have already been mangled.

Good to know, thanks Steve.

[THX]

What kind of problems are you having with your Vue, and how often do they occur?

Hi Steve, thanks for the timely and in-depth reply.

The temperature and humidity sensors have completely failed. Unfortunately, I am not having a great success record with my Vue. My first first Vue lasted 3 years until the temperature sensor started to read too warm, which required a factory refurb ($80US). This latest refurb lasted less than 1 year, which is where I'm now seeing the temp/humidity completely fail. Davis Instruments has raised the price of a refurb to $100US ($112US w/shipping). If this latest refurb fails again, I'll likely go directly to the VP2 system.

Regarding eliminating spikes...I see where you're coming from. Getting into data management of log files starts to veer into database management schemes, which can be time consuming and beyond the scope of the product.. Perhaps a "keep it simple" method can work here such as just assigning a variable to the last 1, 2 or perhaps 3 last records. This way, when loading the dayfile data, we can pick which is the correct record and apply it within the editor. I feel bad suggesting such a thing as this will take more time of your development cycles if this is something you even want to pursue...

[THX]

The best solution is to use an external editor, with Cumulus stopped. The data logs can't be edited from within Cumulus anyway (it's too hard to co-ordinate editing with logging). Something like CSVed doesn't mangle the dates, and allows sorting on any column. It will also correct date formats which have already been mangled.

When editing the files, how do we know what the column header values are?

[mcrossley]

When editing the files, how do we know what the column header values are?

The column header files are in the Cumulus root folder.

[steve]

The temperature and humidity sensors have completely failed.

Ah, that explains it. Mine went too some time ago, and to be honest I still haven't completely cleaned up the mess it made. I'll get around to it eventually.

2014-11-28 08_27_53-Sanday weather.png

Some code to prevent that kind of thing wouldn't be too difficult, so I'll add that at some point.

[RangeRyder]

I've read a lot about "spike removal" settings in these posts but don't understand that. How do I find these "spike removal" settings and how do I fix them?

[THX]

I just spent the better part of a week totally close to 20 hours editing the last 5 years of my monthly .txt files by loading them and formatting the data into Excel.

This was the only way that I could find to locate and analyze my data for accurate spike removal.

One record that I was unable to locate and subsequently remove was the "Highest Daily Range" record.

If there is a recommendation that I have for Cumulus MX is the ability to remove spikes in a easy way. Going through 5 years of monthly.txt files was the very definition of pain...

[steve]

I've read a lot about "spike removal" settings in these posts but don't understand that. How do I find these "spike removal" settings and how do I fix them?

Sorry, I missed your post somehow. The 'spike removal' settings are on the calibration settings screen. Choose values which represent a change between one reading and the next which wouldn't normally happen if the station was working properly. Note that the 'spike removal' settings are for stopping bad data being logged in the first place, not for correcting existing bad data.

[steve]

One record that I was unable to locate and subsequently remove was the "Highest Daily Range" record

It's calculated by subtracting the day's minimum temperature from the day's maximum temperature, and then finding the highest of the those value. So if you want the alltime record editor to find it for you, you have to make sure all of your daily minimum and maximum temperatures in dayfile.txt are correct.

It is much easier to prevent bad data going in than subsequently removing them. Running a station with faulty sensors is a good way to get bad data into the files, and I speak from experience and know exactly what you mean about the pain - I still haven't corrected all of the bad temperature readings generated by my faulty Davis temperature sensor. The difference between the bad data produced by Fine Offset stations, and the unusual situation that you and I have both had, is that FO stations do it as a matter of course, whereas Davis stations only do it when something has broken. If I could think of a way to make removal of large amounts of bad readings easy I would have done it for myself.

[shajoe44]

What would be "typical" settings on the calibration chart. Can you send me a screen print of one. I had a reading of 74 inches of rain today and it is sunshining. I went in and edited that number to zero. I also get wind spikes in the 117 range.

Thanks

[steve]

There aren't any "typical" settings, it depends on the weather you get at your location. Start with fairly large values - the figures are differences between one reading and the next in the native units of the station - and if that doesn't remove your spikes, reduce the value. If you start getting 'flat lines' - values not changing in Cumulus even though they are changing on the weather station console, you have too small a value set. For example, a value of 20 for the wind spike removal settings will cause sudden changes of 20 m/s (45 mph) or more to be ignored, e.g. a sudden change from 20 mph to 65 mph or more would be ignored.

What station type have you selected in Cumulus for your WS-2090?

[shajoe44]

Fine Offset (WH 1080 etc)

[shajoe44]

these are my most recent error logs.

[steve]

Yes, so the spike removal code is now filtering out the false wind readings from the station.