Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Enhancements for correcting bad historical data and records

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
dale
Posts: 23
Joined: Tue 29 May 2012 9:15 pm
Weather Station: Ambient WS-5000
Operating System: Windows 11
Location: Glen Arbor, MI
Contact:

Enhancements for correcting bad historical data and records

Post by dale »

I've been using Cumulus 1 since 2002 with two different Oregon Scientific weather stations and am now working on upgrading to a new weather station and to Cumulus MX. I would like to correct bad data from my old data starting in 2002, but, unless I'm missing something, I find that the tools to do so are not practical.

The root cause of the bad data is past failures of sensors reporting bad data and this data getting recorded in the monthly files, one record every 10 minutes. Sometimes the failure would go unnoticed for weeks or months (while I was not onsite) resulting in 1000s of records in the monthly file with bad data, one every 10 minutes. For example, a barometer reading of 18, or 0 (in of Hg) when the lowest every pressure reading on Earth is about 25.6. Other examples in my data are barometer readings too high (above 32), high temperature (above 140), high dew point (158), high rain rate, etc.

It is not practical with the monthly file editor to edit the many errors. I have a few suggestions:

1) Enhance the nice createmissing tool to check for invalid data in the monthly files and ignore it if it is outside of some specified values for each column (or some reasonable predefined values). This should make the regeneration of more accurate dayfile data (by deleting the bad dayfile records and running createmissing). (I realize that data errors that are in bounds cannot be corrected this way and need manual correction--see #3 below)

2) Slightly enhance the records editor tool to recreate the various records files (monthly and all time records): allow the user to click on one of the suggested values on the right to populate the record, rather than having to type the new record in by hand.

3) I think that doing both 1 and 2 would be sufficient to clean up bad data for most practical purposes. It might also be nice to enhance the dayfile and monthly file editing tools to operated on multiple rows (delete multiple) or columns (change a range of values in one column to a specific value). This would allow for easier corrections of bad data that span multiple days or times, although, this can be done with something like excel, as long as you are careful to maintain the date column in the proper format.
Altocumulus
Posts: 136
Joined: Sat 18 May 2013 1:58 pm
Weather Station: Davis Vantage Pro2 with Solar
Operating System: Windows 10 Home
Location: NE Scotland
Contact:

Re: Enhancements for correcting bad historical data and records

Post by Altocumulus »

Slightly before Storm Arwen hit NE Scotland I had a couple of glitches in my data which resulted in Cumulus alarming with every single record overwritten by the value 6.8.
I am, therefore, interested in finally moving over to MX ( I have run the two in parallel for some months via VirtualVP ).

I know I have errors in my data, as with the OP, usually from sensor malfunction or excessive and ridiculous rain rates.

When editing, is there an accepted figure one can override the data with ( perhaps an "x" ) which any queries will ignore?

I, therefore, would like to second this suggestion, if it is a possibility.
Altocumulus
Posts: 136
Joined: Sat 18 May 2013 1:58 pm
Weather Station: Davis Vantage Pro2 with Solar
Operating System: Windows 10 Home
Location: NE Scotland
Contact:

Re: Enhancements for correcting bad historical data and records

Post by Altocumulus »

I have an opportunity to carry out some serious checking of the dayfile for errors I know exist, -68 temps, 200mm rain rates when no rain occurred. My Davis is being cleaned with a new temperature sensor. I'm also adding on a sun sensor.

The wiki doesn't explain what to do, or at least I haven't found it, with erroneous data - I'd rather find an appropriate replacement in terms of an error entry rather than try and guess a value. Is there a recommendation?

From what I gather I 'just' need to edit the dayfile and not the various records?
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Enhancements for correcting bad historical data and records

Post by mcrossley »

Cumulus does not (currently) have the concept of null data for the core data values. So for wind, temp, humidity, rain you have to enter some sort of value.

Once you have the dayfile edited, then you can look at your records.

The all-time and monthly all-time records files have their own log files which show what each new record is and the date/time/value of the record it is replacing. You can use these files to back out invalid records... Or you can use the record editors which will show the values in the record files along with the records it derives from the dayfile (and optionally the monthly log files).
Altocumulus
Posts: 136
Joined: Sat 18 May 2013 1:58 pm
Weather Station: Davis Vantage Pro2 with Solar
Operating System: Windows 10 Home
Location: NE Scotland
Contact:

Re: Enhancements for correcting bad historical data and records

Post by Altocumulus »

Thanks Mark,

I had come round to understanding that null or -999 values weren't acceptable.

Once the dayfile has been adjusted, what's the next best step to take?
On MX how long does it take the app to process the files and populate the tables?

Cheers

Geoff
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Enhancements for correcting bad historical data and records

Post by mcrossley »

Altocumulus wrote: Wed 01 Dec 2021 7:38 am Once the dayfile has been adjusted, what's the next best step to take?
On MX how long does it take the app to process the files and populate the tables?
Try the records editors, the dayfile will load pretty quickly - couple of seconds, but it depends on the platform.
It's a bit tedious to go through all the records (the monthly ones in particular), but if you have a spare hour or so...!
Altocumulus
Posts: 136
Joined: Sat 18 May 2013 1:58 pm
Weather Station: Davis Vantage Pro2 with Solar
Operating System: Windows 10 Home
Location: NE Scotland
Contact:

Re: Enhancements for correcting bad historical data and records

Post by Altocumulus »

Thanks.
TheBridge
Posts: 115
Joined: Mon 16 Mar 2020 3:23 am
Weather Station: Davis
Operating System: Windows 10
Contact:

Re: Enhancements for correcting bad historical data and records

Post by TheBridge »

Hi ALL,

Similar situation here. I had an errant wind speed of 157mph (I am fairly sure this is an error as my house is still standing this morning). I want to change that single value before it propagates into the monthly and annual records files.

Mark: is there maintenance I need to do with MX to avoid this type situation or is just so rare it is just simpler to do manual modifications to datafiles?

Bridge
freddie
Posts: 2433
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Enhancements for correcting bad historical data and records

Post by freddie »

You can do it all through the interface screens now. Start with the log file editor. If you do the correction before rollover then you can skip the dayfile editor. Then use the records editors. You can load the values from the log files in the records editors, that's why it's best to do the logfile editor first. You shouldn't need to touch the actual files themselves.

You can also set "spike" values using the interface in order to filter out future crazy values.
Freddie
Image
Benpointer
Posts: 30
Joined: Sun 02 Jan 2011 6:44 pm
Weather Station: Davis Vantage Pro2
Operating System: OSX Sierra

Re: Enhancements for correcting bad historical data and records

Post by Benpointer »

freddie wrote: Sun 06 Nov 2022 10:58 pm You can do it all through the interface screens now. Start with the log file editor. If you do the correction before rollover then you can skip the dayfile editor. Then use the records editors. Yo
What is the 'log file editor'?

In Cumulus MX all I see under Edit is the following:

Image

My Vantage Pro2 has gone a bit haywire over the past 24 hours and is posting a continuous rain rate of 281.0mm per hour. I have temporarily set the rain multiplier to zero to stop it being recorded in Cumulus while I sort the VP2 issue out but I would like to strip out the erroneous data. It has been dry throughout the period of rogue readings, which helps - I just need to se the rain rate to zero throughout.
User avatar
PaulMy
Posts: 3775
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2 Plus 24-Hour FARS
Operating System: Windows8 and Windows10
Location: Komoka, ON Canada
Contact:

Re: Enhancements for correcting bad historical data and records

Post by PaulMy »

HI,
What is the 'log file editor'?
The Edit from within the Monthly Logs view.
From Dashboard => Data logs => Monthly logs > select and Load a period > select a line > Press Edit

Enjoy,
Paul
Davis Vantage Pro2+
C1 www.komokaweather.com/komokaweather-ca
MX www.komokaweather.com/cumulusmx/index.htm /index.html /index.php
MX www.komokaweather.com/cumulusmxwll/index.htm /index.html /index.php
MX www. komokaweather.com/cumulusmx4/index.htm

Image
Benpointer
Posts: 30
Joined: Sun 02 Jan 2011 6:44 pm
Weather Station: Davis Vantage Pro2
Operating System: OSX Sierra

Re: Enhancements for correcting bad historical data and records

Post by Benpointer »

PaulMy wrote: Sat 21 Jan 2023 3:11 pm HI,
What is the 'log file editor'?
The Edit from within the Monthly Logs view.
From Dashboard => Data logs => Monthly logs > select and Load a period > select a line > Press Edit

Enjoy,
Paul
Thanks!
Post Reply