Page 1 of 1

Edit Historical Data

Posted: Mon 11 Oct 2021 1:27 pm
by AberDino
Hi All,

I've had a Davis weather station (VP2 ISS with Vue console with a WeatherLinkIP data logger) since 2011. As I live on the coast, the weather station regularly gets a hard time, and I've previously had to replace the outdoor temperature/humidity sensor and the anemometer. A little while ago, my outdoor humidity readings were starting to deviate again, so today I've replaced this sensor (previously replaced in December 2016, so it seems to last ~5 years).

Apologies if these has been asked previously. I did search the forum but couldn't find the answer I was looking for. What is the best way to edit the historical outdoor humidity readings? Should I simply remove these values, or replace them with dummy, more believable, values? What about the calculated data based on outdoor humidity, such as dew point? Can I force these to be recalculated, or should I simply delete/replace these values too? Last but not least, due to the extremely low outdoor humidity readings (0% at times), some of the records are false too. Can these be recalculated once the historical dataset has been corrected?

Thanks in advance for your help,

Pieter

Re: Edit Historical Data

Posted: Tue 19 Oct 2021 1:10 pm
by AberDino
So, after reading some of the reference information on the Cumulus Wiki I decided to have a go...

I stopped the running instance of Cumulus MX, and made copies of the monthly txt log files which contained the incorrect data. In Excel, I merged the "monthlyfileheader.txt" and monthly txt log files so I could see which columns contained the data I was looking for. I then proceeded to delete the erroneous outdoor humidity values, as well as all values which depend on the outdoor humidity values such as dew point, apparent temperature, etc. After removing the header, I saved as CSV files and renamed them back to txt files. I then took a similar approach with the dayfile.txt file (using the deayfileheader.txt file), and copied the updates files back to the Cumulus MX data directory.

After starting Cumulus MX, I used the 'Edit' menu in the web interface to correct the monthly / yearly / all-time records as required. I'm not sure whether this is the recommended approach, but it definitely seems to have worked :).

The only thing to be aware of, is that when you import the txt files into Excel, it automatically converts the date to the "dd/mm/yyyy" format, so you need to force this back to "dd/mm/yy" before you save the files. I found this out the hard way ;).

Thanks,

Pieter

Re: Edit Historical Data

Posted: Tue 19 Oct 2021 4:08 pm
by AndyKF650
Saving as a .csv file gets around the formatting problem

Re: Edit Historical Data

Posted: Tue 19 Oct 2021 4:56 pm
by AberDino
That's what I did, but without formatting the date column, the dates in the saved CSV file were in the "dd/mm/yyyy" format.

Re: Edit Historical Data

Posted: Tue 19 Oct 2021 7:36 pm
by mcrossley
Yes, unless you force Excel to produce the date format you want it will go and do its own thing - probably based on your system wide locale settings.
Always be wary when editing in Excel.