Correcting data from several months ago ?

[WetCoast]

I have some rainfall readings that are obviously incorrect, 1200mm in the 'Highest monthly rainfall'. Now I know I live on the 'wetcoast' but that's a bit much!!

I've taken a look through this forum and the wiki and I almost understand what all the files in the data directory are used for... almost!

The wrong data is in the month of August, so I first looked through the Aug15log.txt file. I believe this is the raw data received from the weather station? If I add up all the daily totals (the number at 23:57 each day) I get 83.1

Now, where should I put this number to correct my historical data? I'm not sure what files roll up to what files and if this is done auto-magically or if I must change all the ??.txt and ??.ini files.

Thanks in advance,
Gord

[steve]

As described in the FAQ, the monthly and annual rain totals are calculated from the daily totals in the dayfile.txt, so you need to check those daily totals and correct if necessary. MX as yet has no editors for adjusting all-time records etc, so you will need to edit the relevant .ini files, with Cumulus not running. The highest all-time monthly rainfall is held in alltime.ini and the highest rainfall for August is held in monthlyalltime.ini. You may also need to correct year.ini, holding this year's highest monthly rainfall.

There are a number of entries in the FAQ about correcting data (rainfall in particular) and these should be useful. You will need to ignore the parts about Cumulus 1.

[WetCoast]

Thanks Steve, sorry for being so stupid.

I think I've found and corrected the erroneous data. I used the data from the Aug15log.txt (column 'j' in a spreadsheet) and compared it to data in column 'o' from the dayfile.txt They seemed to be OK.
Then I looked at the monthlyalltime.ini file and found the highdailyrainvalue to be wrong. I changed that to equal the number from the highest day value in dayfile.txt
I found highmonthyrainvalue to be wrong as well in monthlyalltime.ini so I changed it to the total I got by adding up all the daily totals from dayfile.txt. I'm not sure what the time should be for the highmonthlyrainvalue in monthlyalltime.ini so I just set it to 12:00:00 AM. I'm assuming this value is not really important??
I also found the DailyHigh and MonthlyHigh values in section [Rain] of year.ini to be wrong. I changed the DailyHigh and MonthlyHigh to the highest value from monthlyalltime.ini

I think all is correct now, I hope
And I now have a better understanding of how all these files work/fit together. I can't say I understand it 100% but the fog is starting to lift!! (note the weather reference haha)

Might I suggest some sort of utility that would re-populate the "ini" files from the monthly log files (I.E. Aug15log.txt)? That way I could have checked/changed one fine and got the utility to update all the other files. Just a thought.

Thanks again,
Gord

[steve]

I'm not sure what the time should be for the highmonthlyrainvalue in monthlyalltime.ini so I just set it to 12:00:00 AM. I'm assuming this value is not really important??

Yes, the ini-file routines only have options for saving full timestamps.

Might I suggest some sort of utility that would re-populate the "ini" files from the monthly log files (I.E. Aug15log.txt)? That way I could have checked/changed one fine and got the utility to update all the other files. Just a thought.

That's exactly what the editors in Cumulus 1 do. I'm working on similar facilities for MX.

[WetCoast]

oh, ok, thanks for the reply. I was thinking of a 'simple' script that would parse the log.txt and re-write the ini files but, as you say, if the editor will do that, great.

[WetCoast]

One more question on this same subject.
I have some wind speed readings that are incorrect, showing a max wind speed of 138.24Km/hour. I can find the erroneous number in alltime.ini, alltimelog.txt, dayfile.txt, month.ini, monthlyalltime.ini, Oct15log.txt and year.ini.

This leads me to believe the incorrect data was received by my weather station and passed on to Cumulus. So I can correct the data in Oct15log.txt but how will I know what the number (the max wind speed number) should be for the 'alltime' files? Do I have to search through all the other *15log.txt files looking for the largest wind speed number? If I correct the value in Oct15log.txt and restart Cumulus will it re-build the alltime files?

[steve]

If I correct the value in Oct15log.txt and restart Cumulus will it re-build the alltime files?

No, it doesn't scan the log files every time it starts up looking for highs and lows, that would make it take a very long time to start up - that's why the .ini files exist. You can find the previous all-time record by looking in alltimelog.txt and edit alltime.ini. As MX currently does not have the editors that Cumulus 1 has, you will have to correct the other files by some other method. Note that Cumulus (1 and MX) keeps backups of the relevant files from the last 10 days and the last 10 start ups, so if the problem is still within that range, you can look at the files in the backup folders and obtain the previous values that way. Note also the 'spike removal' settings which are useful for preventing the erroneous figures from Fine Offset stations being logged in the first place.

[WetCoast]

Note also the 'spike removal' settings which are useful for preventing the erroneous figures from Fine Offset stations being logged in the first place.

Thanks, I did have the spike removal in place for some, but not all, values. I've just corrected that!

I'm sorry if I don't understand what the 'editor' will do (or what it does in the Windows version) but I still think it would be helpful to have a utility, perhaps separate from Cumulus itself, to parse the log files and update the 'alltime' files. That is assuming the log.txt files hold the raw data received from the weather station.
Anyway, thanks again. If nothing else I'm, slowly, learning how all the files are related.

[WetCoast]

I apologize for asking so many dumb questions, but here comes another one...
Are the "monthly log files" the most complete (raw) data, stored by Cumulus, from the weather station?

I assume because the monthly log files are only updated (in my case) ever 5 minutes, that it's possible for records to be created at times between the 5 minute 'tick'? The reason I ask is that some of my record data has times other than 5 minute intervals. I.E. 2:48 PM or 7:14 PM

I suspect this means I can not use the monthly log files to correct my record data? It's possible, for example, to have a maximum temperature happen and not be recorded in the monthly log file?

[steve]

Yes, the monthly log files don't necessarily contain the highs and lows (as mentioned in the FAQ), they go in dayfile.txt at the end of the day. Recreating highs and lows from the monthly log files is a fallback position when you have nothing else to work from (e.g. you don't take regular backups of your data for some reason, and the Cumulus limited backups are too recent, if you didn't notice the problem for more than 10 days). In most cases the data in the monthly logs should be close enough.

If the Cumulus data logs aren't sufficient for your needs, you could create a MySQL database using the realtime update facility in MX. It is my intention that MX will eventually have the option of a local SQL-based log to supplement the text log files, which will contain the highs and lows since the last entry - this is what Cumulus 2 did. Whether I will ever actually get around to doing it is another matter...

[WetCoast]

Thanks Steve.
Sorry to be such a pest. I start at something and keep coming up with questions...
I'll look into the MySQL stuff, been meaning to anyway.

Thanks again,
Gord