Possible bug in 1.9.4

[Adrian Hudson]

Hi Steve,

I think there is a problem in the Edit dayfile.txt - Create Missing function.

I am still messing with importing years of data from another weather recording program. I have imported old data several times and used the Create Missing function with no problems. After upgrading to 1.9.4 and importing some more data the total rain reading went wonky. It appears that the Create Missing function no longer calculates the "Total rainfall for the day" column in dayfile.txt correctly. Many readings are missing or much too low. The columns "Maximum rainfall rate" and "Time of maximum rainfall rate" are being populated (I havn't not checked if they are correct though) when there is rain on the day but the "Total rainfall for the day" column is be blank. See screenshot of dayfile.txt data in a spreadsheet.

I restored 1.9.3, deleted the offending dayfile entries, reran Create Missing and the data was imported okay.

Adrian

[steve]

The changes I made in 1.9.4 in the dayfile editor were:

1. To correct the NOAA cooling and heating values (they were the wrong way round)
2. To attempt to cater for all cases where an 0900/1000 start of day is in use. It's possible that in correcting some scenarios I have broken others. But the code to calculate the values is still the same, so any bug would affect all of the data, not just rainfall total. The daily rain calculation just takes the figure that it finds in the 'total rainfall so far today' field in the entry that it thinks is the last for the day in question. So I guess an 'off by one' error would affect the rainfall total much more significantly than the other figures.

Do you actually use an 0900/1000 start of day?

If you'd like to attach a sample monthly data file, I'll give a try here.

[Adrian Hudson]

Hi Steve,

I do indeed use 0900/1000 start of day.

Please find attached log file for Aug. Version 1.9.4. calculates 0.2mm for the month and 1.9.3 calculates 42.6mm

Note, this is a log file generated by my program that extracted data from my old weather program hence several columns with -999.99 figures. Also note I just noticed my extraction program is ignoring BST - I was sure I had tested that! Neither of these things account for the rain problem though.

Regards

[steve]

I'm assuming by what you say that you switch to 10:00 during BST? If so, that's the problem, and the determination of the start of day during BST is what I corrected in 1.9.4, which is why your data works in 1.9.3 but not in 1.9.4. Your rainfall total is resetting at 09:00 when it should be resetting at 10:00 in August.

[Adrian Hudson]

Ahhhhh! The penny drops!
(Quoting myself:

Neither of these things account for the rain problem though.

Perhaps I'll take that back!!)

So, we both had bugs. Yours was masking mine. Now you fixed yours, mine is there for all the world to see (well, now it is, on here, anyway!)

I will fix my program, run it through Cumulus and let you know.

[Adrian Hudson]

Well...

... I think you are right. I fixed my program and produced all the monthly log files again and ran them through Cumulus to make the dayfile.

With the initial data as produced by my faulty program and converted to dayfile by cumulus 1.9.3 , the total rain matched the console within half a millimetre.

With my conversion program fixed and converted to the dayfile by 1.9.4 the yearly total reported by Cumulus console and the Davis console differ by nearly 60 mm. I have spent all evening with the logfile data and the raw data from the old weather program in a spreadsheet and I now have a headache and am more confused than ever

...but I do think Cumulus is correct and my conversion program probably correct... so i am baffled where my rain has gone... still, that's not your problem.

Thanks for your help. I will post the resolution to the problem - if ever I find it. I will be sad if my history is invalidated though.

One question: If I copy the cumulus folder in its entirety and rename it as a test environment that I can mess with the data, will Cumulus be happy to run in there? It won't break the original by referring to a "memory" of where it was installed.?

[steve]

One question: If I copy the cumulus folder in its entirety and rename it as a test environment that I can mess with the data, will Cumulus be happy to run in there? It won't break the original by referring to a "memory" of where it was installed.?

It won't mind, it just uses the data folder where the executable runs.

[Adrian Hudson]

That's great, thanks Steve.

One last question, a bit off topic: Do the NOAA reports honour the 0900/1000 day start setting? i.e. do they throw back temperatures and rain? Not sure if a NOAA report would be meant to do this or not (being all American and all) but its nice to know how the generates them.

[steve]

Yes, they do, because almost all of the data comes from dayfile.txt. The daily average wind speed is calculated from the monthly log file entries, and it appears to use midnight for that. There's a comment in the code to that effect; it's likely that I hadn't worked out how to retrospectively determine whether a date was DST or not, so I just left it using midnight as the result was likely to be wrong anyway. I guess I could look at fixing that one day.

[Adrian Hudson]

Again, thanks for the info Steve.

It would be good to have the NOAA reports self-consistant. Really, either they should fully honor the roll over setting or not at all

Anyway, back to checking through my historical data this eveing for me...

End of thread, I think.