Bugs when importing old data

[dale]

I've been using Cumulus for quite a while and am very happy with it. Yesterday and today I decided to try to reformat the data that my weather station had collected when using FreeWX from 2003 to 2008 before I switched to Cumulus. I reformatted the logfiles for daily data that I had collected from 2005 to 2008. I also reformatted my almanac for data between 2003 and 2005 because I did not have daily data for that period.

I used the "create missing" button in the dayfile editor to add my monthly logfile data to the dayfile. Note that my old data has some missing data. For example, there is no outside temperature for some periods of time when the outside sensor was not operating for a month or so and similarly there is missing barometric data when my battery on the inside sensor died for a few days before being replaced. I observe that for missing data in the log file, the "create missing" button seems to have problem and populated the maximum and minimum with -9999 and 9999 respectively for days with missing data. This seems to be an error in how missing data is handled. I wanted to keep the records where only some data is missing. There are times where a month's data is missing because the data was collected at a remote site and the battery could not be replaced for a while.

It also seems that the apparent temperature is messed up in the dayfile when it is missing.

I also attempted to reformat my almanac data to be in the dayfile format and prepend it to the existing dayfile. This worked as expected, but showed a few additional issues as noted below.

When there are errors in the almanac file the error log gives a line number and a message like "not a fp value" or "not a valid integer" but does not identify the field that it is complaining about. When I look at the offending line, I can't find the issue. If the documentation could clearly specify the format expected of each field and if the create missing function was more forgiving for missing data, it would make it much easier to bring in old data.

Finally a suggestion: It would be nice if there was a report like the "view monthly highs and lows", but where you could specify the year that you want to look at. At least a way to look at the highs and lows of a particular year would be helpful. I guess if you did this it would also be nice to allow the records editor to specify the year as well.

I can provide the old data files that are demonstrating the issues if you wish.

Thanks.

[steve]

The 'create missing' function is really intended to work with Cumulus log files, which wouldn't normally have missing items - Cumulus itself never logs missing items; in general it doesn't have a concept of 'null' data. At some point I could look at changing this - most likely this would eventually be in the new version that I'm working on, rather than the current version, as a lot of work would be involved.

I'm pretty sure the documentation (in the wiki) does already say which fields have to be in which format, but perhaps that's only for the monthly log files. The same rules would apply to the corresponding items in dayfile.txt.

Edit: I had a quick look at the specification of dayfile.txt in the wiki, and it does appear to specify which items have to be integers.

[dale]

Thanks for your quick reply. I failed to look at the WiKi for the data types, only looked in the help file. I will make sure my data types are all correct.

What does cumulus put in the log files if a sensor is not reporting (say the battery dies in the pressure sensor or the outdoor temp sensor)?

[steve]

It continues to report the last good reading that it had, on the assumption that the problem will be fixed quickly.