Bugs when importing old data
Posted: Thu 12 Jun 2014 7:20 pm
I've been using Cumulus for quite a while and am very happy with it. Yesterday and today I decided to try to reformat the data that my weather station had collected when using FreeWX from 2003 to 2008 before I switched to Cumulus. I reformatted the logfiles for daily data that I had collected from 2005 to 2008. I also reformatted my almanac for data between 2003 and 2005 because I did not have daily data for that period.
I used the "create missing" button in the dayfile editor to add my monthly logfile data to the dayfile. Note that my old data has some missing data. For example, there is no outside temperature for some periods of time when the outside sensor was not operating for a month or so and similarly there is missing barometric data when my battery on the inside sensor died for a few days before being replaced. I observe that for missing data in the log file, the "create missing" button seems to have problem and populated the maximum and minimum with -9999 and 9999 respectively for days with missing data. This seems to be an error in how missing data is handled. I wanted to keep the records where only some data is missing. There are times where a month's data is missing because the data was collected at a remote site and the battery could not be replaced for a while.
It also seems that the apparent temperature is messed up in the dayfile when it is missing.
I also attempted to reformat my almanac data to be in the dayfile format and prepend it to the existing dayfile. This worked as expected, but showed a few additional issues as noted below.
When there are errors in the almanac file the error log gives a line number and a message like "not a fp value" or "not a valid integer" but does not identify the field that it is complaining about. When I look at the offending line, I can't find the issue. If the documentation could clearly specify the format expected of each field and if the create missing function was more forgiving for missing data, it would make it much easier to bring in old data.
Finally a suggestion: It would be nice if there was a report like the "view monthly highs and lows", but where you could specify the year that you want to look at. At least a way to look at the highs and lows of a particular year would be helpful. I guess if you did this it would also be nice to allow the records editor to specify the year as well.
I can provide the old data files that are demonstrating the issues if you wish.
Thanks.
I used the "create missing" button in the dayfile editor to add my monthly logfile data to the dayfile. Note that my old data has some missing data. For example, there is no outside temperature for some periods of time when the outside sensor was not operating for a month or so and similarly there is missing barometric data when my battery on the inside sensor died for a few days before being replaced. I observe that for missing data in the log file, the "create missing" button seems to have problem and populated the maximum and minimum with -9999 and 9999 respectively for days with missing data. This seems to be an error in how missing data is handled. I wanted to keep the records where only some data is missing. There are times where a month's data is missing because the data was collected at a remote site and the battery could not be replaced for a while.
It also seems that the apparent temperature is messed up in the dayfile when it is missing.
I also attempted to reformat my almanac data to be in the dayfile format and prepend it to the existing dayfile. This worked as expected, but showed a few additional issues as noted below.
When there are errors in the almanac file the error log gives a line number and a message like "not a fp value" or "not a valid integer" but does not identify the field that it is complaining about. When I look at the offending line, I can't find the issue. If the documentation could clearly specify the format expected of each field and if the create missing function was more forgiving for missing data, it would make it much easier to bring in old data.
Finally a suggestion: It would be nice if there was a report like the "view monthly highs and lows", but where you could specify the year that you want to look at. At least a way to look at the highs and lows of a particular year would be helpful. I guess if you did this it would also be nice to allow the records editor to specify the year as well.
I can provide the old data files that are demonstrating the issues if you wish.
Thanks.