Monthly log files from before 27 Jan 2004

[packman2008]

I get a lot of errors like...

2024-01-14 20:53:47.720 ReadMonthlyLogs: reading data/Dec85log.txt
2024-01-14 20:53:47.724 Error ReadMonthlyLogs: File data/Dec85log.txt skipped, it is from before 27 jan 2004!

I believe that 27 Jan 2004 is the birth date of Cumulus so you might not expect to have data from before that. However, I've loaded data back to Jan 1979 and CumulusMX seems to handle it OK as both monthly logs files and the dayfile. As a result I get the error above about 300 times each time I run CU.

Having said that any charts that contain ALL plots allow me to view all the data back to 1979 so I'm not sure what this error is telling me. What data isn't being read from these files?

1) What am I missing out on by the older monthly files not being read?
2) Is there a technical reason why the older files couldn't be read? Could CU perhaps read the CMX "Records Began Date" (31/12/1978 in my case) and accept any files after that date?

[HansR]

After the start of Cumulus the datafiles have been modified incrementally, every version had additions to the logs and the dayfile and there were many versions of datafiles around. The CreateMissing utility has made it possible to more or less unify the dayfile but the logfiles fields not being there was a nightmare when I started CUtils. Nevertheless it was pretty well documented in the wiki.

It is especially wind and solar which do often not exist in old files. I could take a look at your data and see what can be done to break the limit.
There are no users who have such long dataseries so this is special and definitely interesting.

I started CUtils in 2019 just before Mark started to publish his many CMX changes and over the years things have become more stable and I learned more about the internals so maybe I am able to open the barrier. A condition would be that the format of your logfiles adheres to Cumulus way to the beginning. If there are additional data I would have to check what to do with that.

Another thing is, that I most likely I will have increasingly less time the coming period as my attention will flow temporarily in another direction. I will announce that shortly. I won't be absent but I won't take on large projects while on the other hand CMX itself it going through some rapids in change. Don't know how long that will take, I assume some months.

So in summary: yes, I could take a look at the data, but don't expect much very quickly.

If you want me to take this on, please send the datafile and we will communicate via PM or email on the ins and outs of this.

[packman2008]

PM sent.