Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.9.1 (build 3096) - 31 October 2020 (please see announcement regarding releases since 3.5.0)
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of the NOAA report and Snow Index drop-down menus to 2030)

Use this Wiki link to Download the Software (Cumulus MX / Cumulus 1 and other related items).

Cumulus data files changes

Discussion of the Cumulusutils tool and website generator.

Moderator: HansR

Post Reply
User avatar
PaulMy
Posts: 2306
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2+ Cumulus1 / CummulusMX
Operating System: Windows8 / Windows10
Location: Komoka, ON Canada
Contact:

Cumulus data files changes

Post by PaulMy »

Running the latest Cumulusutils has highlighted the changes over time in the Cumulus data files and the impact on the modules that HansR has implemented - after considerable trouble shooting :oops: :o .

I have Cumulus data from October 2008. I have been interchanging the data files between Cumulus1 install and CumulusMX, ignoring the fact that the old data files did not have all the data fields that are in later Cumulus1 and now CumulusMX versions. Generally that has not been an issue even though the source for Cumulus1 is with Davis Envoy/USB logger, and CumulusMX with Davis Vue and WiFiLogger so after interchanging the data files are a combination Envoy/USB and Vue/WiFiLogger.

A few bad data issues were discovered in dayfile and monthlog files when running Cumulusutils' pwsFWI, dayrecord and Yadr modules and these were quickly found and corrected after reading the cumulusutils.txt log file.

Now the big one was the introduction of the Windrose module as a beta. Eventually Hans was able to identify the issue which was that my oldest files did not have all the data fields that are in the current version of Cumulus1 i.e. Wind bearing. Hans made changes and that issue is now overcome.

After I updated to the latest Cumulusutils v.3.6.5 and still had issues I concluded it was either me or Hans' fault ;)
It seemed that when running the modules individually, like pwsFWI everything processed and FTP/updated. However when I selected the parameter All or Website the files were all created but seemed to me that these were not being FTP/updated. I hadn't yet tried the new Thrifty option as still presumed I had a bug somewhere. Now in rechecking each of the modules after doing a cumulusutils.exe Website run all the updates are on the website www.komokaweather.com/cumulusutils/index.html
My reason for concluding (wrongly) the files were not being FTP/updated was looking at my website files using Filezilla and the files had the old time and date.

However, there was still an underlining issue with my Windrose module, and amongst the many lines in the cumulusutils.txt log file it identified my Oct08log file. I checked this file and found the data was tab delimited for this file rather then comma. Don't know how that occurred and hadn't come up on any other program or script I have been using, but the good thing is that now that has been corrected.

The moral of this post is, as often mentioned in forum replies to support questions, to check your data, and check again, and again :oops:


Enjoy,
Paul

sfws
Posts: 931
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: --
Operating System: --

Re: Cumulus data files changes

Post by sfws »

"Empty"
Last edited by sfws on Thu 21 May 2020 4:53 pm, edited 1 time in total.

User avatar
HansR
Posts: 1553
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: Cumulus data files changes

Post by HansR »

PaulMy wrote:
Wed 20 May 2020 3:21 pm
Running the latest Cumulusutils has highlighted the changes over time in the Cumulus data files and the impact on the modules that HansR has implemented - after considerable trouble shooting :oops: :o .

I have Cumulus data from October 2008. I have been interchanging the data files between Cumulus1 install and CumulusMX, ignoring the fact that the old data files did not have all the data fields that are in later Cumulus1 and now CumulusMX versions. Generally that has not been an issue even though the source for Cumulus1 is with Davis Envoy/USB logger, and CumulusMX with Davis Vue and WiFiLogger so after interchanging the data files are a combination Envoy/USB and Vue/WiFiLogger.

A few bad data issues were discovered in dayfile and monthlog files when running Cumulusutils' pwsFWI, dayrecord and Yadr modules and these were quickly found and corrected after reading the cumulusutils.txt log file.

Now the big one was the introduction of the Windrose module as a beta. Eventually Hans was able to identify the issue which was that my oldest files did not have all the data fields that are in the current version of Cumulus1 i.e. Wind bearing. Hans made changes and that issue is now overcome.

etc...
Paul, thank you for the extensive description of the work we are into at the moment. I will explain initial findings below and will do that too with research results.
PaulMy wrote:
Wed 20 May 2020 3:21 pm
The moral of this post is, as often mentioned in forum replies to support questions, to check your data, and check again, and again
Yes, data is inherently up to the owners to make sure it is correct and uniform. But I have two additional remarks to that:
  1. Beside the user it is the data generation application (i.e. Cumulus) responsibility to make sure the data remains consistent over time. I think in this the usage of the locale, from the machine Cumulus runs on, has proven to be more or less a problem. And it still is I guess.
  2. I think CumulusUtils can help the user maintain the data. I already implemented a rudimentary CheckOnly commanline parameter to only check the dayfile. I may extend this to the Monthly logfiles and will be thinking about repair and conversion options to facilitate still existing locale issues (as far as I know, Cumulus still can't work with some - e.g. Slavic - locales)
https://meteo-wagenborgen.nl / Cumulus 3.9.1 (build: 3096)
Davis Vantage Pro 2+ / Raspberry Pi 3 Model B Plus Rev 1.3
Raspbian Buster / 5.4.51-v7+ armv7l
Mono JIT compiler version 6.12.0.90 (tarball Fri Sep 4 14:21:54 UTC 2020)

User avatar
HansR
Posts: 1553
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: Cumulus data files changes

Post by HansR »

sfws wrote:
Wed 20 May 2020 5:41 pm
By the way, what I say are observations about my scripts, and NOT related to those by anybody else!
@sfws: This last sentence is correct and a kind of a correction remark to your whole post. What Paul writes refers to the issues we have in relation to CumulusUtils. And apparently you really have no idea what that is and what I am doing there.
  1. The work I started is not to be qualified as script or set of scripts. It is an application in itself which the user controls through commandline parameters.
  2. I don't assume anything about the data but consistency and conformity to the wiki.
  3. I don't assume any encoding, I write out UTF8
  4. I do not , in any way, modify the original datafiles. If I change this behaviour in some kind of data correction command in future, that would require an absolute, never implicit, confirmation of the user.
  5. I think you should not say what you say about an application you clearly did not use or even see and of which you clearly do not understand its function.
I highly appreciate you knowledge and efforts for the Wiki and recognise your effort in script making (PHP mostly if I see it well).
But make sure where your limits are.

Regards, Hans
https://meteo-wagenborgen.nl / Cumulus 3.9.1 (build: 3096)
Davis Vantage Pro 2+ / Raspberry Pi 3 Model B Plus Rev 1.3
Raspbian Buster / 5.4.51-v7+ armv7l
Mono JIT compiler version 6.12.0.90 (tarball Fri Sep 4 14:21:54 UTC 2020)

sfws
Posts: 931
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: --
Operating System: --

Re: Cumulus data files changes

Post by sfws »

"Empty"
Last edited by sfws on Thu 21 May 2020 4:53 pm, edited 1 time in total.

User avatar
HansR
Posts: 1553
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: Cumulus data files changes

Post by HansR »

sfws wrote:
Thu 21 May 2020 11:52 am
Once again I stress, none of my last post was directed at you or your application.
So why do you respond in a CumulusUtils thread, in the CumulusUtils forum, saying something to the initial topic which you claim is not about it at all?
It may be valid what you wrote, but placing it here, stating it is not directed at CumulusUtils, makes it irrelevant. And because of the placement you trigger me, as builder of CumulusUtils and as moderator of this subforum.
Don't reply, I will remove any subsequent answers on this.
End of discussion.
https://meteo-wagenborgen.nl / Cumulus 3.9.1 (build: 3096)
Davis Vantage Pro 2+ / Raspberry Pi 3 Model B Plus Rev 1.3
Raspbian Buster / 5.4.51-v7+ armv7l
Mono JIT compiler version 6.12.0.90 (tarball Fri Sep 4 14:21:54 UTC 2020)

water01
Posts: 2004
Joined: Sat 13 Aug 2011 9:33 am
Weather Station: WH1081
Operating System: Windows 10 64bit
Location: Burnham-on-Sea
Contact:

Re: Cumulus data files changes

Post by water01 »

HansR I think sfws was trying to apologise for any misunderstanding he may have caused in his post. I think perhaps it didn't come over like that in translation, but I am sure he meant no offence.
David
Image

User avatar
HansR
Posts: 1553
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: Cumulus data files changes

Post by HansR »

So, to continue on the issue of the datafiles, I continue on the November 2008 monthly logfile
(from the Cumulus 1 era, more specific during the development phase of Cumulus 1).

I find the following errors:

Then Nov08log.txt has multiple errors in different fields. I show one example of each error (both occurring in blocks of roughly 28 records, I stopped counting):

Code: Select all

2020-05-19 20:55:34.251 ReadMonthlyLogs: reading data/Nov08log.txt
2020-05-19 20:55:34.257 MonthfileValue.SetValues fail: String was not recognized as a valid DateTime.
2020-05-19 20:55:34.257 MonthfileValue.SetValues: in field nr 1 (thisTime)
2020-05-19 20:55:34.257 DayfileValue.SetValues : Continuing to read data

Code: Select all

2020-05-19 20:55:34.262 MonthfileValue.SetValues fail: Index was outside the bounds of the array.
2020-05-19 20:55:34.262 MonthfileValue.SetValues: in field nr 14 does  not exist in this file data/copy_Nov08log.txt
2020-05-19 20:55:34.262 MonthfileValue.SetValues: Using fallback.
I give the complete record here:

Code: Select all

01/11/08,0:00,6.6,92,4.8,0,1.6,248,0,0,1025.1,280.6
Note that the 13th of this month, the error of the invalid DateTime disappears, I assume because of installation of a new version of Cumulus 1. The invalid date means that the leading 0 for the registered time is missing. Because the date scan in C# requires strict format it fails with the missing digit. That should be correctable by software one way or another.

It is clear from the record itself that there are only 12 fields, so this is a datafile from before version 1.8.5 (see Cumulus Versions) of which I say in the README:
Any datafile made by a version before 1.8.5 is not documented and thus the software cannot handle those and CumulusUtils may crash. I assume such very old monthly logs are rare in the wild.1
Nice to see rare birds still exist.

I propose the following solution: skip any datafile where record length < 18 (pre-version 1.8.5). This probably means the field 1 issue (invalid date) is solved as well. If it is not, we will deal with it then.

There is one side effect here, @laulau has a major data series because he converted from prior non-cumulus installation to Cumulus. The result might be that his datafile could have record length large enough to skip the check, but still would not hold data in the "Cumulus 'Latest' gust" field. Need to check this one. I guess he would be the only one, but because such long series is valuable (and thus interesting) and rare, I will take it if the field is valid.

Summary: pre-version 1.8.5 datafiles cannot be handled in the WindRose context because of lack of good data. The one known existing exception may get special treatment.
https://meteo-wagenborgen.nl / Cumulus 3.9.1 (build: 3096)
Davis Vantage Pro 2+ / Raspberry Pi 3 Model B Plus Rev 1.3
Raspbian Buster / 5.4.51-v7+ armv7l
Mono JIT compiler version 6.12.0.90 (tarball Fri Sep 4 14:21:54 UTC 2020)

User avatar
HansR
Posts: 1553
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: Cumulus data files changes

Post by HansR »

Laurent promised me his full dataset as well so I'll look at that separately.

The Problems left are in the files of March 2011 - Nov 2011 giving the following error:

Code: Select all

2020-05-19 20:55:30.952 ReadMonthlyLogs: reading data/Apr11log.txt
2020-05-19 20:55:30.960 MonthfileValue.SetValues fail: Index was outside the bounds of the array.
2020-05-19 20:55:30.960 MonthfileValue.SetValues: in field nr 24 does  not exist in this file data/copy_Apr11log.txt
2020-05-19 20:55:30.960 MonthfileValue.SetValues: Using fallback.
I provided for a fall back when the Current Wind Bearing field does not exist in the record. So any file from before version 1.9.2 would need the fallback to the average wind bearing. However in 2011 we had a few months where we had exactly version 1.9.2 and because I combine field 0 and 1 to one date, I made an algorithmic error and it failed on exactly the Current Wind Bearing Field. Unfortunate but easy to correct. Consequences of having the progression of development in the datafiles.

There was one other error message as well in the Oct11log.txt:

Code: Select all

2020-05-22 12:13:14.192 ReadMonthlyLogs: reading data/Oct11log.txt
2020-05-22 12:13:19.376 MonthfileValue.SetValues fail: De indeling van de invoertekenreeks is onjuist.
2020-05-22 12:13:19.377 MonthfileValue.SetValues: in field nr 24 (CurrWindBearing)
2020-05-22 12:13:19.382 MonthfileValue.SetValues: Field CurrWindBearing is Empty
which originates in the record of 3 o'clock at night (I give the next record as well for comparison):

Code: Select all

02/10/11,03:00,2.4,93,1.4,0.0,0.0,0,0.0,0.0,1018.93,658.6,22.0,45,0.0,2.4,2.4,0.0,0,0.00,0.00,0.6,0,0.0,
02/10/11,03:30,2.4,93,1.4,0.0,0.0,0,0.0,0.0,1018.83,658.6,22.0,46,0.0,2.4,2.4,0.0,0,0.00,0.00,0.6,0,0.0,0
We see that at 3h00 there is a comma ending the record iso a number. So here the error message is correct you need to correct the data
(meaning add a zero at the end of the 3h00 line)

I'll push out a fix to accommodate the correct handling of different record lengths which have occurred during development of Cumulus.
Nice to see the development reflected in these records btw. And well described in the Wiki.
https://meteo-wagenborgen.nl / Cumulus 3.9.1 (build: 3096)
Davis Vantage Pro 2+ / Raspberry Pi 3 Model B Plus Rev 1.3
Raspbian Buster / 5.4.51-v7+ armv7l
Mono JIT compiler version 6.12.0.90 (tarball Fri Sep 4 14:21:54 UTC 2020)

User avatar
HansR
Posts: 1553
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: Cumulus data files changes

Post by HansR »

OK. I think I have the reading of the monthly files secure, even the converted dataset by @laulau. Starting in 2002 he has datafiles of 16 fields meaning he assumed a pre-1.8.5 release of CM1. Then from 2008 onwards the number of fields start increasing, seemingly in agreement with the CM1 versions as they were released. In 2013 all 27 fields are there and in 2016 the switch is made to CMX. Some errors are found but minimal and easy to correct. Interesting conversion.

Working with the data sets of Paul and Laurent was a nice exercise. Thanks guys. :D :clap:

Release 3.6.6 coming up soon.
https://meteo-wagenborgen.nl / Cumulus 3.9.1 (build: 3096)
Davis Vantage Pro 2+ / Raspberry Pi 3 Model B Plus Rev 1.3
Raspbian Buster / 5.4.51-v7+ armv7l
Mono JIT compiler version 6.12.0.90 (tarball Fri Sep 4 14:21:54 UTC 2020)

User avatar
PaulMy
Posts: 2306
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2+ Cumulus1 / CummulusMX
Operating System: Windows8 / Windows10
Location: Komoka, ON Canada
Contact:

Re: Cumulus data files changes

Post by PaulMy »

By George Hans, I think you've got it!

After another little ,0 fix in one monthly log file, no more issues :clap:

A good lesson on data integrity as well.

Enjoy,
Paul

Post Reply