Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4019) - 03 April 2024

Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Bug still with 1st of each month in v1.1.0: (BUG: Create Missing Version 1.0.2)

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Bug still with 1st of each month in v1.1.0: (BUG: Create Missing Version 1.0.2)

Post by sfws »

I recently found the middle of my Jun20log.txt file was corrupted on my RPi. I am unsure why, perhaps because June was when I was experimenting with MX.

Luckily, I had a full replacement made when I was testing my "Create Missing php script" last year, so my RPi standard log files are now all correct.

Today I thought I would try out your recent CreateMissing.exe to see whether it would find any updates were needed in my dayfile.txt.


Bug:
After running CreateMissing.exe, I have had to go back to the older dayfile.txt as your program has removed one line from every month in the dayfile.txt where other dates in the month had feels like added, before dates when humidex added.

The output says for example
Date: 01/02/2020 : Adding missing data ...
01/02/2020 : No monthly data was found, not updating this record
...
Date: 01/07/2020 : Adding missing data ...
01/07/2020 : No monthly data was found, not updating this record
on the first of every month. That message is wrong, the monthly data does correctly exist in the standard log file for each of those starts of a new month (so it should have found it), also your program has removed from dayfile.txt the existing line for the first of every month in that period (so it has done an update).

However, from August onwards (after you added Humidex to MX) I was running CumulusMX.exe that had stored a complete line in dayfile.txt correctly, and so your new program reports
Date: 01/08/2020 : Entry is OK
...
Date: 01/06/2021 : Entry is OK
and indeed the line for those first of the month is left in dayfile.txt for all those months.

Observations:
1) I did discover that because I ran this on my Microsoft Windows PC accessing my RPi files, the original and revised files also had different end of line characters (the old file has LF as end of line because it is creating using Linux variant, the new file has CRLF as end of line because I ran it using an emulator on my PC), but that is something I did not think about, not a bug by you.
2) I also noticed that while Cumulus 1 truncated trailing zeroes in decimal places when storing values in the file, your newer program is storing all decimals to a fixed number of decimal places.

EDIT: The title of this thread has been changed, basically to confirm that the newer version of CreateMissing (v.1.1.0) still has the bug that it does not amend any dayfile line that is the first of the month, (? maybe because I use 9am/10am rollover ?). However, the new version of CreateMissing.exe does NOT remove the entry for those earlier months where previous version removed those lines, so there is some improvement. Of course another difference is that the new version adds both Humidex and Cumulative Chill Hours to every line, although because it skips the first of the month, the Chill Hour figures it adds miss one day each month!
Last edited by sfws on Sun 24 Oct 2021 1:18 pm, edited 3 times in total.
User avatar
mcrossley
Posts: 12756
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: BUG: Create Missing Version 1.0.2

Post by mcrossley »

Could you zip up the data files in question for me, and I'll take a look. I've already been amending Create Missing for the next release. Thanks.
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: BUG: Create Missing Version 1.0.2

Post by sfws »

mcrossley wrote: Tue 08 Jun 2021 7:22 pm Could you zip up the data files in question for me
Here you are, I hope I have successfully zipped up files on my RPi, I have not done that before!
Last edited by sfws on Wed 09 Jun 2021 2:41 pm, edited 2 times in total.
User avatar
mcrossley
Posts: 12756
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: BUG: Create Missing Version 1.0.2

Post by mcrossley »

Thanks I'll take a look tomorrow....
User avatar
mcrossley
Posts: 12756
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: BUG: Create Missing Version 1.0.2

Post by mcrossley »

That zip seems to be just a dayfile.txt ?

If it is, it seems to have duplicate data in it, the dates start again.

I'll need all your data files to run them through CreateMissing as you have done.
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: BUG: Create Missing Version 1.0.2

Post by sfws »

Maybe my files need some explanation?
mcrossley wrote: Tue 08 Jun 2021 10:45 pm That zip seems to be just a dayfile.txt ?
I suspect you used a Microsoft Windows machine to read my RPi created zip in its original compression format, and that created problems for you.

I have repeated my compress action on my Raspberry Pi, choosing .zip format this time, and the new file is now in previous post.

I have checked it contains correct files when viewed on either a Raspberry Pi or a Microsoft Windows PC
mcrossley wrote: Tue 08 Jun 2021 10:45 pm If it is, it seems to have duplicate data in it, the dates start again.
My dayfile.txt has no duplicates, but there were 2 dayfile.txt files in the zip, and maybe that confused you? One is dayfile.txt.sav that your program processed, and the other is dayfile(error).txt that is a renamed copy of what it created.

Originally my dayfile.txt contained all my data from 2009 onwards. However, when you introduced MX reading the whole file on start up, to speed processing up, I removed the data prior to 19 January 2020, and that is the file I processed.

[Ignore the data placed in 1 January 2020 of my dayfile.txt, that represents extremes and sums before my current Cumulus start data of 19 January 2020].
mcrossley wrote: Tue 08 Jun 2021 10:45 pmI'll need all your data files to run them through CreateMissing as you have done.
The zip does not include the complete contents of /opt/CumulusMX/data, but it does include every XXX20log.txt and YYY21log.txt, so zip does include every file your program processed.
User avatar
mcrossley
Posts: 12756
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: BUG: Create Missing Version 1.0.2

Post by mcrossley »

Thanks, 7zip seems to just lump all the archive contents together into a single text file!

Anyway, I have found the problem in Create Missing using your zipped data. It will be fixed in the next release which will be released concurrently with the next version of Cumulus MX.
Post Reply