Page 1 of 1

Bug still with 1st of each month in v1.1.0: (BUG: Create Missing Version 1.0.2)

Posted: Tue 08 Jun 2021 7:05 pm
by sfws
I recently found the middle of my Jun20log.txt file was corrupted on my RPi. I am unsure why, perhaps because June was when I was experimenting with MX.

Luckily, I had a full replacement made when I was testing my "Create Missing php script" last year, so my RPi standard log files are now all correct.

Today I thought I would try out your recent CreateMissing.exe to see whether it would find any updates were needed in my dayfile.txt.


Bug:
After running CreateMissing.exe, I have had to go back to the older dayfile.txt as your program has removed one line from every month in the dayfile.txt where other dates in the month had feels like added, before dates when humidex added.

The output says for example
Date: 01/02/2020 : Adding missing data ...
01/02/2020 : No monthly data was found, not updating this record
...
Date: 01/07/2020 : Adding missing data ...
01/07/2020 : No monthly data was found, not updating this record
on the first of every month. That message is wrong, the monthly data does correctly exist in the standard log file for each of those starts of a new month (so it should have found it), also your program has removed from dayfile.txt the existing line for the first of every month in that period (so it has done an update).

However, from August onwards (after you added Humidex to MX) I was running CumulusMX.exe that had stored a complete line in dayfile.txt correctly, and so your new program reports
Date: 01/08/2020 : Entry is OK
...
Date: 01/06/2021 : Entry is OK
and indeed the line for those first of the month is left in dayfile.txt for all those months.

Observations:
1) I did discover that because I ran this on my Microsoft Windows PC accessing my RPi files, the original and revised files also had different end of line characters (the old file has LF as end of line because it is creating using Linux variant, the new file has CRLF as end of line because I ran it using an emulator on my PC), but that is something I did not think about, not a bug by you.
2) I also noticed that while Cumulus 1 truncated trailing zeroes in decimal places when storing values in the file, your newer program is storing all decimals to a fixed number of decimal places.

EDIT: The title of this thread has been changed, basically to confirm that the newer version of CreateMissing (v.1.1.0) still has the bug that it does not amend any dayfile line that is the first of the month, (? maybe because I use 9am/10am rollover ?). However, the new version of CreateMissing.exe does NOT remove the entry for those earlier months where previous version removed those lines, so there is some improvement. Of course another difference is that the new version adds both Humidex and Cumulative Chill Hours to every line, although because it skips the first of the month, the Chill Hour figures it adds miss one day each month!

Re: BUG: Create Missing Version 1.0.2

Posted: Tue 08 Jun 2021 7:22 pm
by mcrossley
Could you zip up the data files in question for me, and I'll take a look. I've already been amending Create Missing for the next release. Thanks.

Re: BUG: Create Missing Version 1.0.2

Posted: Tue 08 Jun 2021 7:33 pm
by sfws
mcrossley wrote: Tue 08 Jun 2021 7:22 pm Could you zip up the data files in question for me
Here you are, I hope I have successfully zipped up files on my RPi, I have not done that before!

Re: BUG: Create Missing Version 1.0.2

Posted: Tue 08 Jun 2021 10:13 pm
by mcrossley
Thanks I'll take a look tomorrow....

Re: BUG: Create Missing Version 1.0.2

Posted: Tue 08 Jun 2021 10:45 pm
by mcrossley
That zip seems to be just a dayfile.txt ?

If it is, it seems to have duplicate data in it, the dates start again.

I'll need all your data files to run them through CreateMissing as you have done.

Re: BUG: Create Missing Version 1.0.2

Posted: Wed 09 Jun 2021 5:53 am
by sfws
Maybe my files need some explanation?
mcrossley wrote: Tue 08 Jun 2021 10:45 pm That zip seems to be just a dayfile.txt ?
I suspect you used a Microsoft Windows machine to read my RPi created zip in its original compression format, and that created problems for you.

I have repeated my compress action on my Raspberry Pi, choosing .zip format this time, and the new file is now in previous post.

I have checked it contains correct files when viewed on either a Raspberry Pi or a Microsoft Windows PC
mcrossley wrote: Tue 08 Jun 2021 10:45 pm If it is, it seems to have duplicate data in it, the dates start again.
My dayfile.txt has no duplicates, but there were 2 dayfile.txt files in the zip, and maybe that confused you? One is dayfile.txt.sav that your program processed, and the other is dayfile(error).txt that is a renamed copy of what it created.

Originally my dayfile.txt contained all my data from 2009 onwards. However, when you introduced MX reading the whole file on start up, to speed processing up, I removed the data prior to 19 January 2020, and that is the file I processed.

[Ignore the data placed in 1 January 2020 of my dayfile.txt, that represents extremes and sums before my current Cumulus start data of 19 January 2020].
mcrossley wrote: Tue 08 Jun 2021 10:45 pmI'll need all your data files to run them through CreateMissing as you have done.
The zip does not include the complete contents of /opt/CumulusMX/data, but it does include every XXX20log.txt and YYY21log.txt, so zip does include every file your program processed.

Re: BUG: Create Missing Version 1.0.2

Posted: Wed 09 Jun 2021 2:12 pm
by mcrossley
Thanks, 7zip seems to just lump all the archive contents together into a single text file!

Anyway, I have found the problem in Create Missing using your zipped data. It will be fixed in the next release which will be released concurrently with the next version of Cumulus MX.