Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4018) - 28 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Updated from Cumulus 1 and can't read dayfile.txt

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
User avatar
Ned
Posts: 258
Joined: Mon 19 Jul 2010 11:15 am
Weather Station: WS2083 (aspirated)
Operating System: Win 10
Location: Auckland NZ

Updated from Cumulus 1 and can't read dayfile.txt

Post by Ned »

After 10 years on Cumulus 1 I decided to try MX. Installed easily enough and I copied 10 years of data over into the new folders, as well as cumulus.ini.

First problem:
MX won't show rainfall for the current month or year and won't display temps in the chart for the past 31 days.
The MXdiags.txt file reports data formating errors in the first 10 lines of dayfile.txt, then gives up.
Here's the very first entry from 19 July 2010:
19/07/10,9.7,45,14:15,9.5,23:12,15.0,13:50,1012.5,14:45,1023.6,22:32,0.0,00:00,0.0,11.8,0.6,3.6,14:24,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Is it the date format? Editing thousands of lines is out of the question.... :bash:
(The date separators changed from / to - in 2016)

Second problem:
The current outside temperature is showing 3 degrees lower than on the Fine Offset console (they always agree in Cumulus 1)
There is no offset or multiplier corrections set for temperature.
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Updated from Cumulus 1 and can't read dayfile.txt

Post by sfws »

Ned wrote: Thu 30 Jul 2020 7:30 am First problem:
MX won't show rainfall for the current month or year and won't display temps in the chart for the past 31 days.
The MXdiags.txt file reports data formatting errors in the first 10 lines of dayfile.txt, then gives up.
Yes, MX is more fussy about locale than Cumulus 1. Date formats are in this locale definition and must therefore be consistent throughout the file.

Cumulus (1 or MX) reads the entire dayfile.txt just to get the 30 or so days needed for this month rainfall and charts, and however many days needed for the rain season. While Cumulus 1 will accept any character (except space or a digit) as a separator for the date parts, MX will only accept the character defined in your locale (for your windows 10 pc that is actually set in Control Panel which windows does not make easy to access preferring you to use its Settings app. Anyway there is a Clock and Region section in Control Panel and in the Region window you define the separator MX will use in the Short date item, most easily by clicking additional settings). You don't need to change this as from what you say it is now set to "-".
Ned wrote: Thu 30 Jul 2020 7:30 am Editing thousands of lines is out of the question
Actually, if on your PC you use something like Notepad++, there are many suitable editors, a global replace of "/" into "-" is easy as the "/" only appears in those 2010 to 2016 dates. The other way round would be harder to do as "-" might be a valid minus sign in any number field.

The basic Notepad that comes with Windows has a Replace option in its Edit menu, but it (and various Google editors) are less safe as they can save files in wrong encoding and so you would need to use "save as" and pick "UTF-8" encoding (if that is not already shown) to ensure MX can read the output.
Ned wrote: Thu 30 Jul 2020 7:30 am Second problem:
The current outside temperature is showing 3 degrees lower than on the Fine Offset console (they always agree in Cumulus 1)
There is no offset or multiplier corrections set for temperature.
The code that reads data from a Fine Offset weather station is not identical in Cumulus 1 and MX. The memory map by Jim Easterbrook that Steve Loft used for Cumulus 1 is not perfect. MX now handles reading pressure correctly where Cumulus 1 did not.
For your temperature difference however, that must be something about your settings (be aware that not all settings in the configuration file appear on the station settings page), I am sure it is not a MX bug. I have a Fine Offset and its console and MX are currently in agreement and I have always seen agreement any time I have looked.
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Updated from Cumulus 1 and can't read dayfile.txt

Post by mcrossley »

Cross posted with @sfws...

It sounds like your date format changed in 2016.

You are going to have to edit your day file to fix this. But you do not need to edit all those lines, you can just do a global replace.

First make sure you have a decent text editor - install Notepad++ or similar if you do not have it already.

Make a backup copy of your Dayfile.txt file.

Open your Dayfile.txt in Notepad++, press Ctrl-H, in the search for box enter "/", in the replace enter "-" (both without the quotes), and click Replace all (I don't have Notepad++ installed here to check that name, but it will be something similar).

Save the file - job done.

As for the temperature difference, I think there must be something hidden in the configuration causing this.

Start Cumulus MX using the -debug command parameter, and then zip and attach the MXdiags folder so we can take a look.
User avatar
Ned
Posts: 258
Joined: Mon 19 Jul 2010 11:15 am
Weather Station: WS2083 (aspirated)
Operating System: Win 10
Location: Auckland NZ

Re: Updated from Cumulus 1 and can't read dayfile.txt

Post by Ned »

Thanks guys for the help....
I corrected the earlier dates to the - separators using Notepad++, easy :D
Then I checked the Windows date format in Control Panel/Region, noticed it was d/MM/yyyy so rightly or wrongly changed it to d/MM/yy which is how Cumulus 1 had been recording dates.
Started MX again, but it still had the errors with dayfile.txt from the start, and in addition couldn't read the current month file, so charts are now all blank.
After that, I changed the date format back to where it was and started MX again, with same errors.
Repeated the swap cycle yet again, including a PC reboot, but no joy :x

I've gone back to Cumulus 1, at least it's not fussy datewise.

BTW, The temperature anomaly disappeared without me changing anything.

MXdIags attached. A whole lot of Fine Offset error rubbish in the first two, had to reboot the console. I wasn't sure how to apply the debug parameter though
You do not have the required permissions to view the files attached to this post.
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Updated from Cumulus 1 and can't read dayfile.txt

Post by sfws »

sfws wrote: Thu 30 Jul 2020 8:18 am You don't need to change this as from what you say it is now set to "-".
Ned wrote: Thu 30 Jul 2020 9:58 am wrongly changed it to d/MM/yy
Apologies if I was unclear. I meant the windows short date determines the separator. It does not matter about the number of "y" in the windows setting. The log file can only accept 2 digits in each part.

If your Windows date format now uses "/" then that clashes with all your dayfile lines having "-".
Edit the Windows date format to use "-" as separator and you should be good to go with MX and to gain all the extra functionality it can offer (although it does not include all that Cumulus 1 can do, MX can do a lot more in other areas). I explained previously why you don't try to edit the log file to use "/". Stay with Cumulus 1 if you like, it will continue to work, but maybe have another attempt with MX?

I won't bother to look at your diags, but Mark may. Glad the anomaly has gone away.

Instructions re debug at https://cumuluswiki.org/a/What_to_do_wh ... em_with_MX
User avatar
Ned
Posts: 258
Joined: Mon 19 Jul 2010 11:15 am
Weather Station: WS2083 (aspirated)
Operating System: Win 10
Location: Auckland NZ

Re: Updated from Cumulus 1 and can't read dayfile.txt

Post by Ned »

Thank you sfws, I must have been having a senior moment and overlooked Windows was using / instead of - separators. After fixing that and correcting some date entries in the current month, MX opened without file errors and all charts are displaying as they should.
That just leaves a problem of this month's and this year's rain showing zero. Maybe I can edit them....
Cheers
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Updated from Cumulus 1 and can't read dayfile.txt

Post by sfws »

Ned wrote: Thu 30 Jul 2020 11:25 amThank you sfws,
That just leaves a problem of this month's and this year's rain showing zero.
Thank you for thanks, glad you have MX mostly working.
Assuming that you have had some rain this year, your remaining problem is puzzling, but I suppose rainfall is different to other parameters in being derived partly from a counter and partly from reading daily summary log.

Now you have MX working you can in the admin interface go to Data logs menu and choose Dayfile. Use the page navigator to move to near end of file so you can see some July 2020 entries are actually showing non-zero numbers in the Total rainfall column of that table. It might also indicate any other issues with fields in a line of the file, you say you have done some date corrections in current month, this might help you see if you mucked up other fields in those lines.

For lines that were created by Cumulus 1, there are a minimum of 15 fields and a maximum of 46 fields.
MX always has 52 fields, and it does strange things like inserting spaces in some (not all) of the otherwise empty fields if you either view or edit a line, so don't worry about that.


To investigate further, we need to see what MX is doing (as Mark's post said):
The last log you posted had

Code: Select all

2020-07-30 20:56:54.829 GetRainfallTotals: Error on line 1 of dayfile.txt: Input string was not in a correct format.

That should be fixed now, so switch on debug and data logging as per https://cumuluswiki.org/a/What_to_do_wh ... em_with_MX.
Turning on both extra logging options will mean the log file in MXDiags will record both reading rain counter from your station and output (even if it is not raining) calculations for rainfall. Next, if you can open the more recent file in MXDiags that is currently updating, preferably in read-only mode, and look at it yourself you may find the problem...

If you cannot see the problem, then leave those options on for a while before you do another zip and post it here.
Of course you could leave that running until it is August so it includes an end of month rollover, but we should not need to see that.
User avatar
Ned
Posts: 258
Joined: Mon 19 Jul 2010 11:15 am
Weather Station: WS2083 (aspirated)
Operating System: Win 10
Location: Auckland NZ

Re: Updated from Cumulus 1 and can't read dayfile.txt

Post by Ned »

Thanks again for your detailed advice, much appreciated.
Starting a new day, MX commenced showing rainfall for the month and the year, but recorded a staggering amount for yesterday, a negative amount of over 9000mm! Unreal unrain!
Swapping back to Cumulus 1 (running in a different folder) there was no rogue value, just 1.8mm for yesterday.
So going back to MX I opened today.ini and corrected the offending value for yesterday's rain. That fixed the issue and all rain amounts are now good.

After that I noticed I was getting several extended instances of wind gusts of 92.5 km/h, on a day of gentle breezes (the console and Cumulus 1 only showed sensible values).
Seems the old software was doing a better job of rejecting the rubbish from a 10 year old Fine Offset station, which seemed to be going nuts even after rebooting.

Finally I replaced the console and transmitter with a newer set that I had in storage, everything settled down and MX and me are finally happy :)
Post Reply