Cutils 6.23.0 beta 1

[AndyKF650]

Hi Hans

Looks like you have been busy in the background again.

I loaded to latest download as normal and ran the website command and noticed it ran a lot quicker than normal. On checking the website most things looked good but the customlogs charts were blank. Looking at the utilslog for that run I can see that the date field is asking for a different format against that which worked well previously.

I have attached the log file and a temperature data file for information.

[Dador]

If I may chime in,
I would like to report that the ExtraSensors Top10 Graphs CustomLogs modules are working fine in the latest release.

[HansR]

Looks like you have been busy in the background again.

I loaded to latest download as normal and ran the website command and noticed it ran a lot quicker than normal. On checking the website most things looked good but the customlogs charts were blank. Looking at the utilslog for that run I can see that the date field is asking for a different format against that which worked well previously.

I have attached the log file and a temperature data file for information.

I think I found it. In the opriginal tekst of the CustomLogs Wiki I had a text saying:

NOTE: The formatting of the datafiles is important as the Custom Logs are basically free format: you can put into the file any text and numbers with any separator you wish. CUtils will NOT handle that Free Format in general. The user has to take care the information in the resulting datafiles is as follows:

1) There MUST be at least one space between the numbers;
2) There cannot be any plain text in the logfile
The date/time will be untouched i.e. the format is: dd-mm-yy;hh:mm;

Unfortunately that text has been modified a lot of times and eventually it disappeared.
The way it works now is : the separator is replaced by a space and then the line is split on the spaces to address the individual members. Unfortunately, the removal of two spaces - you use a comma AND a space as separator - is very time consuming using a regex so I removed that, not realising you use two separators. I would advise you to edit your definition of your custom logs to <webtag>,<webtag> etc... to get a single separator. Note that a single space is also allowed but there can be only one (1) separator between the fields just like in the regular logfiles.

To fix the existing logs just edit them - either switch CMX off or edit between the intervals - by replacing all ", "-sequences with ",".

Than try again.

And of course I will adjust the Wiki.

If I may chime in,
I would like to report that the ExtraSensors Top10 Graphs CustomLogs modules are working fine in the latest release.

Thanks

[PaulMy]

In addition you need to be aware of the following:
The simplification of the logfile reading code is not visible for the user but has major impact if there is a bug in it so please check carefully the results of all CUtils output. There should not be any difference between this version and the previous one. Especially where older series are in play (Yes Paul ) this may have an effect. The separators are checked only once for each logfile type. So any change of locale over the years may have some unwanted effects.

I installed and the website runs seemed to go ok, and the result is fine.
I have it run on two separate CMX:
One uploads to www.komokaweather.com/cumulusmx/index.html and this run took about 2:07 mm:ss where the midnight run on previous version took 6:10 mm:ss
The second uploads to www.komokaweather.com/cumulusmxwll/index.html and this took about 2:51 mm:ss where the previous version took 6:25 mm:ss
These are now both with FTP as I still can't use PHP upload due to ISP issue. The previous version CUtils using php upload, when I was able to use it, took less than a minute.

EVT graphs look nice... and noticed that for 2021 there was huge spike April 13th. Using my CU1 data I found the error in my CMX install where it seems there was a stop and restart and that caused the total ET to drop from 149.78 to 0.00 and then next interval back to 149.78
I think I have the data corrected.

Enjoy,
Paul

[HansR]

I installed and the website runs seemed to go ok, and the result is fine.

Good

I have it run on two separate CMX:
One uploads to www.komokaweather.com/cumulusmx/index.html and this run took about 2:07 mm:ss where the midnight run on previous version took 6:10 mm:ss
The second uploads to www.komokaweather.com/cumulusmxwll/index.html and this took about 2:51 mm:ss where the previous version took 6:25 mm:ss

That is what I call significant improvement and makes me a happy man

These are now both with FTP as I still can't use PHP upload due to ISP issue. The previous version CUtils using php upload, when I was able to use it, took less than a minute.

Yes PHP uploads are lightning fast. Hope you can use that again in future.

EVT graphs look nice... and noticed that for 2021 there was huge spike April 13th. Using my CU1 data I found the error in my CMX install where it seems there was a stop and restart and that caused the total ET to drop from 149.78 to 0.00 and then next interval back to 149.78
I think I have the data corrected.

That is a nice spin off of CUtils, there is always some data improvement

Thanks Paul, this is a nice result

Btw... did you look at and get anywhere with the CustomLog charts?

[PaulMy]

Hi Hans,

Btw... did you look at and get anywhere with the CustomLog charts?

Not looked at that yet as too occupied in trying to get ISP and host issues resolved (I emailed host just before my annual hosting renewal in case they had a better plan, but they didn't get back) so auto-renew at the existing plan today ISP technician is stumped and nothing yet from ticket escalation. So call croxyproxy.com nearly every time I want to do a new session on my websites.

Enjoy
Paul

[HansR]

@Paul: very sorry to hear that, really hope you get it resolved soon

[meteo19]

Hi Hans

New version installed and everything works for me.

Best regards,

[AndyKF650]

Hi Hans

After a lengthy data cleanse on the data/customlog files all looks good this morning. I must pay attention to spacings when generating new logs

[HansR]

New version installed and everything works for me.

Good, thanks

After a lengthy data cleanse on the data/customlog files all looks good this morning. I must pay attention to spacings when generating new logs

Good and thanks.
And I must pay attention to the Wiki/Instruction/Modifications, sorry for this one

[Dador]

I noticed that just after midnight the Cutils modules start up on their own. I have tasks set in my schedule to run after 1 a.m. Interestingly, this only happens for Cutils, where I have the CustomLogs module. In the second instance (for Ecowitt stations) it works normally.

Hans, where does Cutils get information about the annual rainfall norm? Here there is also a difference between cutils for Davis and cutils for Ecowitt. For the Davis station I have a line with the rainfall standard on the charts, for the Ecowitt station I do not have this line. My Cutils settings for both stations are practically the same. Hmmm....

[HansR]

I noticed that just after midnight the Cutils modules start up on their own. I have tasks set in my schedule to run after 1 a.m. Interestingly, this only happens for Cutils, where I have the CustomLogs module. In the second instance (for Ecowitt stations) it works normally.

Totally impossible: CUtils does not start itself, it must be some residual entry in your scheduler (or in CMX?).

Hans, where does Cutils get information about the annual rainfall norm? Here there is also a difference between cutils for Davis and cutils for Ecowitt. For the Davis station I have a line with the rainfall standard on the charts, for the Ecowitt station I do not have this line. My Cutils settings for both stations are practically the same. Hmmm....

The Normal Annual Rainfall is calculated as the sum of the NOAA parameters for rain in Cumulus.ini (so e.g. NOAARainNormJan). So if you spot a difference between two configurations, that would be where to look.

If you are missing the reference line on one of the charts, look at the setting of the UseNormalRainReference parameter in cumulusutils.ini. That parameter can have the setting Normal, Station or Both. Note that the temperature has a similar setting!

[Stigian]

Hi Hans,

Just installed and looking good

[HansR]

Hi Hans,

Just installed and looking good

Thanks!

[Dador]

I don't want to start a new thread, so I'll write here.

Can I ask you to add a translation option for Limit values, Winter, Spring, Summer, Autumn, which are used in Seasons.