Very Basic Stuff

[HansR]

Sorry to butt in but since my Extra Sensor data and graphs were mentioned I've had a look and see that both AirLink and Ecowitt Extra Sensors are out of date since last October

Both http://komokaweather.com/cumulusmx/index.html and http://komokaweather.com/cumulusmxwll/index.html (with beta v6.8.4) use the Davis AirLink. And one uses the Ecowitt 433MHz WH41 and the other the 915MHz WH41B.

One thing which I note is that the compiler charts are generated with Cutils 6.7.7 while the websitre is 6.8.3 and 6.8.4 beta. This is caused by not having a full run after the update (so without thrifty). I advise to do that manually after the update so that the runtime system and other generated code (like the charts) are all upgraded as well (so the update is more than just copying the code, yes, true. I always assumed that was well known it is required to do).

Also necessary in an update (but you once confirmed that you always do that) is that you copy the whole bin directory of the distribution to the bin directory of your utils directory. Only when I send a patch (like 6.8.4 beta) you can deviate from that.

So when all that is done and it still does not work, than please send me the log file of the full run. Let's stick to one of the installs to avoid confusion. Your choice.

I run cumulusutils.exe with UserAskedData daily but now trying by Interval (5-minute) and see that the FTP fails. It has been quite a while since I've done any settings or changes and have likely forgotten what I need to do.

The interval frequency of 5 minutes should not matter, any frequency higher than 1 should do (whether it is useful to have a high frequency is another thing). We will have to look into the FTP one step at a time so if you updated the software correctly and run

Code: Select all

utils/bin/cumulusutils.exe website

and it still does not work, please send the logfile for the UserAskedData run.

The trends in the CumulusMX default template are current http://komokaweather.com/cumulusmx/tren ... airquality

OK, so at least CMX is running perfectly

[HansR]

Sorry, my bad: had the time comparison inverted so the beta is the same as it was.
Attached the new beta 3. Hope that is better.

[PaulMy]

Hi Hans, and thanks for the detailed review.
I always do an all files/folder full new version update (over top of existing) in to \CumulusMXwll\utils except some beta like the two new files to \CumulusMXwll\utils\bin

I do a CMX External Program
Daily program: \CumulusMXwll\utils\bin\cumulusutils.exe
Daily program parameters: website

When a new version is installed I always do a \CumulusMXwll\utils\bin\cumulusutils.exe website
I had not used Thrifty

I have now done a full v6.8.3 install, and then the v6.8.4 two bin files.

I then did a run \CumulusMXwll\utils\bin\cumulusutils.exe website and logfile 2022060913:54 attached
I then also did run \CumulusMXwll\utils\bin\cumulusutils.exe Thrifty website and logfile 2022060913:59 attached
I also did run \CumulusMXwll\utils\bin\cumulusutils.exe UserAskedData and logfile 202206091430 attached

The http://komokaweather.com/cumulusmxwll/index.html site Extra are now more current but not to the time that either parameter Thrifty website or UserAskedData was actually run.

The attached is more than what you asked about, but highlighted showing the update files and that I have older accumulated files in the \utils folder and presume that does not impact the runs.

Enjoy,
Paul

[HansR]

Hi Paul,

OK, thanks. That is thorough, I'll take it one step at a time, probably over more reply's and please note that from tomorrow I'll be away for two days so I'll be fully available again at the end of Saturday.

[ 1 ]
The compiler output is still indicated by version 6.7.7. That is caused by an error in CutilsCharts.def. I advise to take the CutilsCharts-default-for-use.def from the distribution, rename it to CutilsCharts.def and copy it to the utils directory, overwriting the erroneous one. You probably missed some compiler changes in the past, afaics notably the keyword for SolarRadiation change.

[ 2 ]
In the logfile for UserAskedData it appears that it says:

Code: Select all

2022-06-09 14:30:04.913 ReadAirlinklog: start.
2022-06-09 14:30:04.913 AirLinklog: timeStart = 2022-06-06 2:30:04 PM; timeEnd = 2022-06-09 2:30:04 PM
2022-06-09 14:30:04.913 AirLinklog: Require data/AirLink202206log.txt to start
2022-06-09 14:30:04.967 Information Airlinklog : EOF detected
2022-06-09 14:30:04.967 AirLinklog: Require the  next logfile: data/AirLink202206log.txt
[ ad infinitum ]

until it fails with:
2022-06-09 14:31:51.726 Error Exception Unknown : Exception of type 'System.OutOfMemoryException' was thrown.

This means there is a problem in reading the logfile.

Could you post your data/AirLink202206log.txt so that I can check and debug? Quite strange because all others with the AirLink run well and it is not code which has been changed (apart now this beta change for the 9h00 rollover).

Regards,

[PaulMy]

Hi Hans,
Made the CutilsCharts.def file change and did a \CumulusMXwll\utils\bin\cumulus.exe website run and the log file attached. Also Airlink202206.txt
To my surprise there are a lot of errors shown as I thought I had sorted the \data files a long time ago during the initial phases of CUtils.

Enjoy your time away,
Paul

[HansR]

Hi Paul,

Made the CutilsCharts.def file change and did a \CumulusMXwll\utils\bin\cumulus.exe website run and the log file attached.

OK, so that is working now.

Also Airlink202206.txt

The logfile is indicating that the datafile is empty which clearly is not the case. But CUtils always makes a copy of the original datafile to work on so it does not disturb CMX. This means that the copy should be empty. That is difficult to check because the copy is deleted after reading the data. So I created a beta which does not delete the copy. I suggest the following:

1. Remove all 'copy_*' files from the data directory so that no mistakes can be made. Before you do: check if a copy_AirLink202206log.txt does exist, if it exists: check if it is empty)
2. Use the attached beta 4 and run UserAskedData
3. When the run is complete check for the copy_AirLink202206log.txt to exist, if it exists if it is empty.
4. Post the corresponding logfile

To my surprise there are a lot of errors shown as I thought I had sorted the \data files a long time ago during the initial phases of CUtils.

Yes, we had it sorted out during the initial phase of CUtils but we could not add data that did not exist so that remained an issue. As a result Cutils generates the errors max ten times and simply continues. CreateMissing is only for dayfile. I just think this will be there forever: the really old versions of the datafiles simply do not have the data. You might fill in some zero's for the missing fields but it does not really change anything I guess. I would leave it for what it is.

Enjoy your time away,

It's actually visitors which keep me occupied but the effect is the same

[PaulMy]

Hi Hans,

So I created a beta which does not delete the copy. I suggest the following:
Remove all 'copy_*' files from the data directory so that no mistakes can be made. Before you do: check if a copy_AirLink202206log.txt does exist, if it exists: check if it is empty)
Use the attached beta 4 and run UserAskedData
When the run is complete check for the copy_AirLink202206log.txt to exist, if it exists if it is empty.
Post the corresponding logfile

Before the UserAskedData run there was a copy_AirLink202206log.txt file and it was populated for all of June 2022 at 5 minute interval. Also copy_ExtraLog202110.txt and the same for each month up to copy_ExtraLog202206.txt. I deleted (moved to a temp To_delete folder) all of them.
After a UserAskedData run a new copy_AirLink202206.txt exists and populated with all of June 2022 at 5 minute interval. No new copy_ExtraLogxxxxxx.txt were created.

The komokaweather.com/cumulusmxwll/index.html site shows the AirLink and Extra Sensors as current now. So that looks good.

One observation, and that is that the Airlink Graph View Out takes a long time to load and see that each of the airlinkdataOut10.json and airlinkOut2p5.json are now over 70 Mb and perhaps is because this beta does not delete the copy.

Enjoy,
Paul

[MuntyWeather]

Hi Paul I did the Utilscharts,def change and my home graphs are now OK. Loaded beta4 but get infinite loop reading airlink logs as you did. So await wisdom from Hans! Ted

[HansR]

Hi Paul,

So I created a beta which does not delete the copy. I suggest the following:
Remove all 'copy_*' files from the data directory so that no mistakes can be made. Before you do: check if a copy_AirLink202206log.txt does exist, if it exists: check if it is empty)
Use the attached beta 4 and run UserAskedData
When the run is complete check for the copy_AirLink202206log.txt to exist, if it exists if it is empty.
Post the corresponding logfile

Before the UserAskedData run there was a copy_AirLink202206log.txt file and it was populated for all of June 2022 at 5 minute interval. Also copy_ExtraLog202110.txt and the same for each month up to copy_ExtraLog202206.txt. I deleted (moved to a temp To_delete folder) all of them.

OK, that's good.

After a UserAskedData run a new copy_AirLink202206.txt exists and populated with all of June 2022 at 5 minute interval. No new copy_ExtraLogxxxxxx.txt were created.

The ExtraSensors chart indeed has no data up to the current time so that I assume is not correct but apparently the AirLink does run OK.

The komokaweather.com/cumulusmxwll/index.html site shows the AirLink and Extra Sensors as current now. So that looks good.

It may look good but from where I stand the ExtraSensors chart does not show correct data and does not update any longer.
And the AirLink chart indeed is far too slow.

One observation, and that is that the Airlink Graph View Out takes a long time to load and see that each of the airlinkdataOut10.json and airlinkOut2p5.json are now over 70 Mb and perhaps is because this beta does not delete the copy.

I don't think this has to do with the copy of the datafile but with the data.

I tried to analyse the JSON ( notably start and end times of the series) but that was impossible. The file is so big the browsers (both mozilla and chrome) fail to load it and save it and I can't copy it to any editor. Obviously there is something wrong.

So: for the time being, could you set the 7 days to three days and display only one or two series notably the now and the three hour average?
Then if there is an error ate least I can look at it. Always remember that increasing the number of days you wish to see increases the datafiles and so does every series you add. You might hold back a bit on what you wish to display.

@MuntyWeather: OK. it is somewhat similar to Paul's status so OK, let's look one at a time and get back to yours when solved.

[HansR]

btw: don't forget to send it along with the logfile because in that logfile the start and end times of the selected data (transferred to the JSON) are given. Those are important.

[HansR]

@Ted: I checked your site this morning but your charts are not transferred at all. Not the HOME charts and not the AirLink charts (you can switch the extra sensors off: you have none). So in order to have some progress, can you explain:

1. How you run Cutils on the daily basis?
2. How you run Cutils on the interval basis?

And send me the logfile again for both runs (daily and interval). Your issue is really different from Paul's, this must be something not too complex.

[MuntyWeather]

Hi Hans
I had switched CMX off pending your advice. Now on again (13:30 UTC Monday) using beta4. I have switched off 'extra sensors'. Have just run daily (website) run (interactively). Have attached logfiles (I hope)
Regards Ted

[PaulMy]

UIY&*^^U%$%^**&(&^&%
Ok will do again,

I tried things again this morning.
I changed CMX Graph settings to 3 days and only include out-temp, out-hum, sun-hrs

After about 10 minutes to allow CumulusMX to do a couple of interval uploads,
At 9:18 I did a run cumulusutils.exe UserAskedData website
After about 8 minutes I stopped the run
There was no update to website

At 9:34 I did a cumulusutils.exe UserAskedData
It finished after about 3 minutes
There was a website update
The .json files are still very large - 58 Mb each

At 9:44 I did a cumulusutils.exe website
It finished about 9:45
There was a website update.

Enjoy,
Paul

[HansR]

Hi Ted,

1. Can you check the CMX parameter GraphHours (section Graphs) as it tries to create a graph from 19th of May to 13th of June while the HOME graphs only make 10 days (240 hrs). That is strange, the nr of hrs in the charts should be the same.
2. Apparently Cutils makes an error in determining the endTime for the chart and I don't see the error. Could you also give me your Interval period and the Logging Interval (1, 5, 10, 15, 20, 30 minutes) ?
3. I noticed there is still an SSL issue. can you give the command

   Code: Select all

   openssl version

   and post the output here? It probably is 1.1.1 and if it is, could you run

   Code: Select all

   sudo dpkg-reconfigure ca-certificates

   select yes and then, when you see the list of certificates, check mozilla/DST_Root_CA_X3.crt. It should have no asterisk between the square brackets. If it has, please select that line and type space (the asterisk should disappear); type tab to go to OK and press enter. Then wait until you get back to the OS prompt (it takes some 15 - 30 seconds). Please report back whether it had an asterisk or not opening the list.

[HansR]

@Paul, @Ted: I think I solved the issue of the end time for the AirLink charts. Please find attached beta 5.
And please post the resulting log for the UserAskedData.