Home Button was suddenly not refreshing the charts; root cause found and issue resolved via MonthsOfMiracleAndWonder

[lse-cumulus]

I am using cumulusUtil v6.15.3. and I have been adding additional user defined charts to the CutilsCharts.def.

I ran into a problem I could not find a solution for in the forum so far: the Home Button was suddenly not refreshing the charts and also during startup of the webpage the dashboard was shown but no chart in the report panel on the right side. I had to manually select a chart from the dropdown or create an event on the associated tile in the dashboard on the left to produce a chart on the right panel.

After some experiments I think a found the root cause: If there are too many charts (> 19) defined in the Cutilscharts.def for the output file cumuluscharts.txt the home button stops to work and the refresh problem as described above occurs.

I solved it by commenting out some charts in my Cutilscharts.def and reduced the number of active charts in the home charts drop down menue so that no longer a scrollbar is shown.

regards leo
my Website: http://leonas.ddns.net/Wetterstation.Su ... CumulusMX/

[HansR]

Hi Leo,

I am using cumulusUtil v6.15.3. and I have been adding additional user defined charts to the CutilsCharts.def.

I ran into a problem I could not find a solution for in the forum so far: the Home Button was suddenly not refreshing the charts and also during startup of the webpage the dashboard was shown but no chart in the report panel on the right side. I had to manually select a chart from the dropdown or create an event on the associated tile in the dashboard on the left to produce a chart on the right panel.

From my point of view your site runs perfectly well (no errors in console with F12) though a bit slow but since I am the client in NL it may be the connection speed. The observation you have (no initial chart) most likely has to do with a cache issue after an update or a website run (which updates the runtime library). Try ctrl-F5 to reload.

If that does not work, please run

Code: Select all

utils/bin/cumulusutils.exe website

and post the resulting logfile

After some experiments I think a found the root cause: If there are too many charts (> 19) defined in the Cutilscharts.def for the output file cumuluscharts.txt the home button stops to work and the refresh problem as described above occurs.

There should be no intrinsic limit on the number of charts. However, the OUTPUT keyword is the feature to split the charts in a more files. This gives the possibility to group your charts and bring them to separate menu choices (see User Defined Menu System2).

I solved it by commenting out some charts in my Cutilscharts.def and reduced the number of active charts in the home charts drop down menue so that no longer a scrollbar is shown.

If you keep experiencing a limit to the number of charts in cumuluscharts.txt, please post the logfile (website run as described above) and the CutilsCharts.def (change the extension to txt to post it) so that I can run it on my testmachine and see what the problem is. I said that there is no intrinsic limit on the nr of charts in the cumuluscharts.txt but I never tested this so it is possible I am mistaken or that you found a bug.

Regards,

[lse-cumulus]

Hi Hans
Thx for the fast feedback.
The reason why I generated so many charts in the one (1) default ouptput file was the idea to have each of the 24 tiles of the dashboard as fast access to a different chart, especially on a mobile device.

When I encountered the problem, I did test with different browsers (chrome, firefox and edge) and got it on each browser and I also tried to delete the browser cache data, that did not remedy the problem.
The logfile (debug level information) of the chartscompiler did not show any specific problems.
I got rid of the problem by 1st going back to use the CutislCharts-default.def and after that worked I had the idea to shorten the number of charts in my CutilsCharts.def (but maybe I commented now out one of my charts that had other issues and is the real culprit although in that case usually the charts compiler complains and does not created the *sensor.txt file).

I will not find time before the weekend 21/22-Jan to try to reproduce the problem again:
a) increase again number of charts in my CutilsCharts.def.
b) produce website
c) if problem re-occurs keep website in that condition and send you my CututilsCharts.def.txt and the logfile from the cutils.exe website command of the charts compiler (should I change the logfile debug level?).
regards
leo

[HansR]

Hi Leo,

Sounds good action, I'll wait for your possible post of the CUtilsCharts.def
Use errorlevel info. Don't use Verbose ever unless I ask you to: this produces a very large logfile with mostly useless info

Regards,
Hans

[lse-cumulus]

Hi Hans
I could reproduce the problem: the culprit is the RecentEVT chart. As soon as I add this chart to the CutilsCharts.def (regardless of number of active charts), the home button problem occurs.
I have now left my site in this problem condition: http://leonas.ddns.net/Wetterstation.Su ... CumulusMX/

I have attached a zip file with the CutislCharts.def and the logfile of the Cutils website run.

Chart RecentEVT Title EvapoTranspiration
PLOT Temperature As SPline Colour blue zindex 90
PLOT Evapotranspiration As Area Colour green
EndChart

[HansR]

Hi Leo,

Ok, thanks.

On my system I generated the cumuluscharts.txt with your CutilsCharts.def as a standalone HTML. That does not pose a problem.
I'll replace my own def file with yours and then see what happens.

Regards,

[lse-cumulus]

Hi Hans,
maybe my problem is data related:
Because with my data the RecentEVT Chart does not show up any EvapoTranspiration (only temperature values),
although the AllTempEVT chart does show values for DayEVT.

[HansR]

I compiled it on my system and there does not seem to be a problem
(you can check my website, I'll leave it on until problem solved... well within reason ).

In my view it thus seems to be somewhere in your environment or data-driven somewhere deep in the Cutils system: In the console I get the error:

Code: Select all

VM925:1 CUserdataRECENT.json parsererror : SyntaxError: Unexpected token ']', ..."piration":]}" is not valid JSON

meaning the USerAskedData does not run properly (I mean: has a bug somewhere OR the monthly log containing the evaporation data is not correct).

The data the user wants are selected by analysing the variables used and subsequently the JSON (in this case the CUserdataRECENT.json) is generated. When an error is encountered javascript breaks off and only when choosing another chart processing resumes. The JSON's are only read during initialisation of cumuluscharts.txt for performance reasons, so when displaying the chart there is no data in the data arrays for the RECENT data.

I observe that the UserAskedData procedure on my site does produce the correct data so it seems an issue with your data.

From what I gather from the error message, it seems as if you have no data on this variable at all. This is confirmed with your JSON URL.
Your PWS does not calculate the EVT by itself (like e.g. the DAVIS) so you rely on CMX to do the calculation.

Did you tick Station Settings=>Common Options=>Evapotranspiration ?
(If so, please wait the Interval nr of minutes until the new JSON has been generated)

Regards,

[HansR]

Hi Hans,
maybe my problem is data related:
Because with my data the RecentEVT Chart does not show up any EvapoTranspiration (only temperature values),
although the AllTempEVT chart does show values for DayEVT.

DayEVT comes from the dayfile.txt, the RECENT values come from the monthly logfile.
And indeed, as you will have read: I agree with you: data issue

[lse-cumulus]

Thx Hans for narrowing my issue down: data problem with my recentEVT values that just has side effect on home button.

I checked your website with my charts and the charts are working with YOUR data; so you can return back to your own charts.

For the time being I will remove the RecentEVT chart from my chartsdef and my webpage, to get a proper home screen refresh back.

I checked the CumulusMX station setting and the option "Calculate evapotranspiration" has been ticked on (at the time when I started with CumulusMX I had no idea about the purpose of this field for CumulusUtils later).

I checked my monthly logfiles and i can find ET values mostly =0 for the 5 minute intervals in January, but also ET values > 0, but only small ones < 1 (e.g. 0,02 up to 0,44). The Annual ET column is filled out and summed up until now up to 4,99.
I checked the daily logfile and the total ET column is filled out with values for each day. In January 2023 also small values < 1 from 0,12 to 0,45.

And then I found it: I have no monthly logs from 1.1.2023 to 6.1.2023. So there is a GAP from 31.12.2022 23:55 to 7.1.2023 09:55.

I have not mentioned yet, that I run CumulusMX only on weekends in realtime on my PC and fetch the data from Ecowitt Cloud via API Key for the rest of the week (as my PWS is constantly feeding data to the Ecowitt cloud also with my PC not running).
Seems that the catchup was not possible in that time range and I got only day files consistent. While writing this I remember now, that I had a duplicate day entry arround 6.1.2023 and I removed the duplicaty day log entry via edit/delete in CumulusMX console to get my rain totals corrected. I never paid any attention at that time to the monthly logfiles.

Does that inconsistency explain my RecentEVT problem, although since 7.1.2023 the monthly logfile entries are all there and the recentEVT only goes back 36 hours? I remember now also that 2 weeks ago the RecentEVT chart did work (once) and showed up data. At that time also the home button was working. 1 week ago I ran into the home button problem and also did not see RecentEVT values any longer.

[HansR]

I checked your website with my charts and the charts are working with YOUR data; so you can return back to your own charts.

Will do

[...]
I checked my monthly logfiles and i can find ET values mostly =0 for the 5 minute intervals in January, but also ET values > 0, but only small ones < 1 (e.g. 0,02 up to 0,44). The Annual ET column is filled out and summed up until now up to 4,99.
I checked the daily logfile and the total ET column is filled out with values for each day. In January 2023 also small values < 1 from 0,12 to 0,45.

And then I found it: I have no monthly logs from 1.1.2023 to 6.1.2023. So there is a GAP from 31.12.2022 23:55 to 7.1.2023 09:55.

You could fill those missing days will six lines as copies of one before or after, only changing the date. If that results in a correct chart and correct initialisation then you fixed it.

Note that modifying logfiles (or any data for that matter) requires you to shutdown CMX, edit the files and restart CMX. Editing outside the CMX data editors canno be done while CMX running.

I have not mentioned yet, that I run CumulusMX only on weekends in realtime on my PC and fetch the data from Ecowitt Cloud via API Key for the rest of the week (as my PWS is constantly feeding data to the Ecowitt cloud also with my PC not running).
Seems that the catchup was not possible in that time range and I got only day files consistent. While writing this I remember now, that I had a duplicate day entry arround 6.1.2023 and I removed the duplicaty day log entry via edit/delete in CumulusMX console to get my rain totals corrected. I never paid any attention at that time to the monthly logfiles.

I had understood the intermittent realtime situation.

Does that inconsistency explain my RecentEVT problem, although since 7.1.2023 the monthly logfile entries are all there and the recentEVT only goes back 36 hours? I remember now also that 2 weeks ago the RecentEVT chart did work (once) and showed up data. At that time also the home button was working. 1 week ago I ran into the home button problem and also did not see RecentEVT values any longer.

I can't really tell, all I can see is that the CUserdataRECENT.json is empty.
The logfile you posted was on the website generation. Could you post the logfile for the UserAskedData? That should tell us more.

[HansR]

Maybe you can post the januari logfile as well, there must be an error in there such that the file does not read properly.
If we can't find the problem visually, I can try it on my debug system.

[lse-cumulus]

Hans, thank you very much. With your guidance now everything falls into place and I could resolve the problem and understand now exactly what happened:
a) CumulusMX seems to run in German Locale on my PC. The January Logfile is not called Jan23log.txt, but Jän23log.txt.
b) I started with CumulusMX last year in June and the shortnames for the months from June to December are almost identical, except Okt23.log and Dez23.log. But last year I had not yet CumulusUtils in place, so I did not pay attention to the name of the monthly logfiles.
c) I activated CumulusUtils in January and on 7th January I wanted to activate also the external sensors and made 1st run of CumulusUtils with command UserAskedData. And I got an error in the cumulusutils logfile, saying that the file Jan23log.txt could not be found. But I made the wrong turn for the remedy: I copied the Jän23log.txt to Jan23log.txt and the error disappeard from the CumulusUtils UserAskedData logfile.
d) Now I understand, that CumulusMx was continouing to use Jän23log.txt (but without the first 6 days), but CumulusUtils was reading the static and not updated Jan23log.txt, therefore the CUserDataRecent.json was always empty for days > 7.1.2023.
e) The correct remdy would have been, to adapt CumulusUtils.ini and set MonthsOfMiracleAndWonder=Jän,Feb,Mär,Apr,Mai,Jun,Jul,Aug,Sep,Okt,Nov,Dez.

f) I did this now, ran CumulusUtils with commands Website and UserAskedData and also activated RecentEVT chart again in the CutilsCharts.def, and all is working fine now on my site: http://leonas.ddns.net/Wetterstation.Su ... CumulusMX/
1) RecentEVT chart is producing data
2) Home Button is behaving correctly

g) I think I can also fix the gap in my monthly logfile
1) stop CumulusMX
2) insert the lines from Jan23log.txt for days 1.1.2023 until 7.1.2023 at the beginning of my Jän23log.txt
3) starup CumulusMx again and the monthly logfile should have no gaps.
That was no CumulusMX catchup problem with Ecowitt cloud data, but my bad.

Thx again for your help and time. I learned now a lot about the inner workings of CumulsUtils and MX and that should help me to avoid such mistakes in the future. In retrospect I took a number of wrong turns when trying to remedy things until the end result problem/side-effect was no longer easy to diagnose. You helped me to backtrack succesfully and apply the correct remedy to the real underlying root cause.
I should have asked earlier for help, but now I know how responsive you are in this forum.
regards
leo

[HansR]

Leo,

You are spot on with you analysis and it is great you found the solution of MonthsOfMiracleAndWonder

The usage of locale/language and the abbreviated month names in the logfile names does not make is easier and you are not the first to run in this type of problem, but great it is solved. With the evolution of CMX this will be resolved in the future.

Problem solved.
Enjoy the software

Regards,

[HansR]

Leo, I have one thing to add: you probably have your PC in de-AT locale (hence the Jän[ner] month name). But you do have your CUtils language still in (default) English meaning the en-GB locale.

If ever wanting to change the locale of your machine or explicitly (via the commandline) for CMX or when getting a new computer or moving to a RPi, pay attention to your language setting / locale. If you change the locale setting for CMX either through computer setting or explicitly, the filenames and the separators (decimal point, field separators, datetime separators) will change and as a result you will get a different logfile format which will cause issues.

Once started in a locale, keep that locale (or convert old logfiles to your newly wanted format).

For CUtils such CMX locale change has no effect other than adjusting MonthsOfMiracleAndWonder.