Issues after Upgrade to CumulusUtils v6.23.6: heatmap, meteocam, ...

[lse-cumulus]

heatmap.jpg

Hi Hans
I upgraded my webpage http://leonas.ddns.net/Wetterstation.Su ... index.html

• CumulusMX from v3.24.2. build 3235 to 3.28.5 build 3282

• CumulusUtils from v6.17.3 to v6.23.6

Basically upgrade went fine and I got all the new recently added CumulusMX #webtags available (e.g. in Systeminfo and my UserReports, but I have the following issues remaining:

1. I could not resolve all issues with upload.php (could fix CumulusMX realtime upload via GET, but not yet problem when uploading customlogsRecentdata.json under CumulusUtils UserAskedDat) and therfore switched to SFTP for time being. Now all SFTP uploads work without errors. The upload.php is on me as I have an old PHP 5.3.x environment so I had to adapt the orignal upload.php file from the CumulsMX release

• (2) When calling a page to open in the reportView I stil do not see the URL of the page in the browser, but only my main page URL. I thought that should change with this version (or did I confuse this with the v7beta?). But this could also be a security issue on my site not using TLS/SSL

• (3) All graphs are working fine and i got the new Rain vs. EVT, but the temperature graph Heatmap is no longer working. But was working fine until day of upgrade.

• (4) I configured meteocam for my Ecowitt HP10, but only get an empty page in reportview. When calling the camera URL from webtag <#EcowittCameraUrl> directly in my browser it works. Might be a security issue as Ecowitt URL is https and my site plain http. The files meteocam.txt and meteocamrealtime.txt are produced. meteocamrealtime.txt just contains the unprocessed webtag.

• (5) I activated customLogs in CumulusMX (just 1 day before so I have not yet daily CLog Files) and CumulusUtils. I do see the recent and daily webtags but not yet graphs. I assume I have to wait a couple of days to produce data ... I have not yet made translations in the CUstringsEN.ini file.

CustomLogGraphas.jpg

[HansR]

(1) I could not resolve all issues with upload.php (could fix CumulusMX realtime upload via GET, but not yet problem when uploading customlogsRecentdata.json under CumulusUtils UserAskedDat) and therfore switched to SFTP for time being. Now all SFTP uploads work without errors. The upload.php is on me as I have an old PHP 5.3.x environment so I had to adapt the orignal upload.php file from the CumulsMX release

See also my comment on your last point.

OK, maybe your SFTP solves most issues but your customlogsRecentdata.json still has issues because I do not see the charts. Note that if you change the definitions of the customlogs you need to rerun the CustomLogs so the charts will be compiled anew. Did you do that? Because if I look at the chart I see that it does not know the variables in the JSON. For the RECENT charts, there should be data immediately, for the DAILY that might take a rollover before all charts function properly again.

If you can't get the charts in good order you may try a reset of the the uploads : remove the following parameters: LastUploadTime (General), DoneToday (Compiler) and DoneToday (CustomLogs) and run again.

(2) When calling a page to open in the reportView I stil do not see the URL of the page in the browser, but only my main page URL. I thought that should change with this version (or did I confuse this with the v7beta?). But this could also be a security issue on my site not using TLS/SSL

That will be available in version v7 which will be in public beta when CMX v4 goes into public beta. I expect that to be soon.

(3) All graphs are working fine and i got the new Rain vs. EVT, but the temperature graph Heatmap is no longer working. But was working fine until day of upgrade.

The Heatmap does not work because of a bug in HighCharts. The bug has been fixed but not yet propagated in the CDN. So please use version 11.2 See this thread.

(4) I configured meteocam for my Ecowitt HP10, but only get an empty page in reportview. When calling the camera URL from webtag <#EcowittCameraUrl> directly in my browser it works. Might be a security issue as Ecowitt URL is https and my site plain http. The files meteocam.txt and meteocamrealtime.txt are produced. meteocamrealtime.txt just contains the unprocessed webtag.

Strange... that should work out of the box. But I see your site cannot find the required meteocamrealtime.txt which contains the correct URL. So you may want to look at that configuration (as extra web file). See the Wiki.

Schermafbeelding 2024-03-03 130900.png

(5) I activated customLogs in CumulusMX (just 1 day before so I have not yet daily CLog Files) and CumulusUtils. I do see the recent and daily webtags but not yet graphs. I assume I have to wait a couple of days to produce data ... I have not yet made translations in the CUstringsEN.ini file.

See my remarks under (1). If you look at the console (F12 => console) you will see most clarifications of things not functioning.
The CustomLogs indeed take some time to have the charts to load properly and automatically. The remarks I made about the parameters may be ignored if you just started the Customlogs today. Some things take time

[lse-cumulus]

Hans
I appreciate your detailed answers that helped my to solve my issues:
(1) for time being i will stick to SFTP and fix Custom Logs first under (5) customlogsRecentdata.json. Then I will try to correct my version of upload.php

(2) nobug; waiting for V7.x BETA

(3) chart problem caused by latest version of HIghcharts. Sorry that I did not find the corresponding thread in forum by myself.
Fixed by adding the following line as workarround into file cumulusutils.ini in section [General]: UseSpecificHighchartsVersion=11.2.0.
Now the Heatmap graph is working as before and the new rain chart Rain vs. EVT is now also working correctly.

(4) meteocam now working fine with my Ecowitt HP10. I saw that the file utils/meteocamrealtime.txt was not transported by my /web root; but did not think about adding a line to CumulusMX extra Web Files Settings to get file processed to calculate the <webtag> ##EcowittCameraUrl and copy file to webpage (at least once a day) so that file meteocam.txt can do its job.

(5) Custom Logs charts not yet working. But you gave the correct input that customlogsRecentdata.json was not correct. I deleted file on website after upload.php problems. But I also had the CumulusMX Custom Logs not correct in the beginning and changed the setup a number of times. I did try to recreate the files by using command \utils\bin\cumulusutils.exe Website (that should include the argument CustomLogs correct?). I ran command also again with CustomLogs. I also deleted the already generated CLog*.txt Files in the CumulusMX/data directory (only recent files were present not yet daily files), because these files had lines with differnt number of columns after i added/removed <webtags>. Now correct recent CLog*.txt files are generated by CumulusMX. I also updated the update timestamps in cumulusutils.ini LastUploadTime [General], DoneToday [Compiler) and DoneToday [CustomLogs] before I ran command cumulusutils.exe again for Website and Customlogs and UserAskedData.
So far the custom log charts are not yet showing data, but i hope the uploaded file customlogsRecentdata.json is now correct (SFTP upload reports now errors). I will be patient and wait for CumulusMX dayswitch and CLog*.txt daily files and the regeneration of the json file after midnight by command cumulusutils.exe with argument Website.

[HansR]

(4) meteocam now working fine with my Ecowitt HP10. I saw that the file utils/meteocamrealtime.txt was not transported by my /web root; but did not think about adding a line to CumulusMX extra Web Files Settings to get file processed to calculate the <webtag> ##EcowittCameraUrl and copy file to webpage (at least once a day) so that file meteocam.txt can do its job.

It should at least be every Interval because Ecowitt has a new URL if there is a new image.
So don't tick realtime and don't tick EndOfDay: if both are NOT ticked but you do have Upload ticked => THEN it uploads at the interval which is what we need to have

(5) Custom Logs charts not yet working. But you gave the correct input that customlogsRecentdata.json was not correct. I deleted file on website after upload.php problems. But I also had the CumulusMX Custom Logs not correct in the beginning and changed the setup a number of times. I did try to recreate the files by using command \utils\bin\cumulusutils.exe Website (that should include the argument CustomLogs correct?).

That is correct yes. But

Code: Select all

utils\bin\cumulusutils.exe CustomLogs

is the easy, logical and fastest route to achieve updates of the CustomLogs.

I ran command also again with CustomLogs. I also deleted the already generated CLog*.txt Files in the CumulusMX/data directory (only recent files were present not yet daily files), because these files had lines with differnt number of columns after i added/removed <webtags>. Now correct recent CLog*.txt files are generated by CumulusMX.

OK.

If you have modified your RECENT CustomLogs and run into problems then always the easy wway out is to delete those old data files. The regeneration and full dataseries in the charts is achieved withing the time covered in the charts. So no problem.

That is somewhat different for DAILY data because that may contain valuable data. In that case editing the datafiles after modification of the definition may be a way out (I say now in case that may happen to you sometime )

I also updated the update timestamps in cumulusutils.ini LastUploadTime [General], DoneToday [Compiler) and DoneToday [CustomLogs] before I ran command cumulusutils.exe again for Website and Customlogs and UserAskedData.

Don't modify timestamps, that is very error prone.
Just delete those parameters, then run the command again. Effecitively it will regenerate the full JSON for all data available.

So far the custom log charts are not yet showing data, but i hope the uploaded file customlogsRecentdata.json is now correct (SFTP upload reports now errors). I will be patient and wait for CumulusMX dayswitch and CLog*.txt daily files and the regeneration of the json file after midnight by command cumulusutils.exe with argument Website.

Yes, as long as there are no DAILY data, charts may fail in automatic loading.
You may in such case, try however to load a RECENT chart manually by selecting the drop down. That should work if that data is OK.

If the problem remains, lets see again tomorrow after the rollover.

[HansR]

NOTE: there is no customlogsRecentdata.json in your server directory. It should be uploaded similar to extrasensorsdata.json which is present. Could you please post your latest logfile for the UserAskedData command?

[lse-cumulus]

I saw that the files customlogsRecentdata.json (hourly by command Useraskeddata) and customlogsDailydata.json (last time 17:00 with command website/CustomerLogs) are ftped to my webroot, but both are empty. Just contain {}.
The extrasensorsdata.json is also produced hourly by command UserAskedData but is fine.
I have not the latest logfiles (as I am currently travelling and have no remote access to the CumulusUtils PC), but a previous logfile from today from UserAskedData, that looks similar like all the others: no errors, talking about producing customlogsRecentdata.json and extrasensorsdata.json and making succesfull SFTP of both files ...
If you need the latests (recent) logfile it will take me a couple of days before I can provide them.

[HansR]

OK. Thanks, this logfile is enough (assuming no changes).

I see the whole process runs OK but it says:

Code: Select all

2024-03-03 09:55:46.513 Information CustomLogs GenerateCustomLogsDataJson: timeStart = 29.02.2024 09:45:00; timeEnd = 03.03.2024 09:45:00
2024-03-03 09:55:46.513 Information CustomLogs GenerateCustomLogsDataJson: DoneToday = 03.03.2024 09:53:44... DoDailyAsWell = False
[...]
2024-03-03 09:55:46.515 Information CustomLogs ReadRecentCustomLog: data/CLogRain-202402.txt - Start: 29.02.2024 09:45:00 ; End: 03.03.2024 09:45:00 ;

This means the timing is wrong: Start and End are the same!
I advise again to remove the named parameters - make sure it does not interfere with a crontab command! - save the inifile and wait for the scheduler to run. I expect the JSON's to contain data for some Interval minutes and the series will grow every run of UserAskedData.

I also advise to remove the CustomLogs JSON's from the server : if there is an error in those JSON's it won't spoil the process.
Btw: if you use SFTP or other forms of FTP, there will be no concatenation , all JSON's will be sent full.

All this is under the assumption that your logs are correct and reflect the definition of the CustomLogs.

If in a few days you are back please send me the then actual log so we cnn see what goes on.

[lse-cumulus]

Hi Hans,
Custom Logs seem now also to work after day rollover. I still have not yet access to the logfiles of cumulusUtils but I see he results on the weberserver webroot directory.

• customlogsDailydata.json was produced at 01:00 with command Website and contains now data for 1 day. The daily charts do not YET show data, but I am confident I just need more days ...

• customlogsRecentdata.json was last produced at 05:05 with command UserAskedData. This should have happened first at 01:05 and now hourly. The file contains data and the charts display them. GOOD. Should see the hourly growing of data in the jsons file and the charts ...

Seems that i just was too impatient. Should have enabled Custom Logs earlier in CumulusMX before the CumulusUtils upgrade and CU CLogs activation to already have more data from start.
That is a good way to start into a work week. The update from the weekend was a success. All issues resolved. CU working now with more features. Most issues were on me, no CumulusUtils SW problems.
On the next weekend I will check the CU logfiles locally on my PC and try to switch back from SFTP to upload.php. Maybe my latest upload.php issue was just because the file customlogsRecentdata.json was an empty file or it is related to incremental appends ...

[by the way: my wording in yesterday's post was not accurate enough: I did not update = modify the done timestamps in cumulusutils.ini but commented them out with an " ;" and saved the ini file, and cumulusUtils produced new lines of these timestamps LastUploadTime [General], DoneToday [Compiler) and DoneToday [CustomLogs] ]

[HansR]

Hi Leo,

customlogsDailydata.json was produced at 01:00 with command Website and contains now data for 1 day. The daily charts do not YET show data, but I am confident I just need more days ...

The JSON does not seem to load properly, it has an error (see console F12). I checked the JSON and there seems to be an issue with #CO2_24h which appears to have no data (in the logfile?) and therefore has a JSON syntax error. Did you change the CustomLog definition and forgot to either run CustomLogs command or did you not edit the logfile to the new def?

Anyway, that is something which will probably not recover by itself.
We'll see when you are back at the CUtils machine. The rest seems OK yes.

[lse-cumulus]

Hi Hans,
thx for the additional input.
I first started with only 1 attribute in this ClogCO2conctration and then added a 2nd column for webtag #CO2_24h.
I did rerun command Customlogs/Website aftewards but I assume the datafile for 1st day has only 1 column and no value for 2nd column. I guess I will have to edit this customlog file and add manually the missing value. Yesterday before the day switch I did not yet see this daily file on the CMX data directory.
(or I disable this mixed up CustomLog and create a proper new one)

[HansR]

Hi Hans,
thx for the additional input.
I first started with only 1 attribute in this ClogCO2conctration and then added a 2nd column for webtag #CO2_24h.
I did rerun command Customlogs/Website aftewards but I assume the datafile for 1st day has only 1 column and no value for 2nd column. I guess I will have to edit this customlog file and add manually the missing value. Yesterday before the day switch I did not yet see this daily file on the CMX data directory.
(or I disable this mixed up CustomLog and create a proper new one)

Aha... OK.
Let's just wait then, it may pick up after two days having just the first day wrong.
But yes, it may be best to remove the line in the log with only one value.

No need to disable the mixed up CustomLog. Now you explained I think it will be easy to correct.
Next contact when you're behind the computer

[HansR]

Hi Leo,

I just saw on your site that all your problems related to CustomLogs seem to be resolved.
That's good as I won't be available from Saturday to Monday

[lse-cumulus]

hi Hans,
almost done. I could not yet fix my custom logs datafile inconsitency. But in the meantime as workarround I made a daily manual edit of the customlogsDailydata.json file on my webroot and corrected the syntax error in column Co2_24h. After that the customs log graphs were working fine.
thx
leo

[HansR]

OK. So the question is why is that field CO2_24h empty?

Could you check the log for that?

[lse-cumulus]

Hi Hans,
my CustomLogs problem is finally resolved and I could identify the root cause. And again it was a handling error on my side.
When creating the daily Custom Log for CLogCo2Concentration via CMX console Settings - Custom Logs I left a SPACE between the two webtags
<#CO2>; <#CO2_24h>. That caused also a space in the data file and that resulted in a parsing error of command cumulusutils.exe CustomLogs.
As a consequence the generated file CustomLogsDailyData.json contained a syntax error and the daily Custom Logs were not displayed.

Code: Select all

2024-03-09 12:07:02.178 Information CustomLogs ReadDailyLog: data/CLogDailyCo2Concentration.txt
2024-03-09 12:07:02.179 DetectSeparators: date: . | Field: ; | Decimal ,
2024-03-09 12:07:02.179 Warning CustomLogs : There are more/less webtags than values in the log on line 0: 03.03.24;1363; 1632
2024-03-09 12:07:02.179 Warning CustomLogs : The chart may not be what you want, please correct the content of the datafile. Continuing...
2024-03-09 12:07:02.181 Error CustomLogs GenerateCustomLogsDataJson: Cannot parse value in CustomLog data/CLogDailyCo2Concentration.txt
2024-03-09 12:07:02.181 Error CustomLogs GenerateCustomLogsDataJson: Die Eingabezeichenfolge hat das falsche Format.

Corrections:

• fixed CMX Settings - Custom Logs: removed SPACE

• edited CLogCo2Concentration.txt data file and removed the SPACEs there also

• re-run command cumulusutils.exe CustomLogs and produced a correct CustomLogsDailyData.json file