Cumulus Support

Hi Hans

I am still looking at this from my side and have cleaned up the chart labels so here is a 15min update log together with a very recent customlogs log

Hi Andy, Thanks. But I meant a data log as produced by CMX CustomLogs datalogging.

Hi Hans

Sorry for the misunderstanding but here is the log file which covers the 15 min update at 16:00

Wow, that's a lot of processing of HTML pages MX is doing, are all CUtils installs like this? It feels a bit wrong creating static HTML regularly whether anyone looks at it or not, wouldn't client AJAX updates of template pages be better?

@Mark: What is happening is kind of how the old website worked for some parts defined as UserReports. UserReports exist - there are only a few CUtils users which make intensive use of these - which are processed at interval because of the webtags they contain. I think these are what you are pointing to. It is the user who defines what to make available and what to process when. Doing that with Ajax updates may be possible, certainly after the CustomLogs excercise, but is not easy and would require a realtimeUserreports.txt file to generate. I am not convinced that would be more efficient than what is happening now. So in short 'are all CUtils users doing this': No.

Anyway @Andy: you still did not post the data log for the 15 minute sampling (you post the CMX MXdiags log).

The datalog I received from Andy is attached. It sheds some light on the issue. Andy changed the logging frequency from 2 minutes to 15 minutes. The system tries to read in steps of 15 minutes and that does not work well (when changing the CustomLog, CUtils reads all data, rewrites the JSON with all data and then continues incremental.

My advise would be (certainly with such small dataset) remove all two minute data, save and conitnue.

Hi Hans

Thanks for the solution, I will investigate and let you know how it gets on

Hi Hans

Well after cleaning up the data files to remove the 2min data I tried running customlogs cmd and got the same zero result for the RecentData charts.

The next test was to change the file names of the recent data files to stop the data to them, I reran customlogs to restart the process in new data files and ran it for an hour or so. a check on the website showed that no data appeared on the charts. Not helpful! One final observation is that the customlogsRecentData.json file on the host site only shows the basic text headings with no data at all despite being updated regularly.

Of course the daily files work fine and these ones were also edited to get all the back data into the new files.

Not sure where to go with this now, maybe I will give up on the RecentData charts and just live with the Daily charts.

AndyKF650 wrote: ↑Wed 08 Nov 2023 8:09 pm Well after cleaning up the data files to remove the 2min data I tried running customlogs cmd and got the same zero result for the RecentData charts.

The next test was to change the file names of the recent data files to stop the data to them, I reran customlogs to restart the process in new data files and ran it for an hour or so. a check on the website showed that no data appeared on the charts. Not helpful! One final observation is that the customlogsRecentData.json file on the host site only shows the basic text headings with no data at all despite being updated regularly.

That is disappointing.

AndyKF650 wrote: ↑Wed 08 Nov 2023 8:09 pm Of course the daily files work fine and these ones were also edited to get all the back data into the new files.

Daily files are easier and different: there is no timed selection, its just read everything and plot.

AndyKF650 wrote: ↑Wed 08 Nov 2023 8:09 pm Not sure where to go with this now, maybe I will give up on the RecentData charts and just live with the Daily charts.

That is not a good idea. If that would be a solution I can throw away the module. Can you give me the definition (the webtags, the timing, the name) - copy from Cumulus.ini if you wish - and I will replicate it here and see what happens. I'll analyse tomorrow morning (if I can enter the definition).

[EDIT:] I created a CustomLog identical to your TemperatureCL RECENT CustomLog and I log it with 15 minutes. Does that simulate exactly your situation? We'll see what it does. Tomorrow we'll see the result.

Hi Hans

sorry I was on another call this evening and now have copied the customlogs section of the CMX.ini file as follows. The Recent logs are at 15 min intervals.

Hi Andy,

Yes, middle of the night, no not because of this issue

It is beginning to dawn on me what is going on: if I read your charts well, your standard interval (Internet Settings=>Interval Configuration=>Normal Interval Settings=>Interval time is 10 minutes and your logging takes place every minute: Station Settings=>General Settings=>Log Interval.

For the incremental upload procedure however (only with PHP upload ), we need to determine the samples which have been added the past interval periode and only send those data (otherwise we would duplicate data for the chart). So every interval we only send the data logged in the last Internet Interval.

So - here it comes - assuming your 15 minutes for CustomLogs and your 10 minute Internet Interval, when we take a logging interval larger than the Internet Interval we may not find any data at all from 01 - 10. The next interval we will find one entry on the 15th minute from 11 - 20 but that entry is less than the finish time of the interval which is the 20th minute. But i need to store (memorize) somewhere what the last update time was on the Interval so I store 20. The next interval I do at 21 - 30 minutes. This means I'll be missing out 15. It syncs again on the period 21 - 30 where it finds one entry in the log at 30 minutes (being a multiple of 15). That 30 min value is send to the server. At 40 minutes we have again an empty value for the CustomLog as it searches for a value from 31 - 40 minutes. Etc...

If the interval results in no value you get the message you found: because it never finds data on the end time of the Internet Interval.

I first started logging at 5 minutes and then changed to 15 minutes. IT works flawlessly (which I now also observe on your site) so it seems to work as intended. Only when CUtils can't find data on the interval boundary, you get an error message as above. That is wrong but the system does not fail, it only requires some periods to see it is working.

I will change the message to a warning in case the CustomLogs interval is larger than the Internet Interval (pfff... why is this possble anyway ).

And in addition the customlogsRecentdata.json may fail when it does not exist and it is created without data i.e. its first Interval period is run when no data have been registered in the log. I will experiment withh that initial situation and see what I can do about that.

So far the system is running fine and I may have to check only for the initial situation which may lead to an error condition. If you make changes to the CustomLogs, always wait one Interval Period before running CustomLogs (because that reinitialises the JSON) to get at least one data entry in the log and only than run CustomLogs.

And maybe one additional advise: do not use CustomLogs intervals larger than your Internet Interval. It leads to confusing situations as we can see.
If this type of things will happen more often I may even enforce it (i.e. skip CustomLogs with intervals larger than the Internet Interval)

Hope this clarifies a bit.

Hi Hans

Thanks for that, I see that my new start last night has resulted in a working chart

I will look into the detailed explanation this morning.

I just checked CMX and the internet interval is 15mins and the general log interval is 5mins. I hope this helps

Hi Andy,

And I can add, that the original issue - where you used a 2 minute log frequency with a 15 minute Internet Interval - never produced a JSON because the 2 minute logging always had a xx:14 and xx:16 entry so the Internet Interval at xx:15 saw it always end on 14 and looked for a next file.

Complex but solvable either by :

1. forcing Custom logging frequencies synced with Internet Interval
2. disabling certain Custom Logging frequencies
3. changing the algorithm for choosing a next logfile (likely the choice)

Depending on the resulting complexity. I'll think about it.

Hi Hans

Seems like my trial and error approach is a process for refining this new functionality and making it more resilient.

I have just started copying the previous recent data file entries for November into my new working data file to get the full month data. A bit of a fiddle but there are only another 4 more files to attend to.

Keep up the great support work for inveterate system breakers like me.

AndyKF650 wrote: ↑Thu 09 Nov 2023 8:26 am Seems like my trial and error approach is a process for refining this new functionality and making it more resilient.

Definitely, keep up the good work

But one thing for sure: there must be data in the log.
I will force that or make CUtils abandon the CustomLogs command and/or UserAskedData command immediately.
There some finetuning actions here. Fortunately the algorithm itself seems OK