CustomLogs beta

[HansR]

With version 6.21.0 an initial version of the CustomLogs module is released. Custom Logs have been part of CMX for a small year now and offer the possibility to log almost any timeseries (RECENT and DAILY) for any value (sensor or calculated). This is a very versatile feature of CMX and CUtils had to do something with it. Users are invited to use the module and comment on it (both the way to use it, the wiki page and technical issues).

Points of attention:

1. The Custom Logs are basically free format and text can exit between the numbers in the resulting logs. CUtils parses the Log definition for webtags which produce a value so it can plot those but it requires a resulting plot with only the numbers. No text allowed. So the user has to pay attention.
2. RECENT and DAILY logs are combined into one JSON file at the moment. Having long RECENT chart period (7 days max for CMX, but longer for CUtils) with many webtags logged, may result in very large datafiles. It is the users responsibility.
3. Currently RECENT and DAILY data are assembled in one JSON. That is not good and will be changed to two different JSON files.
4. The user can use any webtag in the definition of the Custom Log, CUtils should pick only those which produce a plottable value
5. Currently all charts have a dimensionless axis. Future releases will couple a dimension (e.g. temperature or pressure) to a webtag and attach an axis to the chart accordingly

NOTE: it seems that currently the RECENT charts will only give some points. The reason is known and has to do with the single JSON assembly. That will be priority nr 1 for the next release.

Comments on all levels (Usage, technical and wiki) are highly appreciated.
Is this useful? Do users use the Custom Logs?

NOTE: Any issue please post the log and the CustomLog definitions (section [CustomLogs] in the Cumulus.ini file)

[AndyKF650]

Hi Hans

I have just downloaded the latest beta and tried to install it on my RPi. As usual I extracted the zip to the utils folder and ran the cmd website. This stopped almost immediately, the utilslog is attached.

For info I am using the latest beta for cmx, I am not sure if this would cause a problem.

[HansR]

I had still my test IP for comms with CMX set. That should have been under debug conditional.
Repaired and a new exec is in the distribution zip in the main post.

Please don't make new subjects, any remark for this beta goes in this thread.

[AndyKF650]

Hi Hans

Thanks for the update, all works as expected.

Now I will have to put some time in to get the functionality to work for me. I will let you know how I get on.

[AndyKF650]

Hi Hans

I have spent a couple of hours this morning trying to get the customlog charts to work on my site.

So far I have got the CUtils menu looking about right but I have failed to get the charts to populate. Please could you give me a further clue as to where I am going wrong.

I have attached the latest utilslog files for both the auto useraskeddata and the website cmd together with the CUMenu.def and CUCharts.def for information.

Any help would be appreciated.,

[HansR]

Hi Andy,

Thanks.

1. I am not sure why you made the "ExtraSensors" and "CustomLog" main level menu items and removed the default "Extra" item. If you have the default "Extra" i.s.o. "ExtraSensors" you will get the standard extra modules for free in the menu (see my site). But OK, this is possible as well.
2. It would be a bit easier if you reduce the number of webtags per log and maybe increase the number of logs. That would make it a bit easier to look at and try to find the cause.
3. Could you post the [CustomLogs] section of Cumulus.ini and not change that while we are discussing?
4. There seems to be a syntax error in the resulting chartsprocedure. Currently I cannot determine where that comes from.
5. But I do see that three of your CustomLogs charts end with "EndChart Output customlogscharts.txt". That should not be possible so you may have manipulated the manual override parameter and got this.

Tbh, if I look at your site for the CustomLogs I don't see anything - no table, no charts - so you probably continued testing.

I am already working on some improvements which may - or may not solve - your issues but I would first like to ask you to simplify your test setup. And note that a new daily log will only have a first value tomorrow, things go slow If you change the number of webtags in a log the charting will work but because one or more webtag values will be missing the whole chart gets mixed up. For RECENT charts that will fase out after the RECENT period, for a DAILY log you will need to edit the data log or simply remove it and start anew.

I did split the JSON in two - RECENt and DAILY - and that is working on my site so I have good hope it works at yours.

Please take my advice and for this testing, keep

Code: Select all

UserModificationCustomLogsCharts=false

that makes the tests reproducible and takes your manual edits out of the equation and makes it is only between your definition of the Custom Logs and the output as I can see it on the site.

To restart / redo the testing please delete all CustomLogs lines (including that line you should not touch ) from CutilsCharts.def.

Do you have an explanation why there are multiple CustomLogs charts with an OUTPUT definition ending?

[AndyKF650]

Hi Hans

Thanks for that very full reply.

If I have correctly interpreted the information I should

1 Delete all custom log chart entries on the CUCharts.def and restart from the beginning;
2 Change the UserModification to True;
3 Reset the custom log file on CMX to just monitor similar parameters ie temperature, humidity, dust concentration etc;
4 Change back the menu header to Extra and then make the various subsets from that header.

It was my lack of knowledge that led me to change each block of the EndChart Output items to the same.

It will look good but may take some time!!

[HansR]

1 Delete all custom log chart entries on the CUCharts.def and restart from the beginning;

Yes

2 Change the UserModification to True;

NO => it's value must be false for testing. Only when everything is OK we can start looking at modifying

3 Reset the custom log file on CMX to just monitor similar parameters ie temperature, humidity, dust concentration etc;

Complete reset is not required, just reduce your log to a max of let's say 5 webtags per log

4 Change back the menu header to Extra and then make the various subsets from that header.

That would be nice and it would simplify your menu (make it less wide).
If you have Extra on the top level, the modules will be there automatically.

I have one more thing to do for a new beta. That will be probably Thursday, maybe Wednesday. You may want to wait for that beta before continuing (although setting up the Logs of course is independent of that).

[AndyKF650]

Hi Hans

Thanks for that.

I have attached the Custom logs section of the CMX.ini file which shows the original setup of the log file and may new version which will be running every 15 mins in line with most of my other reports.

[HansR]

OK, but I assume that the lines marked OLD don't exist anymore or are disabled (they are still enabled in this file)

You do not need to run CustomLogs every 15 minutes. It is created when you run website or when run with the command CustomLogs. After that is is only required to run UserAskedData to get the necessary datafiles with the updated data at the data frequency you use (apparently 15 minutes).

[AndyKF650]

Hi Hans

One question where does the customlogsdata.json get stored or loaded, I cannot see it at the RPi level or the tsohost level. The corresponding extrasensors data.json file is certainly seen at the tsoHost level.

Is this why my charts are not working?

I have done multiple website cmd and useraskeddate cmd this afternoon to get to where I am now, just incase you ask.

[AndyKF650]

Hi Hans

This utils log report on useraskeddata might help. The date field is being reported as incorrect?

[HansR]

OK, it says:

Code: Select all

2023-10-10 17:30:05.749 Information CustomLogs GenerateCustomLogsDataJson: timeStart = 10/10/2023 17:16:00; timeEnd = 10/10/2023 17:30:00
2023-10-10 17:30:05.754 Information CustomLogs ReadRecentCustomLog: ExtraTempLog
2023-10-10 17:30:05.754 Information CustomLogs ReadRecentCustomLog: data/ExtraTempLog-202310.txt - Start: 10/10/2023 17:16:00 ; End: 10/10/2023 17:30:00 ;
2023-10-10 17:30:05.759 Information UserAskedData: Failing in GenerateCustomLogsDataJson
2023-10-10 17:30:05.760 Information UserAskedData: Message - String was not recognized as a valid DateTime.)
2023-10-10 17:30:05.760 Information UserAskedData: Continuing

So could you post the data/ExtraTempLog-202310.txt?
Apparently there is an issue with the date field there.

[AndyKF650]

Here is the Data/ExtraTempLog -202310 file

[HansR]

Ah... written conform your locale.
Have to think again. I thought the date format was fixed but the whole thing is conform locale.

I'll get back on this.