CustomLogs beta

[AndyKF650]

thanks, its what beta test is all about!

[Nossie]

reporting in for the beta test - just updated and not making any changes...

it seemed to work and then I find my extra sensors have not been plotting since the first of October (cmx on its own has)

bit strange so just sussing that out.

[mcrossley]

The custom log date format is the same as the other log files "dd/MM/yy", so the separator is locale dependent.

[HansR]

Mmm... I'll probably never get used to this.

[HansR]

OK, @Andy and @Nossie, please set beta on hold, something fundamental needs to be fixed.
My bad, should have known.

[Nossie]

No worries hans, I think my install is still goosed anyway - I probably have a config wrong but the NOAA reports are not generating correctly until I force them manually and then that seemed to sort my cutils readings

happy to go with the flow though - as I said, CMX is fine, just the website that's behaving abnormally.

[HansR]

6.21.0 Beta 2 is published in the release post.

Modified:

1. The logfiles now should be read irrespective of differences because of locales
2. Daily Custom JSON are now only constructed and uploaded once per day (i.e. the first opportunity after midnight)
3. CustomLogs data are now split in two: customlogsRecentdata.json and customlogsDailydata.json
4. A link to the Wiki Webtag list is now displayed above the table(s) on the left pane

[AndyKF650]

Hi Hans

But you were up early this morning!!

I have installed the Beta 2 and ran website cmd twice since on the first run utils log was not running. I also ran useraskeddata cmd and after all runs the custom logs display only shows a blank page.

I have attached copies of the utilslog files for both runs.

Also I have looked at the host site and compared the files for the extrasensors log and the custom log and whilst both have the ....txt file, the ....charts.txt file, and the ....realtime.txt file there is a extrasensosdata.json file but there is not a customlogsdata.json file. This I suppose is the root of the problem.

One other issue I noticed is the persistence of the copy-extratemplog file in the data folder. This is written and I presume used during the website cmd but not removed following its use.

[HansR]

Hi Andy,

It was indeed too early, returned to bed afterwards

1. You do not need to zip txt files unless they are really very big. I prefer not zipped because that makes viewing easier.
2. Running only utils/bin/cumulusutils.exe CustomLogs is much faster
3. Remember once you created the module with the previous command, if nothing changed in your configuration there is no need to run it again (unless there is a new release). So to see a data result just run UserAskedData.
4. Could you repost your ExtraTempLog-202310.txt?

[AndyKF650]

Hi Hans

As requested here is the ExtraTempLog-202310 file. for info.

Please let me know if you need any other reports/files.

[HansR]

Your file is formatted as follows:

Code: Select all

10/10/23,17:00,16.2,59,16.0,20.2,72,15.0

The Wiki says:

NOTE: The formatting of the datafiles is important as the Custom Logs are basically free format: you can put into the file any text and numbers with any separator you wish. CUtils will NOT handle that Free Format in general. The user has to take care the information in the resulting datafiles is as follows:

1) There MUST be at least one space between the numbers;
2) There cannot be any plain text in the logfile
The date/time will be untouched i.e. the format is: dd-mm-yy;hh:mm;

So apparently I read the whole separator thing in the CustomLogs definition header wrong.
*sigh* I have been doing nothing too long.

Still I don't see why CMX goes with a per locale variable separator of the fields if it does nothing with the CustomLogs itself.
Anyway, the user can set the separator between the webtags to spaces, nobody gets hurt, but it will probably get down to the point where CUtils needs to remove all locale dependencies.

Stop testing, I'll come back to this.

[HansR]

OK, beta 3 is online.
I seems the reading of the logs now is fine for every locale so please start testing. If any issue let me know.

Known issue: the CO2 webtags do not work because a webtag like CO2-pm2p5-24h is seen by javascript as an identifier and a calculation (5 - 24). The CUtils compiler which generates the javascript uses the webtag names as variables (how other to identify the webtags?) which corrupts the code. Atm there is no solution for this. Please avoid the CO2 sensor for now.

[HansR]

@Andy I see strange things on your site:

1. The menu still holds an item ExtraCharts under which you defined an item Custom Charts. That only loads the charts and not the realtime table on the left of it. Please make an empty top item Extra in your menu and remove the items under your ExtraCharts item. When running website, that should generate automatically the CustomLogs module (including the table, not only the charts)
2. The customlogsRecentdata.json (e.g. for your ExtraTempLog) has the data twice for each variable. This produces a line running back from the end to the graph start. I don't understand yet but it does not happen (so far) on my site.
3. I notice you changed the frequency for the ExtraTempLog quite often

Could you change your menu conform what I wrote above?

The double data in the JSON may be a result from a previous experiment. Could you set the frequency for ExtraTempLog to one minute and leave it there while experimenting, remove ExtraTempLog-202310.txt and remove the customslogsRecentdata.json on your server. To prevent interference with the old logs it may be best to switch CMX off while removing the files. Then switch CMX on again, run Cutils website and let the data on your server grow naturally (I assume you have the UserAskedData automatically running every 10 minutes or so).

[HansR]

Information for testers.

The system seems relatively stable. However there are some issues:

1. The CO2 sensor webtags as described above => I asked Mark
2. Changing a table name requires a rerun of CustomLogs (and the implicit recompilation) if you forget to run CustomLogs, the compilation will fail and JSON's will no longer be created and sent in subsequent UserAskedData runs. This will show the system failing (charts not updated). The dependencies on user interaction must be avoided or minimized.
3. Adding a log (Recent) leads to a 500 error when uploading with PHP. I asked Mark a question.
4. Adding a log (Daily) and running CustomLogs (as one should), will create a new log/data only at rollover. That means that between the moment of running CustomLog and the needed data in the new log, the UserAskedData will fail because of missing data file. The workaround here is wait for data or run CustomLogs only after some data is available. Again, user action / reflection is required.
5. The user must be aware that webtag values are always instantaneous values which after some thought I think is correct (see here)

Under all circomstances to correct the situation, at least remove the CustomLogs JSON's from the server, rerun CustomLogs and let the system stabilise again (assuming UserAskedData is run automatically always).

If you think the logfile is too large and contains too much info, set the TraceInfoLevel parameter to Warning, this will greatly reduce the nr of lines in the log but always show important warnings and errors.

If any additional info comes up I will post in this thread.

[AndyKF650]

Hi Hans

I just saw your post to me on this forum. It took me a while but I arrived at the data/chart problem solution before I saw the post and had managed to get a running chart for the temperature custom log to work .

The fix for the realtime data had eluded me but your fix has worked fine and now the page looks as it should. One point that is odd, when you start on the customtemplog page the chart does not automatically populate, you need to go to the daily chart and then back to the customtemplog page to get the full detail.

This is no getting there, I will load some more custom logs in CMX and see how they work.