cutils extra sensors

[HansR]

Could you show again the current crontab please.

[Nossie]

Code: Select all

  GNU nano 7.2                                        /tmp/crontab.c4yqaP/crontab
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

0 1-23/6 * * * sudo /home/pi/IOTstack/Archive.sh
0 3 * * *      sudo /home/pi/IOTstack/ArchiveN.sh
#0 1-23/6 * * * sudo docker exec -i MXWeather mono utils/bin/cumulusutils.exe website
6 0  * * *     sudo docker exec -i MXWeather mono utils/bin/cumulusutils.exe thrifty website
*/10 * * * *   sudo docker exec -i MXWeather mono utils/bin/cumulusutils.exe useraskeddata
*/15 * * * *   sudo docker exec -i MXWeather mono utils/bin/cumulusutils.exe sysinfo

Current full contrab

sidenote - I'm getting windbarbs - something I've wanted but couldn't get working before.

[HansR]

OK, thanks, looks OK.

But I thought the internet Interval was 15 mins so I would put the UserAskedData on 15 minutes as well. Put SysInfo and UserAskedData on the same commandline:

Code: Select all

*/15 * * * *   sudo docker exec -i MXWeather mono utils/bin/cumulusutils.exe sysinfo UserAskedData

That would at least make things in sync. It is no use to run UserAskedData if you know the result will be no data so the call must be at least on the frequency of the interval. This is the only difference with my Test Env I can see.

If that does not work I am out of ideas.

Oh... and at least you have the windbarbs

[HansR]

So apparently it is solved with the synchronisation of that crontab command with the CMX Interval.
I'll have to do something with that
Now find out why your CO2 drops to 0 every 15 (!) minutes or so.
[EDIT:] ah, I see you made another topic for this... it is definitely a CMX or sensor issue.

Could you also please re-enter your URL in the map? Thnx.

[HansR]

To make some additional remarks (I continue experimenting) seeing that you activated the lightning sensor again:

1. Your CUtils continues to generate a time series for the lightning sensor. That baffles me. It can't with that version - 6.23.1 - and it does not do that on my test system.
2. The problem appeared to be a UserAskedData command run on a lower frequency than the CMX Interval frequency. So I run my test system now with an Interval of 15 minutes and UserAskedData on 10 minutes in crontab (just like you do/did). As a result I do NOT observe any issue and the charts data generate fine.

So the acute problem may be solved with the synchronisation of the crontab and the Interval, there still is something very strange on your system which I cannot reproduce on my test system.

We'll leave it for now and come back to it when either of us has more clarity about the cause.

[Nossie]

Your CUtils continues to generate a time series for the lightning sensor. That baffles me. It can't with that version - 6.23.1

did it ever do that and is this the result of 2 versions mashed together? I have never modified anything like that, but I may have screwed up an upgrade or install back in the summer.

[HansR]

did it ever do that and is this the result of 2 versions mashed together? I have never modified anything like that, but I may have screwed up an upgrade or install back in the summer.

It did have issues for that in the past yes.
I was wondering: is Docker sensitive to the specific definition of an executable to a container? In other words: do you have to bind an update to your container to redefine the situation (I am totally blank wrt to Docker)

[Nossie]

Not 100% sure what you mean, I bind locations, not binaries.


I'm going to attach a screenshot of something, the top left is the inside of the container - I can remove that completely - download a new version and reinstall in 5 minutes. The right hand side is badly cobbled together way of making it 'easier' to manage remotely - but I do 'pin' these files and the data is kept. As far as cumulus is concerned it's the same file in the same place and it can be written to or updated as normal. The files and folders are mounted on a totally different drive and is shared as a samba share. The bottom image is the samba share from a windows perspective. To update cumulusutils - I would download the latest zip, extract into the utils folder, restart the container and run an update.

I originally wanted to keep all my data in one place, configs in another etc etc - but I'd have been just as well off just sharing the whole folder from the beginning. Running cumulus as a container is really quick and easy, running cumulus and utils at the same time is not so much.

To give you an example of the current system, when I said I was going to remove the cumulusutils.ini and start with a new one. I removed the file, restarted and the container errored out because that file was not present. I then did "touch cumulusutils.ini" and restarted again - utils then found the file and wrote over it with the default config. If I was building the container to also have utils then this issue would be mitigated, but I wanted to keep the container clean so people with no interest in utils could still use it.

Being able to have these mounts is honestly a really powerful feature - but it does add drawbacks, such as the 'touching' of files and in the past I've had problems where if I don't have the file and folder permissions correct I'd save something and restart but it would still try to load a local file rather than where it should be.

Although I say this, it's possible that my problem was created in that I removed some files without thinking that I didn't have a backup for and then tried to restore those files from an older backup - the cumulusutils.ini being one of them So I do think this is more user error than something docker has done wrong. However what I really don't understand is where in the world it would be trying to get this lightning information from - But I do not think it's a coincidence that I had my brain fart inbetween you having a preliminary release of the lightning feature and then you optimizing it later - which is why I'm bringing this up.

I'm cringing just looking at how those volumes are setup haha

[HansR]

Although I say this, it's possible that my problem was created in that I removed some files without thinking that I didn't have a backup for and then tried to restore those files from an older backup - the cumulusutils.ini being one of them So I do think this is more user error than something docker has done wrong. However what I really don't understand is where in the world it would be trying to get this lightning information from - But I do not think it's a coincidence that I had my brain fart inbetween you having a preliminary release of the lightning feature and then you optimizing it later - which is why I'm bringing this up.

I'm cringing just looking at how those volumes are setup haha

OK, thanks for this explanation, I think we are converging on where the problem lies.

You are pointing to "where in the world it would be trying to get this lightning information from".
In that you have to remember CUtils is always multi step:

1. First it runs, reads data and generates code files to some local directory (utils)
2. Second it tries to upload to the webserver (which I believe you have local as well) those files it generated just before.
3. When running UserAskedData it does not generate code but it generates JSON's, uploading those after generation.

My observation is that the server still has the old code, trying the time series for the lightning sensor. So either it does not generate it (old executable) or it does not upload a generated file but takes an old version of that generated file. Something similar seems to be the case with the data generation because neither code nor data error can be reproduced by me.

It seems as if Docker does not understand the concept of 'new file' (either downloaded or generated)
I'll let it sink in and try to come up with a more clever answer but I have to go now for several hours.
Maybe some other Docker user can jump in to assist.

[Nossie]

Strangely as far as utils is concerned, only the config file is being tied up in my install - all the other files are just direct drop replacements from the zip. Is it possible that I screwed this up back then and now your binary is no longer trying to convert it because it assumes it's already done?

As you can see in the screenshot, I renamed the whole utils folder just so I could keep my menus etc so I don't believe it's using the old binary.

Can you confirm what the file would be called once it was generated? Before it's uploaded to the website.


Thanks.

[HansR]

Strangley as far as utils is concerned, only the config file is being tied up in my install - all the other files are just direct drop replacements from the zip. Is it possible that I screwed this up back then and now your binary is no longer trying to convert it because it assumes it's already done?

As you can see in the screenshot, I renamed the whole utils folder just so I could keep my menus etc so I don't believe it's using the old binary.

Can you confirm what the file would be called once it was generated? Before it's uploaded to the website.


Thanks.

For the ExtraSensors the following files are generated:

1. extrasensors.txt : contains the HTML / javascript code for the module and the table values
2. extrasensorscharts.txt : contains the charts for the extra sensors. This is also where I see - in the debugger / F12 - that the lightning sensor is taken in a time series. Specifically check the function extrasensorsdataAjax. Note that in my extrasensorscharts.txt the variable Lightning does not occur while in your version on the server it is full of the word Lightning. You should check if that also is the case in the version in the utils directory where you think the recent generated version is.
3. extrasensorsrealime.txt : contains the webtags of the extrasensor values and is registered as an Extra Webfile. The lightning webtags are listed here because they will generate the values for the left pane table.
4. CutilsCharts.def : only partly for the ExtraSensors. Has its own section. As far as I understand this one also contains a chart for the Lightning sensor but it should not (mine hasn't) in version 6.23.1

The attention lies on the charts (the generated javascript and the generated CutilsCharts.def).

[Nossie]

Code: Select all

Chart ExtraCO2 Title Trend chart of Extra CO2 Sensors
  Plot Extra CO2
  Plot Extra CO2_24h
  Plot Extra CO2_pm2p5
  Plot Extra CO2_pm2p5_24h
  Plot Extra CO2_pm10
  Plot Extra CO2_pm10_24h
  Plot Extra CO2_temp
  Plot Extra CO2_hum
  Plot Extra Lightning
EndChart

When I turn off lightning, it gets removed from this block - when I add it, it gets added to this block

I'm guessing lightning should not be getting snuck into the CO2 sensor - right?

Code: Select all

[ExtraSensors]
ExtraSensors=true
ParticipatesSensorCommunity=true
ExternalExtraSensors=
UserModificationExtraSensorCharts=false
ExtraTemp=1,2,3,4,5
ExtraHum=1,3,5
ExtraDP=1,3,5
SoilTemp=1
SoilMoisture=1,2,3
AirQuality=
UserTemp=
LeafTemp=
LeafWetness=1
SoilTemperature=1
CO2=1
LightningSensor=true
CleanupExtraSensorslog=false

Also - just to check, lightningsensor should always be true and not 1?

I have removed the CO2 sensor for now - it has also removed the lightning error - can you confirm the copyright of this file should be up to 2021?

[HansR]

Code: Select all

Chart ExtraCO2 Title Trend chart of Extra CO2 Sensors
  Plot Extra CO2
  Plot Extra CO2_24h
  Plot Extra CO2_pm2p5
  Plot Extra CO2_pm2p5_24h
  Plot Extra CO2_pm10
  Plot Extra CO2_pm10_24h
  Plot Extra CO2_temp
  Plot Extra CO2_hum
  Plot Extra Lightning
EndChart

When I turn off lightning, it gets removed from this block - when I add it, it gets added to this block

I'm guessing lightning should not be getting snuck into the CO2 sensor - right?

Wow... that is an unexpected surprise. No, this should not happen.
I'll work a fix and release ASAP.

Also - just to check, lightningsensor should always be true and not 1?

true or false (see Wiki).
Definitely not 1 (will be interpreted as false).

I have removed the CO2 sensor for now - it has also removed the lightning error - can you confirm the copyright of this file should be up to 2021?

Thnx, will work that dynamically through the whole generation of the files.
No hurry, copyright is valid many years beyond the claimed date.

[HansR]

OK, found the problem. Won't fix it tonight though, maybe tomorrow otherwise the weekend.
I'll give you a beta first.

[Nossie]

Thanks Hans I can stop driving myself crazy now... bit more concerned about the sensor drops though - but ecowitt doesn't show any of them \o/