CMX Hanging

[HansR]

Suddenly CMX started spawning a lot of threads. See htop screenshot.
No access to the RPi was possible. Had to pull the plug and restart.

Corresponding logfile attached.

Schermafbeelding 2022-06-09 155857.jpg

[mcrossley]

Hmm, looks like WoW is having problems, every request gets a "too many requests" error even though they are every 5 minutes.

I suspect that the thread used to perform the WoW update is not getting deleted for some reason and eventually you run out of memory.

I'll take a look, but I'm pretty sure CMX exits the thread and it's going to be mono that isn't closing it properly for some reason. This could be related to another problem report where the http client seemed to be hanging after an error - but that was a real exception rather than the server returning an error.

It also looks like you FTP host disconnects the real-time FTP session after an hour and CMX has to reconnect. That happens every hour.

[HansR]

Wrt the http client, i assume you know MS advised not to dispose of the client but repeatedly use the same http client object for the connection to prevent socket exhaustion? May this be the issue?

[HansR]

Oh, and btw, this never occurred before, maybe it also has to do with SD card aging?

[mcrossley]

Wrt the http client, i assume you know MS advised not to dispose of the client but repeatedly use the same http client object for the connection to prevent socket exhaustion? May this be the issue?

The same client is reused by CMX. A different http client is created for each active upload and reused until CMX is closed.

[HansR]

It happened again two time this evening. Apparently if there is a network error (outside my control) CMX starts behaving badly. This has not happened before while I did have network errors or disconnects before. When switching the modem off/on CMX just waits until the restart, now it climbs rapidly out of memory with 100% filles swap space. If you need the other logfiles as well let me know. It may be confusing info however because I pulled the plug sooner.

Currently I switched CMX off and hope Ecowitt catchup works
I'll get back to this issue saturday.

[HansR]

I created a new SD card (suspected the issue was there) and now I get errors concerning

Code: Select all

2022-06-11 20:57:04.475 ProcessData: Error, missing wind chill

I have no idea what is going on here, I have made no change in the config. Then I switched off Windchill in the Graph settings but nothing works.
What is the issue here, where does this error come from?

[HansR]

Things seem OK: Timezone was wrong
The message pointed me the wrong way.

[mcrossley]

You get that message if you have configured CMX to use the station wind chill value and the value is not present in the data sent.

[HansR]

As said, I created a new SD card and the original problem of creating endless processes seems to have disappeared so it might be related to being unable to write or read a file. It is now running OK for some time and the swap is also not rising rapidly.

The WindChill error I do not understand but it has disappeared.

But I do still have an issue : when starting as a service it does not see the primary sensors (logfile attached) and then it simply goes on and reads all sensors as expected. I will not change anything unless you think I should.
What is wrong (in the service file as below?) ?

My cumulusmx.service file is as follows:

Code: Select all

[Unit]
Description=CumulusMX service
Documentation=https://cumuluswiki.org/a/Main_Page
Wants=network-online.target
After=network-online.target
Wants=time-sync.target
After=time-sync.target

[Service]
User=pi
Group=pi
ExecStart=/usr/bin/mono-service -d:/home/CumulusMX CumulusMX.exe -service
Type=forking
ExecStopPost=/bin/rm -f /tmp/CumulusMX.exe.lock

[Install]
WantedBy=multi-user.target

[mcrossley]

This message?

Code: Select all

2022-06-12 11:47:00.114 Minimum data set of pressure, temperature, and wind is not available and NoSensorCheck is not enabled. Skip processing

It's because the minute has ticked over and no data has yet been received from the station, it's benign but there should really be a check rather than issuing an error message. Just a timing thing.

[HansR]

OK. Thanks. Than everything is OK again and the issue was a worn out SD card.