Page 1 of 1

CMX Hanging

Posted: Thu 09 Jun 2022 2:05 pm
by HansR
Suddenly CMX started spawning a lot of threads. See htop screenshot.
No access to the RPi was possible. Had to pull the plug and restart.

Corresponding logfile attached.
    Schermafbeelding 2022-06-09 155857.jpg

    Re: CMX Hanging

    Posted: Thu 09 Jun 2022 2:25 pm
    by mcrossley
    Hmm, looks like WoW is having problems, every request gets a "too many requests" error even though they are every 5 minutes.

    I suspect that the thread used to perform the WoW update is not getting deleted for some reason and eventually you run out of memory.

    I'll take a look, but I'm pretty sure CMX exits the thread and it's going to be mono that isn't closing it properly for some reason. This could be related to another problem report where the http client seemed to be hanging after an error - but that was a real exception rather than the server returning an error.

    It also looks like you FTP host disconnects the real-time FTP session after an hour and CMX has to reconnect. That happens every hour.

    Re: CMX Hanging

    Posted: Thu 09 Jun 2022 2:50 pm
    by HansR
    Wrt the http client, i assume you know MS advised not to dispose of the client but repeatedly use the same http client object for the connection to prevent socket exhaustion? May this be the issue?

    Re: CMX Hanging

    Posted: Thu 09 Jun 2022 2:51 pm
    by HansR
    Oh, and btw, this never occurred before, maybe it also has to do with SD card aging?

    Re: CMX Hanging

    Posted: Thu 09 Jun 2022 2:54 pm
    by mcrossley
    HansR wrote: Thu 09 Jun 2022 2:50 pm Wrt the http client, i assume you know MS advised not to dispose of the client but repeatedly use the same http client object for the connection to prevent socket exhaustion? May this be the issue?
    The same client is reused by CMX. A different http client is created for each active upload and reused until CMX is closed.

    Re: CMX Hanging

    Posted: Thu 09 Jun 2022 8:08 pm
    by HansR
    It happened again two time this evening. Apparently if there is a network error (outside my control) CMX starts behaving badly. This has not happened before while I did have network errors or disconnects before. When switching the modem off/on CMX just waits until the restart, now it climbs rapidly out of memory with 100% filles swap space. If you need the other logfiles as well let me know. It may be confusing info however because I pulled the plug sooner.

    Currently I switched CMX off and hope Ecowitt catchup works :)
    I'll get back to this issue saturday.

    Re: CMX Hanging

    Posted: Sat 11 Jun 2022 8:06 pm
    by HansR
    I created a new SD card (suspected the issue was there) and now I get errors concerning

    Code: Select all

    2022-06-11 20:57:04.475 ProcessData: Error, missing wind chill
    
    I have no idea what is going on here, I have made no change in the config. Then I switched off Windchill in the Graph settings but nothing works.
    What is the issue here, where does this error come from?

    Re: CMX Hanging

    Posted: Sat 11 Jun 2022 8:27 pm
    by HansR
    Things seem OK: Timezone was wrong :bash:
    The message pointed me the wrong way.

    Re: CMX Hanging

    Posted: Sat 11 Jun 2022 8:39 pm
    by mcrossley
    You get that message if you have configured CMX to use the station wind chill value and the value is not present in the data sent.

    Re: CMX Hanging

    Posted: Sun 12 Jun 2022 9:58 am
    by HansR
    As said, I created a new SD card and the original problem of creating endless processes seems to have disappeared so it might be related to being unable to write or read a file. It is now running OK for some time and the swap is also not rising rapidly.

    The WindChill error I do not understand but it has disappeared.

    But I do still have an issue : when starting as a service it does not see the primary sensors (logfile attached) and then it simply goes on and reads all sensors as expected. I will not change anything unless you think I should.
    What is wrong (in the service file as below?) ?

    My cumulusmx.service file is as follows:

    Code: Select all

    [Unit]
    Description=CumulusMX service
    Documentation=https://cumuluswiki.org/a/Main_Page
    Wants=network-online.target
    After=network-online.target
    Wants=time-sync.target
    After=time-sync.target
    
    [Service]
    User=pi
    Group=pi
    ExecStart=/usr/bin/mono-service -d:/home/CumulusMX CumulusMX.exe -service
    Type=forking
    ExecStopPost=/bin/rm -f /tmp/CumulusMX.exe.lock
    
    [Install]
    WantedBy=multi-user.target
    

    Re: CMX Hanging

    Posted: Sun 12 Jun 2022 12:49 pm
    by mcrossley
    This message?

    Code: Select all

    2022-06-12 11:47:00.114 Minimum data set of pressure, temperature, and wind is not available and NoSensorCheck is not enabled. Skip processing
    
    It's because the minute has ticked over and no data has yet been received from the station, it's benign but there should really be a check rather than issuing an error message. Just a timing thing.

    Re: CMX Hanging

    Posted: Sun 12 Jun 2022 1:11 pm
    by HansR
    OK. Thanks. Than everything is OK again and the issue was a worn out SD card.