Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Possible memory leak

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

bazhig
Posts: 2
Joined: Fri 05 Apr 2019 10:47 am
Weather Station: Fine Offset
Operating System: Win 10

Possible memory leak

Post by bazhig »

I have been running MX for a week now and on several occasions have had to kill the task and restart it due to to high cpu usage.
I am running it on Windows 10 64 bit and normally the CPU usage is less than 1% or even 0 then it will jump to around 30% and stay there.
Could not see that this has been reported anywhere else on the forum.
Barry
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley »

Switch on debug logging, and zip up the diags folder if it happens again.

You're right though, I haven't seen this reported before.

Note: Debug logging will increase the log file sizes, you may want to switch it off again after capturing an event.
bazhig
Posts: 2
Joined: Fri 05 Apr 2019 10:47 am
Weather Station: Fine Offset
Operating System: Win 10

Re: Possible memory leak

Post by bazhig »

Hi Mark
I have started debug log and will forward to you when it happens again
Barry
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: Possible memory leak

Post by philpugh »

Mark

To add to this I too have noticed this type of behaviour since upgrade to v3047 (I am now on 3048)
I haven't turned debug on yet but have the following observations..

The system is running 'flat out' with the main process taking 94%+ ( I have attached a TOP report of failed system) I also have included the MXdiags for the time of the 'failure'.

I use the Stop/Start script on my system - it's a Pi ZeroW Stretch lite using as TCP/IP connection to a Meteobridge that is directly connected to the Davis Vantage Vue.

In the non working state - even though the main process is at 90%+ - the command sudo ./cumulusmx.sh -r reckons there is NOT an active CumulusMX process but it does successfully restart CumulusMX.

I am away for a few days so I have set the system to restart CumulusMX at 1am every day as a 'workaround'.
You do not have the required permissions to view the files attached to this post.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley »

It almost looks as if the network stack has crashed in some way?

What is the Main process? Is it a result of the script? I see the Cumulus process as mono which is what is actually running the code.

Can't say I seen anything like this, and the longest I have left Cumulus running is about 8 months. I restart it more frequently since then due to testing new builds.

Does this problem occur if you just run Cumulus from the command line rather than via the script? I'm afraid it's too much to support issues that may be due to the method of launch.
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: Possible memory leak

Post by philpugh »

Mark

i have a second system (Weatherflow + meteobridge) logging into the same database server. This appears to have kept on working during this time - I will do some more investigations when I am back home later this week.

Phil
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: Possible memory leak

Post by philpugh »

OK it 'failed' again in the early morning and I had debug logging turned on.

I now have a 1.3Gb log file! I have included snippets from this at what I see to be the possible areas to look at.

It does look like a network stack issue but the connection is to my main WWW and MariaDB server is still operational as far as other data collection systems are concerned.

I have a Weatherflow system and a home brew solar power meter logging into the same server - none of these see a break in network connection.

So, attached are two files (a TOP output when 'failed' showing high CPU usage and the selected extracts from the log file).

In this case CumulusMX was started at reboot by a cron job with the following commands..

@reboot (sleep 30;cd /home/pi/CumulusMX;sudo mono CumulusMX.exe >> /dev/null) &

TOP still reports it as Main for the command - possibly because it is started via cron??

The sleep allows the wifi to get a link established at restart.
You do not have the required permissions to view the files attached to this post.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: Possible memory leak

Post by philpugh »

One further thing....

The last update of index.htm was at 03:05:00 and the log file after this shows similar entries for the rest of the file.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
freddie
Posts: 2434
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Possible memory leak

Post by freddie »

Looks as if your realtime FTP is running into problems. There is an entry in the log that says it is no longer connected, and then doesn't seem to be able to progress from that point. Possibly a freeing of resources issue in the program. You can see (from your index.htm update time and log messages of successful transfers to your database and to Windy as well) that networking is working, but is maybe being bogged down by repeated realtime FTP connections? It may be an idea to switch on FTP logging.

You should still see Mono in the process list,regardless of how the application is started.
Freddie
Image
freddie
Posts: 2434
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Possible memory leak

Post by freddie »

Also, see if the problems stop if you run with realtime switched off for a while.
Freddie
Image
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley »

I agree, let's see an FTP log. Odd how the MySQL stops as well. It does look like it is running out of resources, Ill have another look but I'm pretty certain the realtime reconnect closes the old connection correctly so it should be available for garbage collection - of course if it is hung up for some reason the garbage collection may never run or hang itself.
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: Possible memory leak

Post by philpugh »

I have initiated FTP log - one other thing, on closer inspection of the debug log it is getting messages like ..

2019-04-26 23:24:18.548 Test of realtime FTP connection failed: Timed out trying to read data from the socket stream!
2019-04-26 23:24:18.549 Realtime ftp not connected - reconnecting
2019-04-26 23:24:30.824 !!! loop data not received
2019-04-26 23:24:30.825 Sending command: LPS 2 1, attempt 1
2019-04-26 23:24:31.827 Wait for ACK
2019-04-26 23:24:31.827 Wait for ACK
2019-04-26 23:24:31.854 CANCEL received
2019-04-26 23:24:31.855 Sending command: LPS 2 1, attempt 2
2019-04-26 23:24:32.450 Warning, previous realtime ftp still in progress, skipping this period.
2019-04-26 23:24:32.863 Wait for ACK
2019-04-26 23:24:32.863 Wait for ACK
2019-04-26 23:24:32.864 ACK received
2019-04-26 23:24:32.868 10-min gust from loop2: 12.0
2019-04-26 23:24:32.869 Sending command: LOOP 50, attempt 1
2019-04-26 23:24:33.871 Wait for ACK
2019-04-26 23:24:33.871 Wait for ACK
2019-04-26 23:24:33.873 ACK received
2019-04-26 23:24:33.874 invalid LOOP packet
2019-04-26 23:24:36.876 Discarding bytes from pipeline: 300

With invalid Loop packets.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley »

Phil, this is connecting to a Meteobridge rather direct to the console?
The Meteobridge connections are a lot less reliable than using a Davis IP logger. When working on beta versions of MX with other users, I put in extra checking and reconnecting to try and compensate but ...
For instance sometimes you open the TCP connection ok and it doesn't respond at all.
freddie
Posts: 2434
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Possible memory leak

Post by freddie »

philpugh wrote: Sat 27 Apr 2019 5:54 pm
With invalid Loop packets.
That's probably down to resource exhaustion.
Freddie
Image
User avatar
philpugh
Posts: 381
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: Possible memory leak

Post by philpugh »

It went out again last night...

Final update of index.htm was 23:05:00 although the attached logs show it tried to upload at 23:10:00

Two files attached - the entire FTPlog and the MXdiags log up to and including the problem area.

NOTE Looking at the ftplog it would appear that the real time files continued to be uploaded - although there seemed to be problems occasionally.
You do not have the required permissions to view the files attached to this post.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
Post Reply