Welcome to the Cumulus Support forum.

Latest Cumulus MX release v3.0.0 build 3050 - 1 May 2019.
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014

See the Wiki for downloading either version.

Possible memory leak

Discussion of version 3 of Cumulus, which runs on Windows, Linux, and OS X. All Cumulus MX queries in here, please.

Moderator: mcrossley

bazhig
Posts: 2
Joined: Fri 05 Apr 2019 10:47 am
Weather Station: Fine Offset
Operating System: Win 10

Possible memory leak

Post by bazhig » Tue 09 Apr 2019 9:52 am

I have been running MX for a week now and on several occasions have had to kill the task and restart it due to to high cpu usage.
I am running it on Windows 10 64 bit and normally the CPU usage is less than 1% or even 0 then it will jump to around 30% and stay there.
Could not see that this has been reported anywhere else on the forum.
Barry

User avatar
mcrossley
Posts: 5518
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley » Tue 09 Apr 2019 10:26 am

Switch on debug logging, and zip up the diags folder if it happens again.

You're right though, I haven't seen this reported before.

Note: Debug logging will increase the log file sizes, you may want to switch it off again after capturing an event.

bazhig
Posts: 2
Joined: Fri 05 Apr 2019 10:47 am
Weather Station: Fine Offset
Operating System: Win 10

Re: Possible memory leak

Post by bazhig » Thu 11 Apr 2019 10:13 am

Hi Mark
I have started debug log and will forward to you when it happens again
Barry

philpugh
Posts: 132
Joined: Tue 24 May 2011 8:34 am
Weather Station: DAVIS VantageVue
Operating System: Debian Stretch Lite
Location: Antrobus, Cheshire
Contact:

Re: Possible memory leak

Post by philpugh » Sun 21 Apr 2019 2:43 pm

Mark

To add to this I too have noticed this type of behaviour since upgrade to v3047 (I am now on 3048)
I haven't turned debug on yet but have the following observations..

The system is running 'flat out' with the main process taking 94%+ ( I have attached a TOP report of failed system) I also have included the MXdiags for the time of the 'failure'.

I use the Stop/Start script on my system - it's a Pi ZeroW Stretch lite using as TCP/IP connection to a Meteobridge that is directly connected to the Davis Vantage Vue.

In the non working state - even though the main process is at 90%+ - the command sudo ./cumulusmx.sh -r reckons there is NOT an active CumulusMX process but it does successfully restart CumulusMX.

I am away for a few days so I have set the system to restart CumulusMX at 1am every day as a 'workaround'.
You do not have the required permissions to view the files attached to this post.
Phil Pugh
Davis Vantage Vue Raspberry Pi Zero W Stretch-Lite
Using TCP/IP connection to Meteobridge
http://goosegate.plus.com/

User avatar
mcrossley
Posts: 5518
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley » Sun 21 Apr 2019 3:01 pm

It almost looks as if the network stack has crashed in some way?

What is the Main process? Is it a result of the script? I see the Cumulus process as mono which is what is actually running the code.

Can't say I seen anything like this, and the longest I have left Cumulus running is about 8 months. I restart it more frequently since then due to testing new builds.

Does this problem occur if you just run Cumulus from the command line rather than via the script? I'm afraid it's too much to support issues that may be due to the method of launch.

philpugh
Posts: 132
Joined: Tue 24 May 2011 8:34 am
Weather Station: DAVIS VantageVue
Operating System: Debian Stretch Lite
Location: Antrobus, Cheshire
Contact:

Re: Possible memory leak

Post by philpugh » Mon 22 Apr 2019 2:53 pm

Mark

i have a second system (Weatherflow + meteobridge) logging into the same database server. This appears to have kept on working during this time - I will do some more investigations when I am back home later this week.

Phil
Phil Pugh
Davis Vantage Vue Raspberry Pi Zero W Stretch-Lite
Using TCP/IP connection to Meteobridge
http://goosegate.plus.com/

philpugh
Posts: 132
Joined: Tue 24 May 2011 8:34 am
Weather Station: DAVIS VantageVue
Operating System: Debian Stretch Lite
Location: Antrobus, Cheshire
Contact:

Re: Possible memory leak

Post by philpugh » Sat 27 Apr 2019 12:13 pm

OK it 'failed' again in the early morning and I had debug logging turned on.

I now have a 1.3Gb log file! I have included snippets from this at what I see to be the possible areas to look at.

It does look like a network stack issue but the connection is to my main WWW and MariaDB server is still operational as far as other data collection systems are concerned.

I have a Weatherflow system and a home brew solar power meter logging into the same server - none of these see a break in network connection.

So, attached are two files (a TOP output when 'failed' showing high CPU usage and the selected extracts from the log file).

In this case CumulusMX was started at reboot by a cron job with the following commands..

@reboot (sleep 30;cd /home/pi/CumulusMX;sudo mono CumulusMX.exe >> /dev/null) &

TOP still reports it as Main for the command - possibly because it is started via cron??

The sleep allows the wifi to get a link established at restart.
You do not have the required permissions to view the files attached to this post.
Phil Pugh
Davis Vantage Vue Raspberry Pi Zero W Stretch-Lite
Using TCP/IP connection to Meteobridge
http://goosegate.plus.com/

philpugh
Posts: 132
Joined: Tue 24 May 2011 8:34 am
Weather Station: DAVIS VantageVue
Operating System: Debian Stretch Lite
Location: Antrobus, Cheshire
Contact:

Re: Possible memory leak

Post by philpugh » Sat 27 Apr 2019 12:15 pm

One further thing....

The last update of index.htm was at 03:05:00 and the log file after this shows similar entries for the rest of the file.
Phil Pugh
Davis Vantage Vue Raspberry Pi Zero W Stretch-Lite
Using TCP/IP connection to Meteobridge
http://goosegate.plus.com/

freddie
Posts: 430
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2
Operating System: GNU/Linux (Ubuntu 18.04)
Location: Dorrington, Shropshire
Contact:

Re: Possible memory leak

Post by freddie » Sat 27 Apr 2019 1:19 pm

Looks as if your realtime FTP is running into problems. There is an entry in the log that says it is no longer connected, and then doesn't seem to be able to progress from that point. Possibly a freeing of resources issue in the program. You can see (from your index.htm update time and log messages of successful transfers to your database and to Windy as well) that networking is working, but is maybe being bogged down by repeated realtime FTP connections? It may be an idea to switch on FTP logging.

You should still see Mono in the process list,regardless of how the application is started.
Freddie
Image

freddie
Posts: 430
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2
Operating System: GNU/Linux (Ubuntu 18.04)
Location: Dorrington, Shropshire
Contact:

Re: Possible memory leak

Post by freddie » Sat 27 Apr 2019 1:20 pm

Also, see if the problems stop if you run with realtime switched off for a while.
Freddie
Image

User avatar
mcrossley
Posts: 5518
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley » Sat 27 Apr 2019 3:01 pm

I agree, let's see an FTP log. Odd how the MySQL stops as well. It does look like it is running out of resources, Ill have another look but I'm pretty certain the realtime reconnect closes the old connection correctly so it should be available for garbage collection - of course if it is hung up for some reason the garbage collection may never run or hang itself.

philpugh
Posts: 132
Joined: Tue 24 May 2011 8:34 am
Weather Station: DAVIS VantageVue
Operating System: Debian Stretch Lite
Location: Antrobus, Cheshire
Contact:

Re: Possible memory leak

Post by philpugh » Sat 27 Apr 2019 5:54 pm

I have initiated FTP log - one other thing, on closer inspection of the debug log it is getting messages like ..

2019-04-26 23:24:18.548 Test of realtime FTP connection failed: Timed out trying to read data from the socket stream!
2019-04-26 23:24:18.549 Realtime ftp not connected - reconnecting
2019-04-26 23:24:30.824 !!! loop data not received
2019-04-26 23:24:30.825 Sending command: LPS 2 1, attempt 1
2019-04-26 23:24:31.827 Wait for ACK
2019-04-26 23:24:31.827 Wait for ACK
2019-04-26 23:24:31.854 CANCEL received
2019-04-26 23:24:31.855 Sending command: LPS 2 1, attempt 2
2019-04-26 23:24:32.450 Warning, previous realtime ftp still in progress, skipping this period.
2019-04-26 23:24:32.863 Wait for ACK
2019-04-26 23:24:32.863 Wait for ACK
2019-04-26 23:24:32.864 ACK received
2019-04-26 23:24:32.868 10-min gust from loop2: 12.0
2019-04-26 23:24:32.869 Sending command: LOOP 50, attempt 1
2019-04-26 23:24:33.871 Wait for ACK
2019-04-26 23:24:33.871 Wait for ACK
2019-04-26 23:24:33.873 ACK received
2019-04-26 23:24:33.874 invalid LOOP packet
2019-04-26 23:24:36.876 Discarding bytes from pipeline: 300

With invalid Loop packets.
Phil Pugh
Davis Vantage Vue Raspberry Pi Zero W Stretch-Lite
Using TCP/IP connection to Meteobridge
http://goosegate.plus.com/

User avatar
mcrossley
Posts: 5518
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Possible memory leak

Post by mcrossley » Sat 27 Apr 2019 6:08 pm

Phil, this is connecting to a Meteobridge rather direct to the console?
The Meteobridge connections are a lot less reliable than using a Davis IP logger. When working on beta versions of MX with other users, I put in extra checking and reconnecting to try and compensate but ...
For instance sometimes you open the TCP connection ok and it doesn't respond at all.

freddie
Posts: 430
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2
Operating System: GNU/Linux (Ubuntu 18.04)
Location: Dorrington, Shropshire
Contact:

Re: Possible memory leak

Post by freddie » Sat 27 Apr 2019 6:24 pm

philpugh wrote:
Sat 27 Apr 2019 5:54 pm

With invalid Loop packets.
That's probably down to resource exhaustion.
Freddie
Image

philpugh
Posts: 132
Joined: Tue 24 May 2011 8:34 am
Weather Station: DAVIS VantageVue
Operating System: Debian Stretch Lite
Location: Antrobus, Cheshire
Contact:

Re: Possible memory leak

Post by philpugh » Sun 28 Apr 2019 10:30 am

It went out again last night...

Final update of index.htm was 23:05:00 although the attached logs show it tried to upload at 23:10:00

Two files attached - the entire FTPlog and the MXdiags log up to and including the problem area.

NOTE Looking at the ftplog it would appear that the real time files continued to be uploaded - although there seemed to be problems occasionally.
You do not have the required permissions to view the files attached to this post.
Phil Pugh
Davis Vantage Vue Raspberry Pi Zero W Stretch-Lite
Using TCP/IP connection to Meteobridge
http://goosegate.plus.com/

Post Reply