Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Degraded allocation warning messages

Topics about the Beta trials up to Build 3043, the last build by Cumulus's founder Steve Loft. It was by this time way out of Beta but Steve wanted to keep it that way until he made a decision on his and Cumulus's future.

Moderator: mcrossley

senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Degraded allocation warning messages

Post by senapsys »

Hi there

I’ve been running CumulusMX on a RPi 2 for about 2 weeks now. I had no issues with installing it and it’s been running fine.

I am however seeing error messages from time to time as per below:

Warning: Degraded allocation. Consider increasing nursery-size if the warning persists.
Warning: Degraded allocation. Consider increasing nursery-size if the warning persists.
Warning: Repeated degraded allocation. Consider increasing nursery-size.

CumulusMX still appears to be running ok despite these error messages. Typically they start appearing 2-3 days after the application has been started.

My version information is as follows:

Raspberry Pi 2 running the Jessie release of Raspbian, patched with the latest updates as of about 1 week ago.
CumulusMX - B3034
Mono – 3.2.8

I was previously running B3033 and I also noticed the same message every now and then.

I did a quick search through the latest log file within the MXdiags folder, and I’m not seeing any log entries that match this.

In terms of CPU usage, it’s typically sitting at less than 5% with the occasional spike that ranges between 15 and 35% or thereabouts

Any thoughts on whether this is an error that I should be concerned about, or whether I need to tweak any configuration files would be appreciated.

Regards

Richard
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Post by steve »

It seems to be caused when large numbers of threads are being used, but I don't know why it happens to you.

http://stackoverflow.com/questions/1846 ... g-persists

I think someone else had the same message, and they were having problems with their SD card. Are you using an SD card for MX? SD card corruption is a common problem with the Pi.
Steve
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Degraded allocation warning messages

Post by senapsys »

Hi Steve

I'm aware of the SD card corruption issue so from the very beginning I built this RPi using Berryboot so that I could boot the entire system from a USB connected SSD Drive.

As a precaution I did a "fsck" but everything came through clean and no issues were highlighted.

Regards

Richard
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Post by steve »

You could have a look at the thread usage on your system, something is apparently using lots of threads, quite possibly MX. If it's MX we would need to work out why it's doing that on your system.
Steve
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Degraded allocation warning messages

Post by senapsys »

Hi Steve

I've attached two files, one is a dump of all of the threads associated with the Mono process that is running CumulusMX after the Degraded Allocation messages start appearing. The other is a dump of the threads associated with the Mono process that is running CumulusMX after a restart of Mono. In the degraded state, Mono has 166 threads associated with it whereas just after Normal running appears, there are just 13.

Note that I'm not using Mono for anything else and a search through the process list has confirmed that there are no other instances of Mono running.

I'm reasonably familiar with Linux and there are a bunch of other stats and info I can pull out of the system. Let me know what else might be of use to you and I'll grab them for you.

Regards

Richard
You do not have the required permissions to view the files attached to this post.
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Post by steve »

It does use threads for several things, but 166 seems like quite a lot. I wonder if you are having ftp problems, and the threads which handle ftp are hanging up? Is there any way to find out what those threads are actually doing? I suspect that you know far more about Linux than I do!
Steve
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Degraded allocation warning messages

Post by senapsys »

Hi Steve

I'll do some digging and see what I can find. It might be a day or so before I post again as I have just restarted the application so I'm not seeing the accumulation of threads yet.

One thing I did notice is that there are a lot of threads that are sitting at roughly 6 minutes and 25 seconds. Does this roughly correlate with the scheduled timing any of internal process within CumulusMX?

Regards

Richard
User avatar
rogerthn
Posts: 489
Joined: Thu 11 Apr 2013 6:31 pm
Weather Station: Ecowitt GW1000/GW1003
Operating System: Raspberry Pi OS bullseye aarch64
Location: Trollhättan Sweden
Contact:

Re: Degraded allocation warning messages

Post by rogerthn »

One of my favorites :P

Code: Select all

netstat -taupeen
It could return a lot but sort -kX and grep will help
Image
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Post by steve »

senapsys wrote:a lot of threads that are sitting at roughly 6 minutes and 25 seconds. Does this roughly correlate with the scheduled timing any of internal process within CumulusMX?
Nothing comes to mind. I had a look on my Pi and MX has 13 threads, just as you do at startup, and mine has been running for about 18 hours.
Steve
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Degraded allocation warning messages

Post by senapsys »

Hi Steve

An update for you.

After my last post I deployed some scripts to monitor the number of threads as well as a bunch of other system performance data. As is the case with any fault finding, the moment I put these scripts in place the Mono/CumulusMX app got stage fright and the error didn't manifest itself after the usual 2-3 days :-) Like you, after the app displayed the "Normal running" message and I also observed 13 threads. Sometimes the number of threads increased up to about 17 threads for a brief moment and then returned to 13 or 15 threads. This "quiescent" level of thread variation appeared to be the normal behaviour that I observed over the days whilst waiting for the error to manifest itself again. I'm not sure if this is consistent with what you are seeing?

Anyway, after 9 days I finally started to notice something happening. I got lucky in that I was looking at the system as the thread count started to slowly increase. The increase in threads wasn't dramatic but rather happened at a much more pedestrian pace with an extra one being spawned every 5-10 minutes or thereabouts although this wasn't entirely consistent.

As this was happening I started to look at the various system stats and the one that stood out was an unusually high number of TCP sessions between the RPi system and the FTP server that hadn't been closed properly after a reasonable period of time. As an experiment I logged into the system which was running the FTP server and restarted the FTP service. As soon as I did this the slow increase in threads stopped and it went back to a scenario where the threads would briefly increase and then reduce, albeit with a higher base number of threads.

It's not a definitive answer and I'll need to put in place some more specific monitoring on both the RPi and the FTP server but the initial observations point to an issue with FTP. Whether the fault is with Mono/CumulusMX or the FTP server is impossible to say at this point. I'll do some more monitoring and see whether I'm on the right track.

Regards

Richard
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Post by steve »

I did suspect that it would turn out to be FTP connections not closing down. Turning on the ftp log may give some useful info from the Cumulus end - note that realtime ftp gets logged to the same file, so it can get large quickly.
Steve
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Degraded allocation warning messages

Post by senapsys »

Hi Steve

I turned on FTP Logging in CumulusMX and have attached a log file from the last 24 hours during which time the thread count started to increase. To my untrained eye the only thing that really stands out is a number of instances where the multiple consecutive messages of "Disposing FtpSocketStream..." appear in the file.

The server FTP log didn't reveal anything that stood out so I have increased the level of logging to see if I pick anything up next time.

I hope this is of use to you :-)

Regards

Richard
You do not have the required permissions to view the files attached to this post.
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Degraded allocation warning messages

Post by steve »

I can't see anything there that explains the problem, I'm afraid.
Steve
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Degraded allocation warning messages

Post by senapsys »

Hi Steve

Thanks for looking.

Noting my previous experience where restarting the FTP Server stopped Mono/CumulusMX from spawning additional threads it does point to the root cause being the FTP Server.

I'll put the microscope on the FTP Sever and see whether I can find out what the issue is and report back in due course.

Have a great Christmas/New Years :-)

Regards

Richard
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Degraded allocation warning messages

Post by senapsys »

Hi Steve

I've been looking over various logs and one thing out of the ordinary which I have noticed in the CumulusMX log in the MXdiags directory is a few messages along the lines of:

Error uploading realtime.txt to cudata/realtime.txt : Access is denied.

When I look at the FTP Log within CumulusMX for the corresponding time I can't seem to see anything that correlates with the time stamp of the access denied message so I'm not sure if this is significant or not. I thought I'd mention it in case it's useful.

Unfortunately it's a bit random as to when the issue manifests but as soon as it happens again I'll go hunting through the server FTP logs and see whether I can see anything at that end.

Regards

Richard
Locked