Page 1 of 2

Degraded allocation warning messages

Posted: Thu 03 Dec 2015 10:23 pm
by senapsys
Hi there

I’ve been running CumulusMX on a RPi 2 for about 2 weeks now. I had no issues with installing it and it’s been running fine.

I am however seeing error messages from time to time as per below:

Warning: Degraded allocation. Consider increasing nursery-size if the warning persists.
Warning: Degraded allocation. Consider increasing nursery-size if the warning persists.
Warning: Repeated degraded allocation. Consider increasing nursery-size.

CumulusMX still appears to be running ok despite these error messages. Typically they start appearing 2-3 days after the application has been started.

My version information is as follows:

Raspberry Pi 2 running the Jessie release of Raspbian, patched with the latest updates as of about 1 week ago.
CumulusMX - B3034
Mono – 3.2.8

I was previously running B3033 and I also noticed the same message every now and then.

I did a quick search through the latest log file within the MXdiags folder, and I’m not seeing any log entries that match this.

In terms of CPU usage, it’s typically sitting at less than 5% with the occasional spike that ranges between 15 and 35% or thereabouts

Any thoughts on whether this is an error that I should be concerned about, or whether I need to tweak any configuration files would be appreciated.

Regards

Richard

Re: Degraded allocation warning messages

Posted: Fri 04 Dec 2015 10:10 am
by steve
It seems to be caused when large numbers of threads are being used, but I don't know why it happens to you.

http://stackoverflow.com/questions/1846 ... g-persists

I think someone else had the same message, and they were having problems with their SD card. Are you using an SD card for MX? SD card corruption is a common problem with the Pi.

Re: Degraded allocation warning messages

Posted: Fri 04 Dec 2015 9:15 pm
by senapsys
Hi Steve

I'm aware of the SD card corruption issue so from the very beginning I built this RPi using Berryboot so that I could boot the entire system from a USB connected SSD Drive.

As a precaution I did a "fsck" but everything came through clean and no issues were highlighted.

Regards

Richard

Re: Degraded allocation warning messages

Posted: Sat 05 Dec 2015 4:40 pm
by steve
You could have a look at the thread usage on your system, something is apparently using lots of threads, quite possibly MX. If it's MX we would need to work out why it's doing that on your system.

Re: Degraded allocation warning messages

Posted: Sat 05 Dec 2015 9:37 pm
by senapsys
Hi Steve

I've attached two files, one is a dump of all of the threads associated with the Mono process that is running CumulusMX after the Degraded Allocation messages start appearing. The other is a dump of the threads associated with the Mono process that is running CumulusMX after a restart of Mono. In the degraded state, Mono has 166 threads associated with it whereas just after Normal running appears, there are just 13.

Note that I'm not using Mono for anything else and a search through the process list has confirmed that there are no other instances of Mono running.

I'm reasonably familiar with Linux and there are a bunch of other stats and info I can pull out of the system. Let me know what else might be of use to you and I'll grab them for you.

Regards

Richard

Re: Degraded allocation warning messages

Posted: Sat 05 Dec 2015 10:03 pm
by steve
It does use threads for several things, but 166 seems like quite a lot. I wonder if you are having ftp problems, and the threads which handle ftp are hanging up? Is there any way to find out what those threads are actually doing? I suspect that you know far more about Linux than I do!

Re: Degraded allocation warning messages

Posted: Sat 05 Dec 2015 10:14 pm
by senapsys
Hi Steve

I'll do some digging and see what I can find. It might be a day or so before I post again as I have just restarted the application so I'm not seeing the accumulation of threads yet.

One thing I did notice is that there are a lot of threads that are sitting at roughly 6 minutes and 25 seconds. Does this roughly correlate with the scheduled timing any of internal process within CumulusMX?

Regards

Richard

Re: Degraded allocation warning messages

Posted: Sun 06 Dec 2015 8:50 am
by rogerthn
One of my favorites :P

Code: Select all

netstat -taupeen
It could return a lot but sort -kX and grep will help

Re: Degraded allocation warning messages

Posted: Sun 06 Dec 2015 9:12 am
by steve
senapsys wrote:a lot of threads that are sitting at roughly 6 minutes and 25 seconds. Does this roughly correlate with the scheduled timing any of internal process within CumulusMX?
Nothing comes to mind. I had a look on my Pi and MX has 13 threads, just as you do at startup, and mine has been running for about 18 hours.

Re: Degraded allocation warning messages

Posted: Tue 15 Dec 2015 3:17 am
by senapsys
Hi Steve

An update for you.

After my last post I deployed some scripts to monitor the number of threads as well as a bunch of other system performance data. As is the case with any fault finding, the moment I put these scripts in place the Mono/CumulusMX app got stage fright and the error didn't manifest itself after the usual 2-3 days :-) Like you, after the app displayed the "Normal running" message and I also observed 13 threads. Sometimes the number of threads increased up to about 17 threads for a brief moment and then returned to 13 or 15 threads. This "quiescent" level of thread variation appeared to be the normal behaviour that I observed over the days whilst waiting for the error to manifest itself again. I'm not sure if this is consistent with what you are seeing?

Anyway, after 9 days I finally started to notice something happening. I got lucky in that I was looking at the system as the thread count started to slowly increase. The increase in threads wasn't dramatic but rather happened at a much more pedestrian pace with an extra one being spawned every 5-10 minutes or thereabouts although this wasn't entirely consistent.

As this was happening I started to look at the various system stats and the one that stood out was an unusually high number of TCP sessions between the RPi system and the FTP server that hadn't been closed properly after a reasonable period of time. As an experiment I logged into the system which was running the FTP server and restarted the FTP service. As soon as I did this the slow increase in threads stopped and it went back to a scenario where the threads would briefly increase and then reduce, albeit with a higher base number of threads.

It's not a definitive answer and I'll need to put in place some more specific monitoring on both the RPi and the FTP server but the initial observations point to an issue with FTP. Whether the fault is with Mono/CumulusMX or the FTP server is impossible to say at this point. I'll do some more monitoring and see whether I'm on the right track.

Regards

Richard

Re: Degraded allocation warning messages

Posted: Tue 15 Dec 2015 8:20 am
by steve
I did suspect that it would turn out to be FTP connections not closing down. Turning on the ftp log may give some useful info from the Cumulus end - note that realtime ftp gets logged to the same file, so it can get large quickly.

Re: Degraded allocation warning messages

Posted: Fri 18 Dec 2015 5:07 am
by senapsys
Hi Steve

I turned on FTP Logging in CumulusMX and have attached a log file from the last 24 hours during which time the thread count started to increase. To my untrained eye the only thing that really stands out is a number of instances where the multiple consecutive messages of "Disposing FtpSocketStream..." appear in the file.

The server FTP log didn't reveal anything that stood out so I have increased the level of logging to see if I pick anything up next time.

I hope this is of use to you :-)

Regards

Richard

Re: Degraded allocation warning messages

Posted: Fri 18 Dec 2015 12:58 pm
by steve
I can't see anything there that explains the problem, I'm afraid.

Re: Degraded allocation warning messages

Posted: Fri 18 Dec 2015 8:09 pm
by senapsys
Hi Steve

Thanks for looking.

Noting my previous experience where restarting the FTP Server stopped Mono/CumulusMX from spawning additional threads it does point to the root cause being the FTP Server.

I'll put the microscope on the FTP Sever and see whether I can find out what the issue is and report back in due course.

Have a great Christmas/New Years :-)

Regards

Richard

Re: Degraded allocation warning messages

Posted: Wed 23 Dec 2015 10:46 am
by senapsys
Hi Steve

I've been looking over various logs and one thing out of the ordinary which I have noticed in the CumulusMX log in the MXdiags directory is a few messages along the lines of:

Error uploading realtime.txt to cudata/realtime.txt : Access is denied.

When I look at the FTP Log within CumulusMX for the corresponding time I can't seem to see anything that correlates with the time stamp of the access denied message so I'm not sure if this is significant or not. I thought I'd mention it in case it's useful.

Unfortunately it's a bit random as to when the issue manifests but as soon as it happens again I'll go hunting through the server FTP logs and see whether I can see anything at that end.

Regards

Richard