Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Mono/Cumulus process getting into a high CPU state and losing comms

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Mono/Cumulus process getting into a high CPU state and losing comms

Post by senapsys »

Hi there

We are experiencing a problem with CumulusMX with the Mono instance that is running Cumulus MX as a service. The problem started about 2-3 months ago and is random, but when it happens the Mono/CumulusMX process starts consuming all available CPU power and at the same time it loses comms with the weather station.

Some background info:
- Cumulus MX Version: 3.13.6—b3152 (configured to run as a service)
- Mono Version: 6.12.0.107
- OS: Centos 7 – Kernel 3.10.0-1160.42.2.e17.x86_64
- Weather Station: Davis Vantage Pro2 with a USB Logger [/list]

Looking at the various logs and configuration files…
- The USB interface is configured such that it does not go into sleep mode or anything like that.
- The operating system is not reporting any issues with the USB interface and is not reporting it as disconnecting.
- The timing of the increase in CPU usage happens shortly before the “No input from station” entries start appearing in the Service Console Log.

Other Observations
- The issue has been occurring over the last couple of builds B3148-b3152.
- When we normally stop the CumulusMX service, it usually stops within a second or two. When the Mono/CumulusMX process gets into the high CPU state, it will take around 30 seconds to stop the service.
- The CumulusMX service can be immediately restarted and it works perfectly again without having to do anything. It has no problems with communicating with the weather station after a restart.
- The time between the problem happening varies from a couple of hours to a couple of days.
- When the Mono/CumulusMX process gets into a high CPU state, it's memory usage doesn't appear to increase, even if left for 12 hours or more.
- A restart or the server doesn’t appear to make any difference.

As mentioned this problem started a few months ago. Prior to that CumulusMX was super stable on the same server and would run for months between restarts.

I'm happy to do some more in depth testing and post/send logs as required to help work out the issue.
AndyKF650
Posts: 704
Joined: Thu 20 Aug 2020 8:35 am
Weather Station: Davis VP2 Plus
Operating System: RPiv4 64bit Linux 12 Bookworm
Location: Jersey Channel Islands
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by AndyKF650 »

As a first guess the mono version you are using is a rather old one. I would expect that some of the CMX development requires Mono JIT compiler version 6.12.0.122 (tarball Mon Feb 22 17:44:05 UTC 2021).
freddie
Posts: 2433
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by freddie »

I think the mono version age may be a bit of a red herring, as I am running 6.8.0.105 without any issues.
Did you make any changes around the time the problem started? Have you tried reverting to the build before 3148 to see if that is still stable?
Freddie
Image
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by senapsys »

I've tried going back to b3122 (mainly because I had a backup for that version) and the problem persisted.

In terms of the mono version, when I originally installed mono, I imported the RPM keys so mono gets updated whenever I do an update via yum. I noticed on the mono web site that it shows a slightly later version and I'm guessing that the newer versions are only for CentOS 8 and there aren't any further updates for CentOS 7 as yum update isn't marking mono for an update despite the newer version being available?

In terms of changing anything, apart from the usual updates and patches to packages and the OS, nothing else has changed. Given this, the one thing I have been wondering about is whether I should try reverting back to an older version of mono (e.g. 6.10 or 6.8) as I do vaguely remember seeing an update for mono happening sometime in the last 3-6 months so that is one variable that has changed.
freddie
Posts: 2433
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by freddie »

senapsys wrote: Tue 09 Nov 2021 10:54 am I have been wondering about is whether I should try reverting back to an older version of mono (e.g. 6.10 or 6.8) as I do vaguely remember seeing an update for mono happening sometime in the last 3-6 months so that is one variable that has changed.
That is definitely worth a try.
Freddie
Image
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by senapsys »

After resolving the usual dependency issues that come with rolling back to an older version of a package, I've now successfully installed version 6.8.0.123 of mono and didn't have any issues with starting CumulusMX as a service again.

I'll monitor how things go and report back as to whether this addresses the issue.
User avatar
mcrossley
Posts: 12686
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by mcrossley »

Thanks, it would be good to have some feedback if a particular version of Mono is causing problems.
senapsys
Posts: 24
Joined: Sun 22 Nov 2015 10:43 am
Weather Station: Davis VP2 Plus
Operating System: CentOS
Location: Wolumla
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by senapsys »

A quick update to close the loop on this discussion thread.

It's now been a week since I rolled Mono back to version 6.8.0.123 and since then I’ve deliberately not made any other changes to the server and not done any updates or patches.

After the roll back, CumulusMX has been super stable and I've not experienced any of the problems described previously in this discussion thread.

In my scenario where I'm running Centos 7 as the Operating System, it would appear that Mono version 6.12.0.107 results in the CumulusMX application becoming unstable and failing.
Big Daddy
Posts: 265
Joined: Tue 10 Sep 2013 8:40 pm
Weather Station: Ecowitt GW1003 (GW1000)
Operating System: Raspbian 10 (Buster) / Mono 6.12
Location: Freiston, Lincolnshire, UK
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by Big Daddy »

I experienced a similar issue some time ago and was running a very old mono version at the time (3.x.x) or something. I am now running 6.12.0.107 at the moment and have a script that runs at 04:00 every day to restart Cumulus as the mono CPU usage climbs over time (I monitored using "top" command from command prompt of Pi) and eventually would lock out and stop Cumulus. Retarting Cumulus seems to free the CPU and start again.

Currently mono is running at between 7.3% and 23.8%. I will restart CMX and stop the script and monitor over next 24-48 hours and see if it starts to climb again.

EDIT: I just monitored "top" when restarting Cumulus and CPU% goes up to 148% and then settles down to 7.3%

Andy
Big Daddy
Posts: 265
Joined: Tue 10 Sep 2013 8:40 pm
Weather Station: Ecowitt GW1003 (GW1000)
Operating System: Raspbian 10 (Buster) / Mono 6.12
Location: Freiston, Lincolnshire, UK
Contact:

Re: Mono/Cumulus process getting into a high CPU state and losing comms

Post by Big Daddy »

I stopped the script on Sunday I havent seen any issues. CMX has been running perfectly. Mono process in "top" seems to be fairly constant between 7% and 23% so I assume this increase is when CMX is processing updates / log files etc.

Will keep monitoring
Post Reply