Welcome to the Cumulus Support forum.
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Mono/Cumulus process getting into a high CPU state and losing comms
Moderator: mcrossley
-
- Posts: 24
- Joined: Sun 22 Nov 2015 10:43 am
- Weather Station: Davis VP2 Plus
- Operating System: CentOS
- Location: Wolumla
- Contact:
Mono/Cumulus process getting into a high CPU state and losing comms
Hi there
We are experiencing a problem with CumulusMX with the Mono instance that is running Cumulus MX as a service. The problem started about 2-3 months ago and is random, but when it happens the Mono/CumulusMX process starts consuming all available CPU power and at the same time it loses comms with the weather station.
Some background info:
- Cumulus MX Version: 3.13.6—b3152 (configured to run as a service)
- Mono Version: 6.12.0.107
- OS: Centos 7 – Kernel 3.10.0-1160.42.2.e17.x86_64
- Weather Station: Davis Vantage Pro2 with a USB Logger [/list]
Looking at the various logs and configuration files…
- The USB interface is configured such that it does not go into sleep mode or anything like that.
- The operating system is not reporting any issues with the USB interface and is not reporting it as disconnecting.
- The timing of the increase in CPU usage happens shortly before the “No input from station” entries start appearing in the Service Console Log.
Other Observations
- The issue has been occurring over the last couple of builds B3148-b3152.
- When we normally stop the CumulusMX service, it usually stops within a second or two. When the Mono/CumulusMX process gets into the high CPU state, it will take around 30 seconds to stop the service.
- The CumulusMX service can be immediately restarted and it works perfectly again without having to do anything. It has no problems with communicating with the weather station after a restart.
- The time between the problem happening varies from a couple of hours to a couple of days.
- When the Mono/CumulusMX process gets into a high CPU state, it's memory usage doesn't appear to increase, even if left for 12 hours or more.
- A restart or the server doesn’t appear to make any difference.
As mentioned this problem started a few months ago. Prior to that CumulusMX was super stable on the same server and would run for months between restarts.
I'm happy to do some more in depth testing and post/send logs as required to help work out the issue.
We are experiencing a problem with CumulusMX with the Mono instance that is running Cumulus MX as a service. The problem started about 2-3 months ago and is random, but when it happens the Mono/CumulusMX process starts consuming all available CPU power and at the same time it loses comms with the weather station.
Some background info:
- Cumulus MX Version: 3.13.6—b3152 (configured to run as a service)
- Mono Version: 6.12.0.107
- OS: Centos 7 – Kernel 3.10.0-1160.42.2.e17.x86_64
- Weather Station: Davis Vantage Pro2 with a USB Logger [/list]
Looking at the various logs and configuration files…
- The USB interface is configured such that it does not go into sleep mode or anything like that.
- The operating system is not reporting any issues with the USB interface and is not reporting it as disconnecting.
- The timing of the increase in CPU usage happens shortly before the “No input from station” entries start appearing in the Service Console Log.
Other Observations
- The issue has been occurring over the last couple of builds B3148-b3152.
- When we normally stop the CumulusMX service, it usually stops within a second or two. When the Mono/CumulusMX process gets into the high CPU state, it will take around 30 seconds to stop the service.
- The CumulusMX service can be immediately restarted and it works perfectly again without having to do anything. It has no problems with communicating with the weather station after a restart.
- The time between the problem happening varies from a couple of hours to a couple of days.
- When the Mono/CumulusMX process gets into a high CPU state, it's memory usage doesn't appear to increase, even if left for 12 hours or more.
- A restart or the server doesn’t appear to make any difference.
As mentioned this problem started a few months ago. Prior to that CumulusMX was super stable on the same server and would run for months between restarts.
I'm happy to do some more in depth testing and post/send logs as required to help work out the issue.
-
- Posts: 704
- Joined: Thu 20 Aug 2020 8:35 am
- Weather Station: Davis VP2 Plus
- Operating System: RPiv4 64bit Linux 12 Bookworm
- Location: Jersey Channel Islands
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
As a first guess the mono version you are using is a rather old one. I would expect that some of the CMX development requires Mono JIT compiler version 6.12.0.122 (tarball Mon Feb 22 17:44:05 UTC 2021).
-
- Posts: 2433
- Joined: Wed 08 Jun 2011 11:19 am
- Weather Station: Davis Vantage Pro 2 + Ecowitt
- Operating System: GNU/Linux Ubuntu 22.04 LXC
- Location: Alcaston, Shropshire, UK
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
I think the mono version age may be a bit of a red herring, as I am running 6.8.0.105 without any issues.
Did you make any changes around the time the problem started? Have you tried reverting to the build before 3148 to see if that is still stable?
Did you make any changes around the time the problem started? Have you tried reverting to the build before 3148 to see if that is still stable?
-
- Posts: 24
- Joined: Sun 22 Nov 2015 10:43 am
- Weather Station: Davis VP2 Plus
- Operating System: CentOS
- Location: Wolumla
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
I've tried going back to b3122 (mainly because I had a backup for that version) and the problem persisted.
In terms of the mono version, when I originally installed mono, I imported the RPM keys so mono gets updated whenever I do an update via yum. I noticed on the mono web site that it shows a slightly later version and I'm guessing that the newer versions are only for CentOS 8 and there aren't any further updates for CentOS 7 as yum update isn't marking mono for an update despite the newer version being available?
In terms of changing anything, apart from the usual updates and patches to packages and the OS, nothing else has changed. Given this, the one thing I have been wondering about is whether I should try reverting back to an older version of mono (e.g. 6.10 or 6.8) as I do vaguely remember seeing an update for mono happening sometime in the last 3-6 months so that is one variable that has changed.
In terms of the mono version, when I originally installed mono, I imported the RPM keys so mono gets updated whenever I do an update via yum. I noticed on the mono web site that it shows a slightly later version and I'm guessing that the newer versions are only for CentOS 8 and there aren't any further updates for CentOS 7 as yum update isn't marking mono for an update despite the newer version being available?
In terms of changing anything, apart from the usual updates and patches to packages and the OS, nothing else has changed. Given this, the one thing I have been wondering about is whether I should try reverting back to an older version of mono (e.g. 6.10 or 6.8) as I do vaguely remember seeing an update for mono happening sometime in the last 3-6 months so that is one variable that has changed.
-
- Posts: 2433
- Joined: Wed 08 Jun 2011 11:19 am
- Weather Station: Davis Vantage Pro 2 + Ecowitt
- Operating System: GNU/Linux Ubuntu 22.04 LXC
- Location: Alcaston, Shropshire, UK
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
That is definitely worth a try.
-
- Posts: 24
- Joined: Sun 22 Nov 2015 10:43 am
- Weather Station: Davis VP2 Plus
- Operating System: CentOS
- Location: Wolumla
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
After resolving the usual dependency issues that come with rolling back to an older version of a package, I've now successfully installed version 6.8.0.123 of mono and didn't have any issues with starting CumulusMX as a service again.
I'll monitor how things go and report back as to whether this addresses the issue.
I'll monitor how things go and report back as to whether this addresses the issue.
- mcrossley
- Posts: 12686
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
Thanks, it would be good to have some feedback if a particular version of Mono is causing problems.
-
- Posts: 24
- Joined: Sun 22 Nov 2015 10:43 am
- Weather Station: Davis VP2 Plus
- Operating System: CentOS
- Location: Wolumla
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
A quick update to close the loop on this discussion thread.
It's now been a week since I rolled Mono back to version 6.8.0.123 and since then I’ve deliberately not made any other changes to the server and not done any updates or patches.
After the roll back, CumulusMX has been super stable and I've not experienced any of the problems described previously in this discussion thread.
In my scenario where I'm running Centos 7 as the Operating System, it would appear that Mono version 6.12.0.107 results in the CumulusMX application becoming unstable and failing.
It's now been a week since I rolled Mono back to version 6.8.0.123 and since then I’ve deliberately not made any other changes to the server and not done any updates or patches.
After the roll back, CumulusMX has been super stable and I've not experienced any of the problems described previously in this discussion thread.
In my scenario where I'm running Centos 7 as the Operating System, it would appear that Mono version 6.12.0.107 results in the CumulusMX application becoming unstable and failing.
-
- Posts: 265
- Joined: Tue 10 Sep 2013 8:40 pm
- Weather Station: Ecowitt GW1003 (GW1000)
- Operating System: Raspbian 10 (Buster) / Mono 6.12
- Location: Freiston, Lincolnshire, UK
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
I experienced a similar issue some time ago and was running a very old mono version at the time (3.x.x) or something. I am now running 6.12.0.107 at the moment and have a script that runs at 04:00 every day to restart Cumulus as the mono CPU usage climbs over time (I monitored using "top" command from command prompt of Pi) and eventually would lock out and stop Cumulus. Retarting Cumulus seems to free the CPU and start again.
Currently mono is running at between 7.3% and 23.8%. I will restart CMX and stop the script and monitor over next 24-48 hours and see if it starts to climb again.
EDIT: I just monitored "top" when restarting Cumulus and CPU% goes up to 148% and then settles down to 7.3%
Andy
Currently mono is running at between 7.3% and 23.8%. I will restart CMX and stop the script and monitor over next 24-48 hours and see if it starts to climb again.
EDIT: I just monitored "top" when restarting Cumulus and CPU% goes up to 148% and then settles down to 7.3%
Andy
-
- Posts: 265
- Joined: Tue 10 Sep 2013 8:40 pm
- Weather Station: Ecowitt GW1003 (GW1000)
- Operating System: Raspbian 10 (Buster) / Mono 6.12
- Location: Freiston, Lincolnshire, UK
- Contact:
Re: Mono/Cumulus process getting into a high CPU state and losing comms
I stopped the script on Sunday I havent seen any issues. CMX has been running perfectly. Mono process in "top" seems to be fairly constant between 7% and 23% so I assume this increase is when CMX is processing updates / log files etc.
Will keep monitoring
Will keep monitoring