Cumulus Support

Hi there

We are experiencing a problem with CumulusMX with the Mono instance that is running Cumulus MX as a service. The problem started about 2-3 months ago and is random, but when it happens the Mono/CumulusMX process starts consuming all available CPU power and at the same time it loses comms with the weather station.

Some background info:
- Cumulus MX Version: 3.13.6—b3152 (configured to run as a service)
- Mono Version: 6.12.0.107
- OS: Centos 7 – Kernel 3.10.0-1160.42.2.e17.x86_64
- Weather Station: Davis Vantage Pro2 with a USB Logger [/list]

Looking at the various logs and configuration files…
- The USB interface is configured such that it does not go into sleep mode or anything like that.
- The operating system is not reporting any issues with the USB interface and is not reporting it as disconnecting.
- The timing of the increase in CPU usage happens shortly before the “No input from station” entries start appearing in the Service Console Log.

Other Observations
- The issue has been occurring over the last couple of builds B3148-b3152.
- When we normally stop the CumulusMX service, it usually stops within a second or two. When the Mono/CumulusMX process gets into the high CPU state, it will take around 30 seconds to stop the service.
- The CumulusMX service can be immediately restarted and it works perfectly again without having to do anything. It has no problems with communicating with the weather station after a restart.
- The time between the problem happening varies from a couple of hours to a couple of days.
- When the Mono/CumulusMX process gets into a high CPU state, it's memory usage doesn't appear to increase, even if left for 12 hours or more.
- A restart or the server doesn’t appear to make any difference.

As mentioned this problem started a few months ago. Prior to that CumulusMX was super stable on the same server and would run for months between restarts.

I'm happy to do some more in depth testing and post/send logs as required to help work out the issue.

As a first guess the mono version you are using is a rather old one. I would expect that some of the CMX development requires Mono JIT compiler version 6.12.0.122 (tarball Mon Feb 22 17:44:05 UTC 2021).

I think the mono version age may be a bit of a red herring, as I am running 6.8.0.105 without any issues.
Did you make any changes around the time the problem started? Have you tried reverting to the build before 3148 to see if that is still stable?

I've tried going back to b3122 (mainly because I had a backup for that version) and the problem persisted.

In terms of the mono version, when I originally installed mono, I imported the RPM keys so mono gets updated whenever I do an update via yum. I noticed on the mono web site that it shows a slightly later version and I'm guessing that the newer versions are only for CentOS 8 and there aren't any further updates for CentOS 7 as yum update isn't marking mono for an update despite the newer version being available?

In terms of changing anything, apart from the usual updates and patches to packages and the OS, nothing else has changed. Given this, the one thing I have been wondering about is whether I should try reverting back to an older version of mono (e.g. 6.10 or 6.8) as I do vaguely remember seeing an update for mono happening sometime in the last 3-6 months so that is one variable that has changed.

senapsys wrote: ↑Tue 09 Nov 2021 10:54 am I have been wondering about is whether I should try reverting back to an older version of mono (e.g. 6.10 or 6.8) as I do vaguely remember seeing an update for mono happening sometime in the last 3-6 months so that is one variable that has changed.

That is definitely worth a try.

After resolving the usual dependency issues that come with rolling back to an older version of a package, I've now successfully installed version 6.8.0.123 of mono and didn't have any issues with starting CumulusMX as a service again.

I'll monitor how things go and report back as to whether this addresses the issue.

Thanks, it would be good to have some feedback if a particular version of Mono is causing problems.

A quick update to close the loop on this discussion thread.

It's now been a week since I rolled Mono back to version 6.8.0.123 and since then I’ve deliberately not made any other changes to the server and not done any updates or patches.

After the roll back, CumulusMX has been super stable and I've not experienced any of the problems described previously in this discussion thread.

In my scenario where I'm running Centos 7 as the Operating System, it would appear that Mono version 6.12.0.107 results in the CumulusMX application becoming unstable and failing.

I experienced a similar issue some time ago and was running a very old mono version at the time (3.x.x) or something. I am now running 6.12.0.107 at the moment and have a script that runs at 04:00 every day to restart Cumulus as the mono CPU usage climbs over time (I monitored using "top" command from command prompt of Pi) and eventually would lock out and stop Cumulus. Retarting Cumulus seems to free the CPU and start again.

Currently mono is running at between 7.3% and 23.8%. I will restart CMX and stop the script and monitor over next 24-48 hours and see if it starts to climb again.

EDIT: I just monitored "top" when restarting Cumulus and CPU% goes up to 148% and then settles down to 7.3%

Andy

I stopped the script on Sunday I havent seen any issues. CMX has been running perfectly. Mono process in "top" seems to be fairly constant between 7% and 23% so I assume this increase is when CMX is processing updates / log files etc.

Will keep monitoring