CumulusMX running out of memory on Debian

[ahagadorn]

My Cumulus installation on Debian 11.8 is consistently dying after only a few hours. It has been running flawlessly for years up until a few weeks ago. Both Cumulus and Debian are up to date. Mono is the latest version. I turned off MySQL inserts. The logs state that it is running out of memory. Im stumped at this point and need assistance. MXdiags zip is attached.

Thanks!!
ANdrew

[HansR]

As in your other thread, the SD card might be worn out.

If not try increasing your swap disk like described in this procedure.

[mcrossley]

Can you take a screen shot of a htop command?

That will show how much memory each process is using.

[ahagadorn]

System memory and swap are fine. It is the process that is running out of memory. Nothing on the server is affected besides Cumulus.

[mcrossley]

And how much memory is MX consuming?

[ahagadorn]

See the attached screenshot

[ahagadorn]

And how much memory is MX consuming?

I'm watching it with top -p <PID> and memory is constantly creeping up.

Code: Select all

 1018 root      20   0 2068228   1.9g  29808 S   6.0  49.3   1:33.14 mono                                                                                                       

Sure looks like a memory leak to me...

[freddie]

I agree, but the leak is likely in mono, not in MX code. Also, it is not a common issue as there are many Linux users (I'm one) who don't see this behaviour. It's likely distro-specific.

[ahagadorn]

I agree, but the leak is likely in mono, not in MX code. Also, it is not a common issue as there are many Linux users (I'm one) who don't see this behaviour. It's likely distro-specific.

I agree. I realize that it is not common too. Here is the latest top:

Code: Select all

 1018 root      20   0 2124036   1.9g  29616 S   6.3  50.7   2:44.53 mono                                                                                                          

I don't know what to do about it at this point.

[mcrossley]

Wow, that is a serious memory leak! Just out of curiosity, does it still leak if you run MX from the command line rather than as a service (I expect it will).

It could be worth trying a slightly older version of Mono?

[ahagadorn]

Yes, same running from command line.

Working on older versions of mono.

[jlmr731]

Did you install mono from mono's site? Im running debian 12 (12.4) and debians package for mono is 6.8.0.105 and works fine.
so try to install it from deb's repository.

[ahagadorn]

Did you install mono from mono's site? Im running debian 12 (12.4) and debians package for mono is 6.8.0.105 and works fine.
so try to install it from deb's repository.

I did try 6.8.0 and the issue persisted so maybe it isn't mono.

I do have a Davis AirLink which might be uncommon...

I just now upgraded to the most recent MX release. See how it goes.

[ahagadorn]

Still no joy, watching it bleed memory with top right now, back to the latest version of everything. I guess the next step is a compete reinstall of Cumulus with a fresh setup. Its very unfortunate because I have come to depend on my Cumulus generated display.

Did anyone analyze the MXdiags dump I posted?

[freddie]

The only thing that stood out to me in your MXDiags was this:

Code: Select all

2023-12-11 18:52:56.669 Some values in Cumulus.ini had invalid values, or new required entries have been created.

This is a fair enough message to see if you had just upgraded to a new release (when the new required entries would be created), but I wouldn't anticipate seeing it on every restart like you appear to be seeing. Might be worth looking into this before anything else.

@mcrossley does that sound reasonable?