Cumulus Support

Pardon me if I'm in the wrong category. I'm a little stumped.

I'm running Cumulus MX v.3.0.0 build 3043 on a Pi and the first 3 tabs on the "local" site (port 8998) don't seem to be updating, i.e. Dashboad, Now, and Gauges. As far as I can see everything else is working correctly. I don't really follow the Javascript that supports updates to those pages. The external pages on my hosted server seemed to be OK.

I restarted CumulusMX, which is running as a systemctl service. That usually clears up issues. I've not rebooted yet, though I will next. I am curious though if anyone has experienced or understands the overall architecture weel enough to theorize why that small portion of functionality might not be working.

Thanks!
Bob

Hi,
Sorry I can't answer your question technically, but a common reply would be, try current version. A lot has changed over time since b3043. Again don't know what other technical issues you may encounter with your Pi setup but current version CumulusMX is great!

Enjoy,
Paul

Hi,

I second PaulMy's reply.

Version 3043 is the first release of the series we are in now and really a lot has happened since then. Because we have some information about the context you are running CMX in (your site), I would also remark that you run mono 3.2.8 where 6.12 is current at the moment. The runtime libraries have been upgraded because the development environment progresses and chance that some interaction between old and new versions is present.

So my advice would be to update your Linux (mono inclusive) preferably to the Buster release and then move to the lastest release of CumulusMX.
If you then still have a problem, start CMX with debuglogging and datalogging (through the menu if that works) and let it run for some time.
Post the logfiles here.

By that time, no doubt Mark will be back if really detailed look at things is required.

I have good confidence however that the updates will solve the problem (assuming there is no hardware issue) as this is not a situation I have heard of before.

Good luck.

Yeah guys, I started working on an update when we still messing around with oul versions. I think I may have even tested, but could not get it to work. It may be a good time if we're now using a current mono. I just need to find the time.

One thing that I was never got comfortable with was which files I need to copy to the new install. Seemed like the further one gets behind the harder it is to understand the accumulated file changes. I believe I've accumulated unneeded files from previous updates as well. Say, you guys aware of a fine offset station emmulator? It'd be really helpful to set up the newest version and the supporting stuff I've wrapped around CMX (like systemctl, etc.).

Thanks much!!
Bob

Oh ... a reboot of the Pi seemed to fix the issue. It often does, but this seemd like an odd issue in how little it affected. Thought that might be interesting somehow.

Thanks again!
Bob

One thing that I was never got comfortable with was which files I need to copy to the new install.

Start a total new install, copy in your existing\data folder and Records folder. Copy in your existing Cumulus.ini and you should be all set. Mark has included an \Updates.txt that has all the changes between versions all in one file. If you upload to MySQL than a bit more to it, and also there are added data in the monthly log.txt and dayfile.txt files but that should not be an issue as it will just be there from when you start the current version.

There have been several added uploads and other changes so you'd want to go through the settings after you have it running, and you don't need to stop and restart for most changes to take effect.

Enjoy,
Paul

PaulMy wrote: ↑Fri 18 Sep 2020 11:39 pm Start a total new install, copy in your existing\data folder and Records folder....

Enjoy,
Paul

Sort of funny I suppose. I was trying to clean up the filesystem on the Pi after having copied over the latest distro. Thought I was in a directory with a newer, but not newest distro that I had been running some diffs against. Then ...

rm -rf *

Turns out the bash tab-completion didn't work as expected ... and I cd'ed into the wrong directory (CumulusMX). I guess the priority of migrating to the new version just went up a tad!!!

Sadly, I also thought I had a nightly rsynch to back up data. Apparently it's be broken for a long time (like a year). Oh well.

Thanks for the pointers!! I never went to a DB so that makes life simpler (and looking at the package I see systemctl support was added). Sorta glad (so far) that I did something so incredibly stupid.

Ho, and whatever that issue was, it had come back ... another reason to get caught up!

Thanks again!!

Got the latest version up and running. Already seems much more stable. Still spawning quite a few mono threads, but otherwise resource utilization seems lower.

Forgotton how much I had changed (but sadly didn't have backup up). Embarrassing since all my Python code is in git (some watchdogs and other supporting scripts) - so I have to bake that in for CMX going forward.

I need to look at the provided systemctl stuff. Differs from what I had been using, particularly the "service" options for mono and CMX. Either way. that with journalctl seems way better than the old init.d.

Thanks folks!

radilly wrote: ↑Mon 28 Sep 2020 2:13 pm Got the latest version up and running. Already seems much more stable. Still spawning quite a few mono threads, but otherwise resource utilization seems lower.

Spawning those threads is by design. It is not a single threaded application. That in itself has nothing to do with resources.

And apart from that even simple mono applications do spawn multiple threads organised by mono (or maybe I am now a bit off track as I refer to a .NET Core application I made which has its own runtime without mono). Anyway, I think I am not far off

Hi @HansR

I have some familiarity with threading, though it's been many years since I wrote code using them. The little parallelism I employ today is at the process level.

HansR wrote: ↑Mon 28 Sep 2020 2:27 pm Spawning those threads is by design. It is not a single threaded application. That in itself has nothing to do with resources.

Understood. I noticed that lighttpd also starts a mono process with 8 threads, where CMX currently has 34 (after 8-9 days). There are 5 processes out of 114 which have more than 1 thread. Your assertion that multiple thread "has nothing to do with resources" is a little surprising. It has been my understanding that the Unix scheduler and dispatcher operate at a thread level to manage resources.

At any rate, my experience with build 3043 was that CMX would appear to stop, or miss readings because it seemed to get sluggish. I noted the load average (e.g. from uptime) would go up after some time, though anywhere between a day and several weeks to get into double digits. It seemed a CMX restart or a reboot was needed to clear it. That lead me to build up a "watchdog" that logged various metrics with the intent of triggering some corrective action when some metrics indicated a problem was likely. In investigating the loading, I found sometimes hundreds of mono threads running. I don't think I ever logged a decrease in that number, but alas, all that data is gone now. I can't say with any certainty that these threads caused the high loads, but I observed a strong correlation.

At some point I'll deploy the watchdog on the updated system because it was interesting and it did alert me to other issues too, like network or hosted server outages. I had read that folks would run CMX unattended for years, while for me something around a few months was as good as I seemed to get. I have a much better feeling about build 3094 so far!!

Bob

There's been a lot done to tidy up code which would spawn threads that then would remain idle for a long time - for example, the handling of FTP connections is much better now, especially for realtime data. It will be interesting to hear what your watchdog discovers when you deploy it.

radilly wrote: ↑Thu 01 Oct 2020 5:22 pm Your assertion that multiple thread "has nothing to do with resources" is a little surprising. It has been my understanding that the Unix scheduler and dispatcher operate at a thread level to manage resources.

Well, without going into details - first of all because I don't know all inner workings - the threading itself may be in cooperation with the OS but afaik it is threading and parallelism itself, which is done by C# (the programmer can say what and when in quite an easy way) and thus it is in the system libraries of C#. It has its own state machine calculator for the efficiency of async operations and parallel threads. Task is one step further and more course I guess but spawning tasks in threads asfaik is also done by the runtime system, no doubt in coop with the OS. But is not as straight forward as it was in the early days of Unix. I know CMX has some threading in it, but you'd have to ask Mark on details. The whole C# runtime/mono complex is quite a delicate balance as far as I can see. Pretty brilliant btw. And C# is.... well.... (I have seen some languages and have only seen maybe 40% of C#)

And I would not have the watchdog stop some threads just because you think they have not been doing enough or are afraid they stopped. It is more complex than that

HansR wrote: ↑Thu 01 Oct 2020 7:06 pm And I would not have the watchdog stop some threads just because you think they have not been doing enough or are afraid they stopped. It is more complex than that

That would be unwise!! No, the intent is just to restart CMX or reboot the Pi depending. Actually I just got the watchdog / logger running and the number of mono threads seems to be very stable around 24 +/- a little. That is way better than the old version I had been running. Could well have been an issue with the old mono required. The key to me is that threads are being destroyed / released. In the old version, the number only ever grew ... into the hundreds.

Both processes are now under systemctl and working well.

Bottom line: The current CMX is waaaaay better than what I had been running. Well worth updating!!

Bob

radilly wrote: ↑Tue 06 Oct 2020 6:43 pm Both processes are now under systemctl and working well.

Bottom line: The current CMX is waaaaay better than what I had been running. Well worth updating!!

Good