Page 1 of 1
cumulusmx service crashing after 5 sec
Posted: Sat 22 May 2021 6:53 pm
by inimeg
Hi,
I have a strange problem regarding cumulusmx service stopping just 5 sec after autostart after boot or restart by hand.
About my setup: I run this on RPi zero W with the RPi Cumulus Image. I tried both the latest v3.11.3 build 3132 and v3.10.2 - same result. This is a clean install, nothing connected - just bare RPi running. Used different SD cards, to rule out the SD card problem - no change.
When the service crashes I get this when running
systemctl status cumulusmx.service:
Code: Select all
cumulusmx.service - CumulusMX service
Loaded: loaded (/etc/systemd/system/cumulusmx.service; enabled; vendor preset
Active: failed (Result: signal) since Sat 2021-05-22 20:34:23 CEST; 2s ago
Docs: https://cumuluswiki.org/a/Main_Page
Process: 529 ExecStart=/usr/bin/mono-service -d:/opt/CumulusMX CumulusMX.exe -
Process: 543 ExecStopPost=/bin/rm /tmp/CumulusMX.exe.lock (code=exited, status
Main PID: 530 (code=killed, signal=ABRT)
I also found some strange json files in root directory named
mono_crash.15b1c48ca.0.json and the number at the end increases with the next file (
mono_crash.15b1c48ca.1.json and so on).
This is what is inside:
Code: Select all
"protocol_version" : "0.0.6",
"configuration" : {
"version" : "(6.12.0.122) (tarball)",
"tlc" : "__thread",
"sigsgev" : "normal",
"notifications" : "epoll",
"architecture" : "armel,vfp+hard",
"disabled_features" : "none",
"smallconfig" : "disabled",
"bigarrays" : "disabled",
"softdebug" : "enabled",
"interpreter" : "enabled",
"llvm_support" : "610",
"suspend" : "preemptive"
},
"memory" : {
"minor_gc_time" : "0",
"major_gc_time" : "0",
"minor_gc_count" : "0",
"major_gc_count" : "0",
"major_gc_time_concurrent" : "0"
},
"threads" : [
{
"is_managed" : true,
"offset_free_hash" : "0x15b1c48ca",
"offset_rich_hash" : "0x15b1c4952",
"crashed" : true,
"native_thread_id" : "0xffffffffb6fbd910",
"thread_info_addr" : "0x1c604b8",
"thread_name" : "mono",
"ctx" : {
"IP" : "0xb4513338",
"SP" : "0xbee271d8",
"BP" : "0xbee271f8"
},
"managed_frames" : [
{
"is_managed" : "true",
"guid" : "533173D2-4DAE-4608-99D2-B10975534BB0",
"token" : "0x00000",
"native_offset" : "0x0",
"filename" : "mscorlib.dll",
"sizeofimage" : "0x472000",
"timestamp" : "0x99dbfea7",
"il_offset" : "0x00027"
}
,
{
"is_managed" : "false",
"native_address" : "unregistered"
}
,
{
"is_managed" : "true",
"guid" : "533173D2-4DAE-4608-99D2-B10975534BB0",
"token" : "0x00000",
"native_offset" : "0x0",
"filename" : "mscorlib.dll",
"sizeofimage" : "0x472000",
"timestamp" : "0x99dbfea7",
"il_offset" : "0x00061"
}
]
}
]
}
The cumulus was runing fine for about 3 days, and then suddenly the service stopped. After restart I got the above mentioned results. Clean install changed nothing.
Can you please help me?
Re: cumulusmc service crashing after 5 sec
Posted: Sat 22 May 2021 8:02 pm
by HansR
Hi, this is a mono issue. I see you're running 6.x I would suggest going back one version (5.x) which afaik is the standard version which goes with buster.
Re: cumulusmc service crashing after 5 sec
Posted: Sat 22 May 2021 8:22 pm
by sutne
I am running Raspberry Buster and Mono version 6.12.0122.
It should not be a problem.
Re: cumulusmx service crashing after 5 sec
Posted: Sat 22 May 2021 9:16 pm
by sfws
sutne wrote: ↑Sat 22 May 2021 8:22 pm
I am running Raspberry Buster and Mono version 6.12.0122.
I too have been using Raspberry Buster and Mono version 6.12.0122 (for a long time now for MX without issue). So I suspect inimeg has some required component of Mono missing.
By all means, inimeg can try to revert to earlier version, although I don't believe that is problem.
inimeg wrote: ↑Sat 22 May 2021 6:53 pm
I have a strange problem regarding cumulusmx service stopping just 5 sec after autostart after boot or restart by hand.
Can you confirm that you installed using "sudo apt install mono-complete" (try that again), because I am wondering if your mono has some components missing? (Maybe you used "sudo apt install mono", which is wrong).
You could also try "sudo apt upgrade mono-complete".
Re: cumulusmx service crashing after 5 sec
Posted: Sat 22 May 2021 9:34 pm
by inimeg
Can you confirm that you installed using "sudo apt install mono-complete"
No I did not install mono separately.
As I said in my first post I'm using the RPi Cumulus Image, lastest build available. This has preinstalled all required components. Nothing to add later. It should work out of the box (and it did for around 3 days).
Re: cumulusmx service crashing after 5 sec
Posted: Sat 22 May 2021 11:13 pm
by mcrossley
Have you tried just running it from the command line?
Are any MXdiags files created? Please post them if so.
Re: cumulusmx service crashing after 5 sec
Posted: Sun 23 May 2021 4:27 am
by inimeg
Yes, I did. No change, it keeps crashing. No MXdiags files are created.
I'm starting to think, that this could be a hardware related issue. Don't have another RPi zero to verify, though.
edit:
This is how it "starts" from command line:
Code: Select all
=================================================================
Native Crash Reporting
=================================================================
Got a UNKNOWN while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
(No frames)
=================================================================
Telemetry Dumper:
=================================================================
Could not exec mono-hang-watchdog, expected on path '/etc/../bin/mono-hang-watchdog' (errno 2)
Entering thread summarizer pause from 0x3069331728x
Finished thread summarizer pause from 0x3069331728x.
Failed to create breadcrumb file (null)/crash_hash_0x15b1c4952
Waiting for dumping threads to resume
=================================================================
External Debugger Dump:
=================================================================
[New LWP 617]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
__waitpid (options=0, stat_loc=0xbe9a3054, pid=619) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30 ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
Id Target Id Frame
* 1 Thread 0xb6f24910 (LWP 616) "mono" __waitpid (options=0, stat_loc=0xbe9a3054, pid=619) at ../sysdeps/unix/sysv/linux/waitpid.c:30
2 Thread 0xb5fff430 (LWP 617) "SGen worker" futex_wait_cancelable (private=0, expected=0, futex_word=0x40d290) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
Thread 2 (Thread 0xb5fff430 (LWP 617)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x40d290) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0x40d268) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x40d268, mutex=0x0) at pthread_cond_wait.c:655
#3 0x002f495c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 1 (Thread 0xb6f24910 (LWP 616)):
#0 __waitpid (options=0, stat_loc=0xbe9a3054, pid=619) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1 __waitpid (pid=619, stat_loc=0xbe9a3054, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:25
#2 0x0009cbc4 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
[Inferior 1 (process 616) detached]
=================================================================
Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0xb4513338):0xb4513328 00 00 00 ea fc d7 66 00 00 00 9f e7 10 00 8d e5 ......f.........
0xb4513338 5b f0 7f f5 10 00 9d e5 00 10 80 e5 18 d0 8d e2 [...............
0xb4513348 00 81 bd e8 40 41 2d e9 1c d0 4d e2 00 60 a0 e1 ....@A-...M..`..
0xb4513358 10 10 8d e5 00 00 56 e3 0d 00 00 0b 08 10 86 e2 ......V.........
=================================================================
Managed Stacktrace:
=================================================================
at System.Exception:.cctor <0x00038>
at System.Object:runtime_invoke_void <0x0007f>
at <unknown> <0xffffffff>
at System.SystemException:.ctor <0x0001b>
at System.OutOfMemoryException:.ctor <0x0001b>
at <Module>:runtime_invoke_void__this___object <0x0010b>
=================================================================
Aborted
Re: cumulusmx service crashing after 5 sec
Posted: Sun 23 May 2021 11:49 am
by mcrossley
BTW, I don't really recommend a Pi Zero, it is marginal on power to run Cumulus MX. It just about manages if you don't ask it to do too much, but really you would be better off on a 3 Model B or B+.
Re: cumulusmx service crashing after 5 sec
Posted: Sun 23 May 2021 12:59 pm
by HansR
I would seriously consider to try mono 5.x if you have the time to install it iso 6.x
Especially with a zero where everything is already on the limits.
Re: cumulusmx service crashing after 5 sec
Posted: Wed 26 May 2021 2:01 pm
by inimeg
Just a small update.
After switching to RPi3b+ cumulus started without any issues. No change to hardware or software other than the RPi.
Strange, but it works now so I'm happy. Thanks!
Re: cumulusmx service crashing after 5 sec
Posted: Wed 26 May 2021 4:12 pm
by mcrossley
Thinking a bit more about this, the pi zero has a different processor type from the 3/4. IIRC Mono compiles some components during install, as the image was created on a pi 3, the Mono compilation may be wrong for the zero.
So to use the pre-built image on a zero you would have to remove and reinstall the mono-complete package. I think!
Re: cumulusmx service crashing after 5 sec
Posted: Wed 26 May 2021 4:56 pm
by inimeg
mcrossley wrote: ↑Wed 26 May 2021 4:12 pm
So to use the pre-built image on a zero you would have to remove and reinstall the mono-complete package. I think!
I did that and no success.
The pre-built image is capable of running on RPi zero. As I mentioned in my first post I did my very first install like this and it worked for around 3 days.