cumulusmx service crashing after 5 sec

[inimeg]

Hi,

I have a strange problem regarding cumulusmx service stopping just 5 sec after autostart after boot or restart by hand.
About my setup: I run this on RPi zero W with the RPi Cumulus Image. I tried both the latest v3.11.3 build 3132 and v3.10.2 - same result. This is a clean install, nothing connected - just bare RPi running. Used different SD cards, to rule out the SD card problem - no change.
When the service crashes I get this when running systemctl status cumulusmx.service:

Code: Select all

cumulusmx.service - CumulusMX service
   Loaded: loaded (/etc/systemd/system/cumulusmx.service; enabled; vendor preset
   Active: failed (Result: signal) since Sat 2021-05-22 20:34:23 CEST; 2s ago
     Docs: https://cumuluswiki.org/a/Main_Page
  Process: 529 ExecStart=/usr/bin/mono-service -d:/opt/CumulusMX CumulusMX.exe -
  Process: 543 ExecStopPost=/bin/rm /tmp/CumulusMX.exe.lock (code=exited, status
 Main PID: 530 (code=killed, signal=ABRT)

I also found some strange json files in root directory named mono_crash.15b1c48ca.0.json and the number at the end increases with the next file (mono_crash.15b1c48ca.1.json and so on).
This is what is inside:

Code: Select all

"protocol_version" : "0.0.6",
  "configuration" : {
    "version" : "(6.12.0.122) (tarball)",
    "tlc" : "__thread",
    "sigsgev" : "normal",
    "notifications" : "epoll",
    "architecture" : "armel,vfp+hard",
    "disabled_features" : "none",
    "smallconfig" : "disabled",
    "bigarrays" : "disabled",
    "softdebug" : "enabled",
    "interpreter" : "enabled",
    "llvm_support" : "610",
    "suspend" : "preemptive"
  },
  "memory" : {
    "minor_gc_time" : "0",
    "major_gc_time" : "0",
    "minor_gc_count" : "0",
    "major_gc_count" : "0",
    "major_gc_time_concurrent" : "0"
 },
  "threads" : [
 {
    "is_managed" : true,
    "offset_free_hash" : "0x15b1c48ca",
    "offset_rich_hash" : "0x15b1c4952",
    "crashed" : true,
    "native_thread_id" : "0xffffffffb6fbd910",
    "thread_info_addr" : "0x1c604b8",
    "thread_name" : "mono",
    "ctx" : {
      "IP" : "0xb4513338",
      "SP" : "0xbee271d8",
      "BP" : "0xbee271f8"
  },
    "managed_frames" : [
  {
      "is_managed" : "true",
      "guid" : "533173D2-4DAE-4608-99D2-B10975534BB0",
      "token" : "0x00000",
      "native_offset" : "0x0",
      "filename" : "mscorlib.dll",
      "sizeofimage" : "0x472000",
      "timestamp" : "0x99dbfea7",
      "il_offset" : "0x00027"
   }
,
  {
      "is_managed" : "false",
      "native_address" : "unregistered"
   }
,
  {
      "is_managed" : "true",
      "guid" : "533173D2-4DAE-4608-99D2-B10975534BB0",
      "token" : "0x00000",
      "native_offset" : "0x0",
      "filename" : "mscorlib.dll",
      "sizeofimage" : "0x472000",
      "timestamp" : "0x99dbfea7",
      "il_offset" : "0x00061"
   }

  ]
 }
 ]
}

The cumulus was runing fine for about 3 days, and then suddenly the service stopped. After restart I got the above mentioned results. Clean install changed nothing.
Can you please help me?

[HansR]

Hi, this is a mono issue. I see you're running 6.x I would suggest going back one version (5.x) which afaik is the standard version which goes with buster.

[sutne]

I am running Raspberry Buster and Mono version 6.12.0122.

It should not be a problem.

[sfws]

I am running Raspberry Buster and Mono version 6.12.0122.

I too have been using Raspberry Buster and Mono version 6.12.0122 (for a long time now for MX without issue). So I suspect inimeg has some required component of Mono missing.
By all means, inimeg can try to revert to earlier version, although I don't believe that is problem.

I have a strange problem regarding cumulusmx service stopping just 5 sec after autostart after boot or restart by hand.

Can you confirm that you installed using "sudo apt install mono-complete" (try that again), because I am wondering if your mono has some components missing? (Maybe you used "sudo apt install mono", which is wrong).
You could also try "sudo apt upgrade mono-complete".

[inimeg]

Can you confirm that you installed using "sudo apt install mono-complete"

No I did not install mono separately.
As I said in my first post I'm using the RPi Cumulus Image, lastest build available. This has preinstalled all required components. Nothing to add later. It should work out of the box (and it did for around 3 days).

[mcrossley]

Have you tried just running it from the command line?
Are any MXdiags files created? Please post them if so.

[inimeg]

Yes, I did. No change, it keeps crashing. No MXdiags files are created.
I'm starting to think, that this could be a hardware related issue. Don't have another RPi zero to verify, though.

edit:
This is how it "starts" from command line:

Code: Select all

=================================================================
        Native Crash Reporting
=================================================================
Got a UNKNOWN while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

=================================================================
        Native stacktrace:
=================================================================
         (No frames)


=================================================================
        Telemetry Dumper:
=================================================================
Could not exec mono-hang-watchdog, expected on path '/etc/../bin/mono-hang-watchdog' (errno 2)
Entering thread summarizer pause from 0x3069331728x
Finished thread summarizer pause from 0x3069331728x.
Failed to create breadcrumb file (null)/crash_hash_0x15b1c4952

Waiting for dumping threads to resume

=================================================================
        External Debugger Dump:
=================================================================
[New LWP 617]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
__waitpid (options=0, stat_loc=0xbe9a3054, pid=619) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30      ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id                                 Frame
* 1    Thread 0xb6f24910 (LWP 616) "mono"        __waitpid (options=0, stat_loc=0xbe9a3054, pid=619) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  2    Thread 0xb5fff430 (LWP 617) "SGen worker" futex_wait_cancelable (private=0, expected=0, futex_word=0x40d290) at ../sysdeps/unix/sysv/linux/futex-internal.h:88

Thread 2 (Thread 0xb5fff430 (LWP 617)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x40d290) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x0, cond=0x40d268) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x40d268, mutex=0x0) at pthread_cond_wait.c:655
#3  0x002f495c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xb6f24910 (LWP 616)):
#0  __waitpid (options=0, stat_loc=0xbe9a3054, pid=619) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  __waitpid (pid=619, stat_loc=0xbe9a3054, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:25
#2  0x0009cbc4 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
[Inferior 1 (process 616) detached]

=================================================================
        Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0xb4513338):0xb4513328  00 00 00 ea fc d7 66 00 00 00 9f e7 10 00 8d e5  ......f.........
0xb4513338  5b f0 7f f5 10 00 9d e5 00 10 80 e5 18 d0 8d e2  [...............
0xb4513348  00 81 bd e8 40 41 2d e9 1c d0 4d e2 00 60 a0 e1  ....@A-...M..`..
0xb4513358  10 10 8d e5 00 00 56 e3 0d 00 00 0b 08 10 86 e2  ......V.........

=================================================================
        Managed Stacktrace:
=================================================================
          at System.Exception:.cctor <0x00038>
          at System.Object:runtime_invoke_void <0x0007f>
          at <unknown> <0xffffffff>
          at System.SystemException:.ctor <0x0001b>
          at System.OutOfMemoryException:.ctor <0x0001b>
          at <Module>:runtime_invoke_void__this___object <0x0010b>
=================================================================
Aborted

[mcrossley]

BTW, I don't really recommend a Pi Zero, it is marginal on power to run Cumulus MX. It just about manages if you don't ask it to do too much, but really you would be better off on a 3 Model B or B+.

[HansR]

I would seriously consider to try mono 5.x if you have the time to install it iso 6.x
Especially with a zero where everything is already on the limits.

[inimeg]

Just a small update.
After switching to RPi3b+ cumulus started without any issues. No change to hardware or software other than the RPi.
Strange, but it works now so I'm happy. Thanks!

[mcrossley]

Thinking a bit more about this, the pi zero has a different processor type from the 3/4. IIRC Mono compiles some components during install, as the image was created on a pi 3, the Mono compilation may be wrong for the zero.

So to use the pre-built image on a zero you would have to remove and reinstall the mono-complete package. I think!

[inimeg]

So to use the pre-built image on a zero you would have to remove and reinstall the mono-complete package. I think!

I did that and no success.
The pre-built image is capable of running on RPi zero. As I mentioned in my first post I did my very first install like this and it worked for around 3 days.