Cumulus stalls at start-up

[flort]

Hi,

I have had to do a couple of restarts of my RPi in the last week as well as having a power outage tonight and each time Cumulus has stalled on start-up. I have it set to ping an address on start-up with a ping escape time of 5 minutes. Even after 10 minutes it is still sitting there waiting for ping reply and yet I can manually ping the address without any issues. The only way I can get it going is to restart the Cumulus service and the ping works immediately and the system continues to initialise.

I have attached a copy of the Console Log and MXDiags. Below is a copy of my settings.

Settings.jpg

Does anyone have any ideas as to what might be going on?

Regards,
Trevor

[HansR]

By accident I had a similar issue a few days ago.

It appeared to be a crash (so no matter how long you wait, CMX simply is not there, you should be able to check that with htop). The crash can be see in the journal (make sure you look at the right day/time). Mark Advise to add Wants=network-online.target.
So the start of the service file of systemd becomes:

Code: Select all

[Unit]
Description=CumulusMX service
Documentation=https://cumuluswiki.org/a/Main_Page
After=network-online.target
Wants=network-online.target

[flort]

Thanks Hans. I have added the line to service file but a reboot still has it hanging. Excuse my ignorance, but where do I find the journal to check the crash?

Checking with htop, it definitely appears nothing is happening with the Cumulus service.

Regards,
Trevor

[HansR]

command for the journal is:

Code: Select all

journalctl

Read the manual pages.
Cursors keys make you scroll, if you type a / you can give a search string, / without string repeats the previous search.
Search for cumulus or cumulusmx should give you maybe some lines.

[Phil23]

Search for cumulus or cumulusmx should give you maybe some lines.

Caught me out for a moment, Case Sensitive....

[flort]

Attached is the journal from when I did a reboot. Hoping you can help me interpret this because the first mention of cumulus is when I issued the command to restart the service as it had hung at the ping stage.

I have now done several reboots and it hangs every time during the ping process. A restart of the service is required to kick things off.

[HansR]

Yes, I do not see the start of the cumulusmx service, only the kill attempt and then a start, which is kind of weird.
It does not seem to be able to remove the lockfile (in /tmp). You might want to check if it still is there when CMX is not running.
If CMX is NOT running and you see a lock file, please remove it.

Then the journal tells me it starts (apparently correctly).
This means you must have two MXdiags logfiles in quick succession. Can you confirm that? If not then CMX must have stopped before it started logging. That would be pretty quick.

Could you post both logfiles and if not two then the one(s) with the the reboot?

[HansR]

And btw: My journal only says:

Code: Select all

systemd[1]: Starting CumulusMX service...
[...]
systemd[1]: Started CumulusMX service.

for version 3.15.0 b3169

[flort]

Hi Hans,

Firstly, I think the tmp file was a hangover because Cumulus didn't finish starting up. I stopped the service and the tmp file was deleted then I started the service and the tmp file was created again.

So I did another reboot so as to create a log file where it hung and then did a service restart to create a log file where everything ran through successfully. Attached are both logs along with the journal file from the reboot. It does appear to contain the initial instance of Cumulus starting this time, then the termination and restart when I issued the terminal command to restart the service.

As a trial, I changed the host address to a different IP address but the issue still persisted. I also removed the IP address and put a system start-up delay of 30 seconds and that continued with the Cumulus start-up once the 30 seconds had passed.

Appreciate your help with this.

Regards,
Trevor

[HansR]

Atm I am out of ideas.
I'll look at it again later today, maybe others have a look at it as well (invitation to do so )

[flort]

Thanks Hans. If I can't get it working I'll run with the time delay option but it would be nice to get to the bottom of the issue as I prefer the ping method.

Regards,
Trevor

[bburns]

Hi:
I am seeing the same startup problem.
The following is the MXDiag log with debug enabled:

2022-02-05 11:55:47.302 ========================== Cumulus MX starting ==========================
2022-02-05 11:55:47.303 Command line: /usr/lib/mono/4.5/mono-service.exe -d:/opt/CumulusMX CumulusMX.exe -service
2022-02-05 11:55:47.303 Cumulus MX v.3.15.0 build 3169
2022-02-05 11:55:47.305 Platform: Unix
2022-02-05 11:55:47.305 OS version: Unix 5.10.0.11
2022-02-05 11:55:47.305 Current culture: English (United States) [en-US]
2022-02-05 11:55:47.305 Mono version: 6.8.0.105 (Debian 6.8.0.105+dfsg-3.2 Tue Jun 29 21:01:01 UTC 2021)
2022-02-05 11:55:47.305 Checking required folders
2022-02-05 11:55:47.319 Reading Cumulus.ini file
2022-02-05 11:55:47.330 AvgSpdMins=10 AvgSpdTime=00:10:00
2022-02-05 11:55:47.332 Cumulus start date: Friday, January 1, 2021
2022-02-05 11:55:47.345 Reading Cumulus.ini file completed
2022-02-05 11:55:47.345 Directory separator=[/] Decimal separator=[.] List separator=[,]
2022-02-05 11:55:47.345 Date separator=[/] Time separator=[:]
2022-02-05 11:55:47.345 Standard time zone name: EST
2022-02-05 11:55:47.345 Daylight saving time name: EDT
2022-02-05 11:55:47.348 Daylight saving time? False
2022-02-05 11:55:47.348 2/5/2022 11:55:47 AM
2022-02-05 11:55:47.348 Stop second instance: No other running instances of Cumulus found
2022-02-05 11:55:47.349 System uptime = 4 secs
2022-02-05 11:55:47.349 Delaying start for 30 seconds
2022-02-05 11:56:17.349 Start-up delay complete, continuing...
2022-02-05 11:56:17.350 Waiting for PING reply from 10.1.1.1
2022-02-05 11:56:17.399 Ping reply: System.Net.NetworkInformation.PingReply
2022-02-05 11:56:19.861 Ping Error: The PING failed to return after the timeout, cancelling it...


1. Startup delay set for 30 seconds. (appears be working ok)
2. Ping address is to my local router. (the ping fails from CUMX but with a simultaneous Linux console session the ping is 100% successful)
3. Ping Escape time (mins) is set to 5 mins.

A couple of observations:
1. The startup delay seems to begin before the Ping Request. The configuration screen indicates the delay should start after a successful ping.
2. The ping error occurs about 2.5 seconds after the first ping timeout. The expected time is set in cumulus.ini to 5 minutes.
3. CUMX hangs forever after this error.

Thanks Mark and everyone for having a look.
Regards,
Byron

[HansR]

I too have a ping and a timeout of 3 minutes. It works flawlessly (with b3169).
Note that my ping is to an outside network server, yours on a local address - is that true for a 185.27.134.3 address???

So the only thing I can come up with is that it must be in the configuration of your RPi. I assume you have configured using raspi-config. There is an option :

Code: Select all

System Options=>NetWork at Boot

(see screenshot). I always activate this. Do you? If not I would advise it. If this is not it, I would ask @mcrossley to kick in because in my opinion that would mean that in the code around the ping there would be a possibility for an endless loop.

Schermafbeelding 2022-02-06 122710.jpg

[mcrossley]

@florts case is slightly different in that the PING hangs and CMX never escapes from it. I can see a possible cause for that and will fix it in the next release.

The start-up delay is BEFORE the ping, contrary to what it says on the settings page - I will get the settings page help text fixed. The idea being to allow time for the network to initialise.

The code uses the same ping object for every attempt. I'll change that so that it creates a new ping object for every attempt - let's see if that changes the behaviour where the ping seems to be perpetually "locked" into not seeing the network...

[flort]

Thanks everyone. I'll post the result after the next release.

Regards,
Trevor