MQTT stops working after reboot

[DrVoidberg]

I ran into some issues when connecting my CMX to Home Assistant via MQTT.

I'm running the latest CMX version on a Raspberry Pi 5, which also runs Home Assistant OS in a proxmox VM. I have a DIY NAS connected to the Pi as well shared via samba, but for some reason I can't access it from my Windows machines after around a week (bear with me, this is an important piece to the puzzle). Rebooting fixes this issue, so the Pi does a scheduled reboot every night at 5:00 after the HA backup.

After setting MQTT up initially, everything works fine up until the reboot at night. Then the sensors become unavailable and no data is coming in any more. Rebooting the whole system does not fix it. I switched from Mosquitto to EMQX, but it showed the exact same behavior. I was hoping to get some more insight with EMQX, but after the failure there was no sign of CMX even being connected. So no logs, no errors, nothing.

During a manual reboot I noticed that CMX is up and running a few seconds later, while HA takes a few minutes (running in a VM and all that). So I wondered if that might be the problem somehow and restarted the cumulusmx service. Lo and behold, everything came back to life and data started pouring in again. To verify I repeated the whole process with longer time between rebooting and restarting the service with exactly the same outcome.

It seems like when CMX starts, it tries to connect to the MQTT broker, fails and simply does not try again?
I'll try with scheduling a service restart 5 minutes after the reboot for now. I know it's not an elegant solution, but I didn't find an easy way to delay the start of a service on boot either.

I figured I report the issues here as it seems like unintended behavior.
Sorry if this is just plain stupidity on my part. I'm quite proficient with Windows, but when it comes to Linux half of the time I don't even know what I'm doing Even less so with MQTT. So please, no hesitation in pointing out stupidity on my part

P.S.: I really love the piece of software all of you have created

[mcrossley]

I'll look at the not retrying to connect, but there are start-up delay settings in Cumulus, under Program Settings.

Could you also post a log file from when Cumulus failed to connect - there may be some clues in there.

[DrVoidberg]

Oh I didn't know about this setting (obviously). I'll try it out, thanks!

I rebooted and gave it 5 mins before restarting the service.
These are the first MQTT lines, after that it goes straight to the errors:

Code: Select all

2025-03-01 21:15:35.934 MQTT Error: failed to connect to the host
2025-03-01 21:15:35.934 Error while connecting host 'Unspecified/192.168.1.3:1883'.
2025-03-01 21:15:35.949 MQTT: Reading template file - mqtt/CumulusMX.txt
[...]
2025-03-01 21:15:36.135 MQTT: Error - Not connected to MQTT server - message not sent
2025-03-01 21:15:36.135 MQTT: Error - Not connected to MQTT server - message not sent
2025-03-01 21:15:36.135 MQTT: Error - Not connected to MQTT server - message not sent

From the other log files it looks like this just goes on forever, giving several MB worth of log files.

192.168.1.3 is the IP of the VM that runs Home Assistant. I don't know how far in the start up process it is at that point in time though. The Pi running CumulusMX and the VM has a different IP, so localhost won't work.

__________________
Then I restarted the cumulusmx service and gave it another 5 mins. No MQTT errors.

Both log files attached.

(Don't mind the copy errors. That probably comes from a wrong setting I did when first setting CumulusMX up ~2 years ago. I'll take care of that.)

--------------------------
Edit:
Update: I set the start-up delay to 180s and it seems to have done the trick. Thanks again for pointing me in the right direction. I'll keep a close eye on it in the next days.

If you need any more info and/or experimentation, please let me know.

[mcrossley]

I have completely rewritten how Cumulus connects/reconnects to the MQTT server in the next release. It is simplier, more robust and invokes an active check of the connection every 20 seconds.

[DrVoidberg]

I did not expect to trigger a feature rewrite with this

I'll make sure to test it in the next release.
So far the delay has been working reliably.

[mcrossley]

Don't worry, it needed doing, and it has triggered thoughts of a new better way for me to handle FTP connection handling as well That will have to wait for a future release though.