No auto reconnect to Maria DB10 after DB Backup

[jrichert]

Hi

I have a problem with Cumulus MX version 3.13.2 build 3147 connecting to a Maria DB 10 on Synology NAS.
With the old Maria DB 5 Synology OS 6 and older Cumulus it was working fine.
After upgrading my Synology to Maria DB 10 and Synology OS 7x I have the problem, that Cumulus MX is no longer connecting automatically to the DB after I have done a Synology Backup.
I can't exactly tell if the problem occured with the latest Cumulus or SynOS version update.
In Cumulus MX Settings/MySQL/GeneralOptions "Buffer command on failures" is ticked.
When the backup starts, Cumulus MX is sending endless E-Mail warnings with sql connection error. If I just restart CumulusMX, connection is re established and all is fine.
Any ideas what I could do?

Many thanks
Jochen

P.S. and it seems, that the AWEKAS transfer process is also stopped or not working

[jlmr731]

Not sure how it does that backup on synology, but sure sounds like on how its done is the problem. Cumulus only sends an insert command to the database and yes will complain if it fails. Maybe synology take the database offline for the backup.
Dont know if you can manually run commands on it, like to ssh in or if there is a shell app for it, but if you can try to run mysqldump to see what happens

Code: Select all

mysqldump --user=username --password=password databasename > path/to/backup.sql

[jrichert]

Hi Jeff

Thank for your response. The backup product is called Hyper Backup from Synology and yes, it takes the DB service offline for the backup.
If I receive the error messages during the backup it would be no problem, but it seems, that Cumulus itself stalls internal.
In the same time I receive also a message from Awekas, that Cumulus is no longer delivering data to it.
After the backup has finished I still receive warning mails from cumulus. The backup job end at 05:30 and Cumulus stil sent warning e-mails to me on 09:30, until I stopped and started cumulus again.

What I could do, I try to make a single DB backup JOb which should shorten the whole backup process and see, if Cumulus will tolerate the shorter offline time.

Any further ideas?
Many thanks
Jochen

[mcrossley]

Can you switch on debug logging, then post the MXdiags file after it next fails, thanks.

[jrichert]

Hi Mark

I've just activated debug logging.
It seems, I found a workaround. I separated the Maria DB Backup Job from the full backup. A test run this afternoon took around 17 minutes and I received only one error message from Cumulus.
I'll wait a few days and if all is working fine, I would try to add Maria DB Backup into the full Backup for testing.

Cheers
Jochen

[jlmr731]

Would still send that mxdiag to Mark seeing you still got an error maybe there is something there to help.
Are you running cumulus on the synology? if so can you remove it from the backup job too?

[jrichert]

Hi Jeff

No, Cumulus MX is running on a Virtual W10 PC in a VMware environment. And it looks, with a single backup job only for my 17 GB Maria DB, Cumulus MX only sends one warning e-mail and after that connecting again.
I have to check, when I could do the test to implement Maria DB in the full backup again, as if I miss the error messages, I am loosing data, data which is sent to AWEKAS and also into my allsky cam.

[jrichert]

OK, here we go. Cumulus stalled again.
I had debug logging activated. Cumulus MX still filling up my mailbox by sending endles warning mails, until cumulus is stopped and started again.

Here's the Zip File...

Jochen

[mcrossley]

I see the real-time SQL connection is working. But the interval connection is failing. That is odd as they both use the same login details.
The only difference is the interval connection is created anew each time it is required, but the real-time connection is left permanently connected.

I can amend the checking of the MySQL connection before the upload of the buffered SQL statements takes place to perform a command and output the result.

It looks like the "check connection" works and closes the test connection, but then when it immediately tries to reconnect to upload the data it is failing. Perhaps you could check your server MySQL logs to see if there is a reason in there?
When you restart CMX, the buffered commands are lost and only a single connection is made to upload the data. It looks like you server does not like the two connections one immediately after the other.

As for you emails it sound like you have not configured latch or thresholds value for the alarm so it will alarm afresh every time a connection fails. You probably want to throttle that to once an hour and only if you get 2 or more failures?


other observations...
- You really want a real-time interval of 1 second!
- You are creating realtime.txt (do you use it?) and realtimegauges.txt as standard files, then in extra files you are processing realtimegaugesT.txt again - twice?
- What is there in you gauges-ssT.htm page that needs updating once a second (again twice)? Normally that is a static page that never changes?

[jrichert]

Hi Marc

Thanks for the feedback. I'll try to find out, how to access the Maria DB logs on my Synology NAS.

As for you emails it sound like you have not configured latch or thresholds value for the alarm so it will alarm afresh every time a connection fails. You probably want to throttle that to once an hour and only if you get 2 or more failures?
Thanks for the hint, I did not realized it, bad English understanding, never asked myself, what Latch would mean. Now I reconfigured it to 2 times in 1 hour. Thanks makes my life/mailbox handling easier

other observations...
- You really want a real-time interval of 1 second!
I have to check. Most of these parameters where brought over from an old Cumulus Version. In early times with cumulus I liked to have realy fast respondig instruments for wind and gust.

- You are creating realtime.txt (do you use it?) and realtimegauges.txt as standard files, then in extra files you are processing realtimegaugesT.txt again - twice?
I am using realtime.txt for my Allsky Cam SW to get some weather data

- What is there in you gauges-ssT.htm page that needs updating once a second (again twice)? Normally that is a static page that never changes?
I probably had not yet understood the mechanism behind these files I was switching over from the Cumulus version and played around with the realtime gauges, which are on different pages. the homemade one under http://wetter.richert.ch/gauges/gauges-ss.htm and the other realtime gauges implemeted into http://wetter.richert.ch/cumulusmx/index.htm

Many thanks for your help and hints
Jochen

[mcrossley]

I have changed the code slightly for the next release - it may helps a little for you when reconnecting.

The "check connection" now fetches some data from the server and logs it - so we will definitely know if it succeeded. It also actively closes the connection, then waits a short time before allowing the main upload to connect.

[jrichert]

OK, so I'll wait for the 3155, 3154 does not makes sense to me, as there is only an update for a GW1000 WS or so.

I'll kepp you informed ...

thanks again
Jochen