Cumulus Support

I was away for a few days and CMX crashed.
Normal operation is standard running and at 9:00 am a shutdown, backup, restart (takes a bit less then 2 minutes).
I have the logs for 5/4, 6/4 and the service console log attached.

There seems to be an issue with the cleanup of the realtime SQL table (just starting up the use of MySQL, just three standard tables, realtime standard retain two days) and it generates an error of too many open files. I now switched realtime off.

Addition: it also started having problems transferring realtime files and data files. Now it does upload the data files but still does not do the realtime.txt/gauges files.
Actually it did/does create 0 byte files. It may have to do with the provider, I have no idea.

Restarted with build 3122 and everything seems OK again.
So the only error that holds is the SQL error for the realtime.

It leaves me with the question why it started failing in the first place.

HansR wrote: ↑Tue 06 Apr 2021 8:25 pm Addition: it also started having problems transferring realtime files and data files. Now it does upload the data files but still does not do the realtime.txt/gauges files.
Actually it did/does create 0 byte files. It may have to do with the provider, I have no idea.

just curious, any of your devices having time issues?

ConligWX wrote: ↑Tue 06 Apr 2021 9:27 pm just curious, any of your devices having time issues?

Not that I am aware of though i'l double check the database as that one is with my provider.

And again this morning:
Using FTPS (on port 21!) and WinSCP works with the same setting.

It did work until this morning then I changed the setting to port 990 (which did not work) and reverted back to port 21. Rebooted but can't get it going.
Will reset the whole internet connection now with reboot of modem.

very strange. so it works for a while, then just stops?

Just wondering it the connections are not being disconnected once the data upload has finished, and therefore multiple connections build up until you reach a maximum amount of connections to the server.

might need debugging etc enabled until a fail again to decipher what's going on.

Resetting the whole chain does not help, CMX does not send the realtime files.
I stopped everything.

HansR wrote: ↑Wed 07 Apr 2021 8:14 am Resetting the whole chain does not help, CMX does not send the realtime files.
I stopped everything.

Server connections are most likely still active at the server end perhaps. they have not timed out yet.

One for Mark perhaps.

ConligWX wrote: ↑Wed 07 Apr 2021 8:14 am very strange. so it works for a while, then just stops?

Just wondering it the connections are not being disconnected once the data upload has finished, and therefore multiple connections build up until you reach a maximum amount of connections to the server.

might need debugging etc enabled until a fail again to decipher what's going on.

Will do a run with debug and then post it

OK here is the run with debug AND with FTP-logging on.
There seems to be something wrong on the FTP line, and error is not read and that is why it keep reconnecting which probably drains the resources and gives the too many files open. This may have to do with sockets resources being exhausted (a MS warning not opening too many connections).

To me it looks like a problem with the data channel only. FTPS (like FTP) uses two channels/connections to the server - the command channel (usually on port 21) and the data channel (a random port, can be specified by the server or the client). The command channel is also used for authentication. From your FTP log, authentication is working, as are commands. It is the data transfer channel that is failing - and this generally happens with "enthusiastic" firewalls.

This is why I advocate SFTP over FTPS. It is simpler to configure firewalls to accept as it uses a single port/channel for all traffic (port 22). I believe antagonist.nl offer SFTP connections. Perhaps try using SFTP?

@freddie: Yes, fine but this is something which suddenly failed. Both CMX as well as my CumulusUtils. So I asked them if they changed something. Nothing changed on my side and suddenly yesterday morning at 6h40 it started to fail. Then yesterday evening I restarted the whole thing with the new CMX version and everything worked like a shining sun until this morning when I tried something and it broke down again. Rebooted and reset everything two, three times and don't get it running again. SFTP is not an option at this moment.

I am not the evasive type, running around problems. If FTP(S) is offered as a service it needs to work and I need to understand.

HansR wrote: ↑Wed 07 Apr 2021 11:31 am @freddie: Yes, fine but this is something which suddenly failed. Both CMX as well as my CumulusUtils. So I asked them if they changed something. Nothing changed on my side and suddenly yesterday morning at 6h40 it started to fail. Then yesterday evening I restarted the whole thing with the new CMX version and everything worked like a shining sun until this morning when I tried something and it broke down again. Rebooted and reset everything two, three times and don't get it running again. SFTP is not an option at this moment.

I am not the evasive type, running around problems. If FTP(S) is offered as a service it needs to work and I need to understand.

I dont know about CumulusUtils sorry, but does that util also access ftp/ftps with the same login account? maybe have separate accounts for each program? just a thought.

Yes it uses the same login as for CMX. CumulusUtils runs for a minute or so, CMX runs 24/7. So CMX is actually what I am focussing on.
I could give it a try but it would surprise me though.