Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.10.5 (build 3122) - 06 April 2021 (please see announcement regarding releases since 3.5.0)
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

CMX crash and realtime retain failure

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

CMX crash and realtime retain failure

Post by HansR »

I was away for a few days and CMX crashed.
Normal operation is standard running and at 9:00 am a shutdown, backup, restart (takes a bit less then 2 minutes).
I have the logs for 5/4, 6/4 and the service console log attached.

There seems to be an issue with the cleanup of the realtime SQL table (just starting up the use of MySQL, just three standard tables, realtime standard retain two days) and it generates an error of too many open files. I now switched realtime off.
You do not have the required permissions to view the files attached to this post.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

Addition: it also started having problems transferring realtime files and data files. Now it does upload the data files but still does not do the realtime.txt/gauges files.
Actually it did/does create 0 byte files. It may have to do with the provider, I have no idea.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

Restarted with build 3122 and everything seems OK again.
So the only error that holds is the SQL error for the realtime.

It leaves me with the question why it started failing in the first place.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
ConligWX
Posts: 1223
Joined: Mon 19 May 2014 10:45 pm
Weather Station: Davis vPro2 Plus w/DFARS
Operating System: Ubuntu 20.04.2 LTS - NUC
Location: Bangor, NI
Contact:

Re: CMX crash and realtime retain failure

Post by ConligWX »

HansR wrote:
Tue 06 Apr 2021 8:25 pm
Addition: it also started having problems transferring realtime files and data files. Now it does upload the data files but still does not do the realtime.txt/gauges files.
Actually it did/does create 0 byte files. It may have to do with the provider, I have no idea.
just curious, any of your devices having time issues?
Regards Simon

https://www.conligwx.org - @conligwx
Davis Vantage Pro2 Plus with Daytime FARS • WeatherLink Live • Davis AirLink • PurpleAir PA-II-SD • CumulusMX •

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

ConligWX wrote:
Tue 06 Apr 2021 9:27 pm
just curious, any of your devices having time issues?
Not that I am aware of though i'l double check the database as that one is with my provider.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

And again this morning:

Code: Select all

2021-04-07 09:55:25.819 Realtime[2]: Uploading - realtime.txt
2021-04-07 09:55:25.990 FTP[2]: Error uploading realtime.txt to /public_html/realtime.txt : One or more errors occurred. (Authentication failed because the remote party has closed the transport stream.)
2021-04-07 09:55:25.990 Realtime[2]: Uploading - realtimegauges.txt
2021-04-07 09:55:27.098 FTP[2]: Error uploading web/realtimegauges.txt to /public_html/realtimegauges.txt : One or more errors occurred. (Authentication failed because the remote party has closed the transport stream.)
2021-04-07 09:55:27.099 Realtime[2]: Uploading extra web file[1] utils/airlinkrealtime.txttmp to /public_html/airlinkrealtime.txt
2021-04-07 09:55:28.213 FTP[2]: Error uploading utils/airlinkrealtime.txttmp to /public_html/airlinkrealtime.txt : One or more errors occurred. (Authentication failed because the remote party has closed the transport stream.)
2021-04-07 09:55:39.933 SendLoopCommand: Starting - LOOP 20
2021-04-07 09:55:55.869 Realtime[3]: Uploading - realtime.txt
2021-04-07 09:55:56.053 FTP[3]: Error uploading realtime.txt to /public_html/realtime.txt : One or more errors occurred. (Authentication failed because the remote party has closed the transport stream.)
Using FTPS (on port 21!) and WinSCP works with the same setting.

It did work until this morning then I changed the setting to port 990 (which did not work) and reverted back to port 21. Rebooted but can't get it going.
Will reset the whole internet connection now with reboot of modem.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
ConligWX
Posts: 1223
Joined: Mon 19 May 2014 10:45 pm
Weather Station: Davis vPro2 Plus w/DFARS
Operating System: Ubuntu 20.04.2 LTS - NUC
Location: Bangor, NI
Contact:

Re: CMX crash and realtime retain failure

Post by ConligWX »

very strange. so it works for a while, then just stops?

Just wondering it the connections are not being disconnected once the data upload has finished, and therefore multiple connections build up until you reach a maximum amount of connections to the server.

might need debugging etc enabled until a fail again to decipher what's going on.
Last edited by ConligWX on Wed 07 Apr 2021 8:15 am, edited 1 time in total.
Regards Simon

https://www.conligwx.org - @conligwx
Davis Vantage Pro2 Plus with Daytime FARS • WeatherLink Live • Davis AirLink • PurpleAir PA-II-SD • CumulusMX •

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

Resetting the whole chain does not help, CMX does not send the realtime files.
I stopped everything.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
ConligWX
Posts: 1223
Joined: Mon 19 May 2014 10:45 pm
Weather Station: Davis vPro2 Plus w/DFARS
Operating System: Ubuntu 20.04.2 LTS - NUC
Location: Bangor, NI
Contact:

Re: CMX crash and realtime retain failure

Post by ConligWX »

HansR wrote:
Wed 07 Apr 2021 8:14 am
Resetting the whole chain does not help, CMX does not send the realtime files.
I stopped everything.
Server connections are most likely still active at the server end perhaps. they have not timed out yet.

One for Mark perhaps.
Regards Simon

https://www.conligwx.org - @conligwx
Davis Vantage Pro2 Plus with Daytime FARS • WeatherLink Live • Davis AirLink • PurpleAir PA-II-SD • CumulusMX •

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

ConligWX wrote:
Wed 07 Apr 2021 8:14 am
very strange. so it works for a while, then just stops?

Just wondering it the connections are not being disconnected once the data upload has finished, and therefore multiple connections build up until you reach a maximum amount of connections to the server.

might need debugging etc enabled until a fail again to decipher what's going on.
Will do a run with debug and then post it
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

OK here is the run with debug AND with FTP-logging on.
There seems to be something wrong on the FTP line, and error is not read and that is why it keep reconnecting which probably drains the resources and gives the too many files open. This may have to do with sockets resources being exhausted (a MS warning not opening too many connections).
You do not have the required permissions to view the files attached to this post.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

freddie
Posts: 1232
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2
Operating System: GNU/Linux Ubuntu 20.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: CMX crash and realtime retain failure

Post by freddie »

To me it looks like a problem with the data channel only. FTPS (like FTP) uses two channels/connections to the server - the command channel (usually on port 21) and the data channel (a random port, can be specified by the server or the client). The command channel is also used for authentication. From your FTP log, authentication is working, as are commands. It is the data transfer channel that is failing - and this generally happens with "enthusiastic" firewalls.

This is why I advocate SFTP over FTPS. It is simpler to configure firewalls to accept as it uses a single port/channel for all traffic (port 22). I believe antagonist.nl offer SFTP connections. Perhaps try using SFTP?
Freddie
Image

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

@freddie: Yes, fine but this is something which suddenly failed. Both CMX as well as my CumulusUtils. So I asked them if they changed something. Nothing changed on my side and suddenly yesterday morning at 6h40 it started to fail. Then yesterday evening I restarted the whole thing with the new CMX version and everything worked like a shining sun until this morning when I tried something and it broke down again. Rebooted and reset everything two, three times and don't get it running again. SFTP is not an option at this moment.

I am not the evasive type, running around problems. If FTP(S) is offered as a service it needs to work and I need to understand.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
ConligWX
Posts: 1223
Joined: Mon 19 May 2014 10:45 pm
Weather Station: Davis vPro2 Plus w/DFARS
Operating System: Ubuntu 20.04.2 LTS - NUC
Location: Bangor, NI
Contact:

Re: CMX crash and realtime retain failure

Post by ConligWX »

HansR wrote:
Wed 07 Apr 2021 11:31 am
@freddie: Yes, fine but this is something which suddenly failed. Both CMX as well as my CumulusUtils. So I asked them if they changed something. Nothing changed on my side and suddenly yesterday morning at 6h40 it started to fail. Then yesterday evening I restarted the whole thing with the new CMX version and everything worked like a shining sun until this morning when I tried something and it broke down again. Rebooted and reset everything two, three times and don't get it running again. SFTP is not an option at this moment.

I am not the evasive type, running around problems. If FTP(S) is offered as a service it needs to work and I need to understand.

I dont know about CumulusUtils sorry, but does that util also access ftp/ftps with the same login account? maybe have separate accounts for each program? just a thought.
Regards Simon

https://www.conligwx.org - @conligwx
Davis Vantage Pro2 Plus with Daytime FARS • WeatherLink Live • Davis AirLink • PurpleAir PA-II-SD • CumulusMX •

User avatar
HansR
Posts: 1996
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

Yes it uses the same login as for CMX. CumulusUtils runs for a minute or so, CMX runs 24/7. So CMX is actually what I am focussing on.
I could give it a try but it would surprise me though.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3122 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

Post Reply