Page 1 of 1

FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Mon 31 Dec 2018 4:55 pm
by Dinant
Today I replaced b3043 with b3045 because I want to know if the FTP upload with TLS1.2 is working right now.
I got errors during the upload.
In MXdiag is says:

Code: Select all

2018-12-31 17:45:00.668 Error connecting ftp - The handshake failed due to an unexpected packet format.
2018-12-31 17:45:00.715 Error uploading web\graphconfig.json to /cumulus/graphconfig.json : The handshake failed due to an unexpected packet format.
2018-12-31 17:45:00.762 Error uploading web\tempdata.json to /cumulus/tempdata.json : The handshake failed due to an unexpected packet format.
I enabled FTP logging.

Code: Select all

2018-12-31 17:45:00.590 Connecting to ftp.xxx.xx

# Connect()
Status:   Connecting to xxx.xxx.xxx.xxx:21
Response: 220 ::ffff:xxx.xxx.xxx.xxx FTP server ready
Command:  AUTH TLS
Response: 234 AUTH TLS successful
2018-12-31 17:45:00.668 Uploading web\graphconfig.json to /cumulus/graphconfig.json

# OpenWrite("/cumulus/graphconfig.json", Binary)
Status:   There is stale data on the socket, maybe our connection timed out or you did not call GetReply(). Re-connecting...
Status:   Disposing FtpSocketStream...
I also tried to add disableEPS to cumulus.ini:

Code: Select all

[FTP site]
DisableEPS=1
but without success. Anyone a suggestion?

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Mon 31 Dec 2018 5:11 pm
by Dinant
I've read the release notes of b3045 again and now I see I missed the line:
Now supports SFTP over TLS1.1 and 1.2

I 've tested with FTPS not SFTP! So FTPS won't work.
I'll try with SFTP and let you know the results.

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Mon 31 Dec 2018 5:26 pm
by Dinant
My provider only permits me to connect to the web server with FTPS with username/password or SSH with a SSH private/public keypair.
Cumulus 3.00 b3045 only supports SFTP with username and password authentication.
So it seems I cannot connect to the web server with Cumulus b3045.

On the Github site of FluentFTP is says:
Full support for FTP, FTPS (FTP over SSL), FTPS with client certificates and FTPS with CCC (for FTP firewalls)
That's confusing. Does Cumulus support SFTP or FTPS?

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Mon 31 Dec 2018 7:00 pm
by mcrossley
Oops, a slip of the keyboard - it is of course FTPS.

CMX does not support SSH file transfers, nor does it support FTPS with client certificates.

B3045 is working OK in my test environment to my NAS using FTPS (and EPSV).

Code: Select all

Status:   Connecting to #######:21
Response: 220 #### FTP server ready.
Command:  AUTH TLS
Response: 234 AUTH TLS command successful.
Status:   FTPS Authentication Successful
Status:   Time to activate encryption: 0h 0m 0s.  Total Seconds: 0.7225609.

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Tue 01 Jan 2019 11:04 am
by Dinant
OK, I understand. FTPS is the only protocol supported.
In the meantime I've send an email to my provider. Maybe they can see something in the logfiles of the server why the FTPS upload is not working.

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Wed 02 Jan 2019 11:47 am
by Dinant
I've tested with my Synology NAS and the FTP upload is working fine.
I see a 15 seconds delay between the files.

Code: Select all

2019-01-02 12:10:01.046 Uploading web\graphconfig.json to /weer/cumulus/graphconfig.json
2019-01-02 12:10:16.046 Uploading web\tempdata.json to /weer/cumulus/tempdata.json
2019-01-02 12:10:31.312 Uploading web\pressdata.json to /weer/cumulus/pressdata.json
2019-01-02 12:10:46.578 Uploading web\winddata.json to /weer/cumulus/winddata.json
2019-01-02 12:11:01.845 Uploading web\wdirdata.json to /weer/cumulus/wdirdata.json
2019-01-02 12:11:17.110 Uploading web\humdata.json to /weer/cumulus/humdata.json
2019-01-02 12:11:32.423 Uploading web\raindata.json to /weer/cumulus/raindata.json
2019-01-02 12:11:47.689 Uploading web\solardata.json to /weer/cumulus/solardata.json
2019-01-02 12:12:02.970 Uploading web\dailyrain.json to /weer/cumulus/dailyrain.json
2019-01-02 12:12:18.251 Uploading web\sunhours.json to /weer/cumulus/sunhours.json
2019-01-02 12:12:33.501 Uploading web\dailytemp.json to /weer/cumulus/dailytemp.json
Does anyone else see such a delay?

The upload to the webserver is still not working. I am still investigating this together with my provider.

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Wed 02 Jan 2019 1:38 pm
by mcrossley
I have found a problem with the TLS implementation in b3045, it affects the 'normal' periodic uploads using TLS1.2, but not the realtime uploads. I'll fix it in b3046. Sorry about this.

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Wed 02 Jan 2019 3:33 pm
by Dinant
No problem. I am very happy that you are being able to update the Cumulus software. I tried it myself but I'm not able to understand the whole package but I can help with certain things. So if you need someone to help, test, etc. I'll be there to help.

My provider told me the software is not connecting with TLS 1.1 or TLS 1.2.
In the Cumulus.cs file I think there is an error.

Code: Select all

if (Sslftp)
{
    conn.EncryptionMode = FtpEncryptionMode.Explicit;
    conn.DataConnectionEncryption = true;
    conn.ValidateCertificate += Client_ValidateCertificate;
    // b3045 - switch from System.Net.Ftp.Client to FluentFTP allows us to specifiy protocols
    RealtimeFTP.SslProtocols = SslProtocols.Default | SslProtocols.Tls11 | SslProtocols.Tls12;
 }
I think this should be change to:

Code: Select all

if (Sslftp)
{
    conn.EncryptionMode = FtpEncryptionMode.Explicit;
    conn.DataConnectionEncryption = true;
    conn.ValidateCertificate += Client_ValidateCertificate;
    // b3045 - switch from System.Net.Ftp.Client to FluentFTP allows us to specifiy protocols
    conn.SslProtocols = SslProtocols.Default | SslProtocols.Tls11 | SslProtocols.Tls12;
 }
Is this the error you are talking about?

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Wed 02 Jan 2019 3:48 pm
by mcrossley
Yep - classic cut and paste, and when I tested it my NAS was allowing fallback to SSL :(
I just retested it now with enforced TLS 1.2 only and it works fine.

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Wed 02 Jan 2019 7:28 pm
by Dinant
I have installed B3046 and now the FTPS upload is fine.

The first time I enabled ActiveFTP. I got an error saying "500 Illegal PORT command".

Code: Select all

# OpenActiveDataStream(PORT, "STOR /xxx/graphconfig.json", 0)
Command:  PORT xxx,xxx,xxx,xxx,yyy,zzz
Response: 500 Illegal PORT command
Status:   Disposing FtpSocketStream...
After disabling Active FTP mode the upload went well.
In the ftplog.txt file is says:

Code: Select all

# OpenPassiveDataStream(AutoPassive, "STOR /xxx/dailytemp.json", 0)
Command:  EPSV
Response: 229 Entering Extended Passive Mode (|||35047|)
Status:   Connecting to xxx.xxx.xxx.xxx:yyy
Command:  STOR /xxx/dailytemp.json
Response: 150 Opening BINARY mode data connection for /xxx/dailytemp.json
Status:   FTPS Authentication Successful
Status:   Time to activate encryption: 0h 0m 0s.  Total Seconds: 0.0156678.
Status:   Disposing FtpSocketStream...
Response: 226 Transfer complete
Status:   Disposing FtpSocketStream...
Now the 15 seconds delay is also gone. The 12 json files are transferred in less than one second.

Re: FTP TLS 1.2 upload not working with v3.0.0 build 3045

Posted: Wed 02 Jan 2019 8:59 pm
by mcrossley
Dinant wrote: Wed 02 Jan 2019 7:28 pm Now the 15 seconds delay is also gone. The 12 json files are transferred in less than one second.
Good, I suspect it was failing the connection and negotiating down the protocols. The way it is configured the periodic upload shares the realtime connection, it just creates another control channel. So the connection was made originally as TLS then the periodic connection didn't have TLS enabled - confusion reigns!