SFTP and processed files

[sutne]

I have had problems with FTP and SFTP uploads to my web-host server for some years, but for the last months it has been quite stable, only a few errors each day.
Two days ago it was a lot of errors probably because of a change at the web-host provider. (They have not verified this.)
From yesterday CumulusMX can no longer upload Processed-files.
I have had discussions with Support at the web-host, but since FileZilla is working they have concluded that it is a problem with my program.

Enclosed is two diag-files, one where the files that shall be Processed before upload is excluded, and one where they are included.

I think there must be a timing problem within CumulusMX that causes the SFTP to malfunction when the files are processed.

[mcrossley]

All the file processing happens before the transfer.
The realtimegauges.txt file is also a processed template file, and that is uploading OK.

SFTP does not use the FTP log file, that is only used by the standard FTP component. Additional messaging for SFTP to the MXdiags log is via both FTP and debug logging being enabled.

The error back from SFTP isn't very helpful = "Failure"


I'll see if an inner exception is generated and add that to the logged message in a future build.

[sutne]

All the file processing happens before the transfer.

Yes, but there is a difference when it starts on the extra files. Is it an open connection or something like that?

Additional messaging for SFTP to the MXdiags log is via both FTP and debug logging being enabled.

Do you want me to enable both?

[mcrossley]

Yes, please enable both FTP and Debug logging. Not that I think it will give us much more to go on.

[sutne]

No, probably not much more here:

20210519-141855.txt

[mcrossley]

Nope nothing extra.

EDITED:

My suggestions...
Check the file "realtime-x.txttmp" on your web server....
Is it zero length? If so delete it.
Are the permissions correct? (though I would expect a permissions error rather than a session abort)

Then try setting the upload type to binary, does that work?

[mcrossley]

Post above edited.

Then try disabling that extra file and see if the next one works.

[sutne]

Mark,
I tried your suggestions without really getting anywhere, and then there was a (planned) 5 hours power cut up in the mountains where the weatherstation and the Raspberry Pi is located.

When it came back again, I inserted the failing Extra web-files and now everything runs fine.

So maybe there is something with RPi or the SD-card.

Thank you for your involvement in my problems.
Regards
Steinar

Edit: I have been thinking: If there is a problem with the files itself, why can I always upload the files manually with FileZilla after CumulusMX has failed?
And I wonder what does SFTP mean with "disposed object"?
The main problem here is that the error messages are not very informative.

[sutne]

Still problems:
Yesterday I noticed the realtime had stopped on my web-site at 15:34. The graphs (json-files) were updated but not the realtime.
It was repeating error messages each realtime run.

Code: Select all

2021-05-25 15:40:58.493 Realtime[28]: Warning, a previous cycle is still trying to connect to FTP server, skip count = 1
2021-05-25 15:40:58.493 Realtime[28]: No FTP attempted this cycle
2021-05-25 15:43:58.486 Realtime[29]: Warning, a previous cycle is still trying to connect to FTP server, skip count = 2
2021-05-25 15:43:58.486 Realtime[29]: Realtime has been in progress for more than 5 minutes, attempting to reconnect.
2021-05-25 15:44:22.320 Realtime[29]: Reconnected with server OK

I deleted the realtime-files on the web-site. Same error.
I deleted the local realtime-files. They were recreated but same ftp-error.
I uploaded the realtime-files with FileZilla, no problem.
Then I restarted CumulusMX and it has been running OK.

In the log file you will see it was OK at 15:34, but at 15:37 there is only one message for the first realtime-file, then nothing more. not OK, not Error.
Then from 15:40 there are the same error messages repeated every 3 min.

It seems CumulusMX get stuck in the FTP-process only for realtime, not for the main upload (hourly).

[mcrossley]

It seems to be getting into a state where it does not recover itself.

The interval update works differently, it creates a new FTP connection each time it is invoked. The real time updates maintain a single connection and keep it open permanently

I'm wondering if your very long real time interval is causing some timeouts. Having checked the library it appears the keep-alive heartbeat is disabled by default. CMX should probably enable it if the real time period is longer than 30 seconds.

Is there any reason you could not decrease your real time interval from three minutes? I think the original intent was it would be used for fast updates of 30 seconds or less (guessing here as I don't know the history).


It doe not help you now, but I have re-written the real time FTP recovery process in the next full release to improve error handling.

[sutne]

I agree that 180s is not really real-time but it is the best I can afford at the momemt.
The Internet line is a limited mobile access at my cabin up in the mountain.
I would also like to have some data available when I am up there.

The hourly upload also fails from time to time. But last week only two times, and it recovers.

I can live with this, and hopefully there will be a fix in the future.

[mcrossley]

OK, fair enough. As I say I have improved the real time recovery (especially of SFTP) in the next release, I'll enable the keep-alive too which may help.