Build 3081 - new SFTP errors

[Vegit8]

Upgraded to 3081 this morning.

I am now getting a sharing violation error.
These started within 3 minutes of loading MX
I have reverted to 3080, where all seems well at the moment.

Code: Select all

2020-05-29 10:16:06.541 Realtime FTP[6]: Error during update: Sharing violation on path /home/geoff/CumulusMX/web/betel_datasummaryT.htmtmp

I have attached the MX Diags file

[water01]

Mine is working fine, so it looks more like a permissions failure on the file on hosting site. Check it has the same permissions as the other files. If there is a zero length file present, delete it.

[mcrossley]

I need to see a log file with debug and FTP logging enabled please. The FTP connection has clearly failed for whatever reason.

There was a change to the realtime process in that previous releases do NO realtime processing if the FTP failed. Now it does all the processing each time, even if it cannot FTP. But if the FTP has hung and still has a file locked then I can this error occurring, in that case it it pretty benign.

[Vegit8]

Running 3081 on Ubuntu

Debug log attached, No FTP logging file found, rechecked FTP logging enable and is still enabled even after running up 3080 - confusing

repeat of previous error, this time with a different file.

Nothing has change either at the local end or the remote end.
Now reverted to 3080, which was more stable, but still had FTP issues.

MX Diags attached

[mcrossley]

Thanks, I have found the problem with real time SFTP. One step at a time it seems to sort out this real time FTP!

SFTP does not log to the FTPlog as its not really FTP - it enables additional logging in the MXdiags file.

[Vegit8]

Further update...
Another set of failed Realtime updates from my friends site last night.

Code: Select all

2020-05-31 22:30:00.634 Error during realtime update: Sharing violation on path /home/geoff/CumulusMX/web/gauges-ssT.htmtmp
2020-05-31 22:30:00.634 Error connecting ftp - Client not connected.
2020-05-31 22:30:00.751 Writing log entry for 31/05/2020 22:30:00
2020-05-31 22:30:00.752 Written log entry for 31/05/2020 22:30:00
2020-05-31 22:30:00.752 Writing today.ini, LastUpdateTime = 31/05/2020 22:30:00 raindaystart = 54.8 rain counter = 54.8
2020-05-31 22:30:00.752 Warning, previous web update is still in progress,second chance, aborting connection
2020-05-31 22:30:00.752 Trying new web update
!!! Unhandled Exception !!!

I now have remote access to the laptop, and the first thing I noticed when connecting was that the screen refresh was exceptionally slow.

I have since found out that there was a video upload going on at the same time, to another part of his site.
So I am guessing that there must be some interaction between the SFTP sessions (maybe as a result of potential restrictions placed by the server at the far end).
Perhaps the 'Sharing violation' is a mislabeled error.

I will contact the person who manages the site on the behalf of my friend and ask if this may be the case.
There is a secondary issue (probably not related) that the site will not display the gauges_ss.htm page any more. It just hangs.

MX Diags file attached

[Vegit8]

Update to above...

Once a file has been locked

Code: Select all

2020-06-01 08:27:38.909 Error during realtime update: Sharing violation on path /home/geoff/CumulusMX/web/realtimegaugesT.txttmp

It will not unlock until MX is restarted

[mcrossley]

Please try MX version 3.6.6 and see if that improves matters.

I will have to look at the file locking as a separate issue, I think I know the cause, but it will have to wait for another release. Meantime the new release should reduce the occurrence of that error.

[Vegit8]

Thanks Mark - will do
Just as an additional bit of info.
When he uploads files they go to YouTube, NOT to his website, so there is no clash of (S)FTP connections.

I'm wondering if it's some sort of timing issue due to poor bandwidth, he is on standard ADSL - no fibre.

[freddie]

Have you tried experimenting with a longer realtime interval?

[Vegit8]

In response to freddie, I added 50% to the realtime upload parameter - so it is now 45 seconds.

After running for a good number of hours, I get the same problem

Code: Select all

2020-06-02 15:56:52.888 Realtime FTP[164]: Uploading - /home/geoff/CumulusMX/web/betel_datasummaryT.htmtmp
2020-06-02 15:56:52.888 SFTP: Starting Upload of /home/geoff/CumulusMX/web/betel_datasummaryT.htmtmp to public_html/weather/betel_datasummary.php
2020-06-02 15:56:54.306 Realtime FTP[165]: Error during update: Sharing violation on path /home/geoff/CumulusMX/web/betel_datasummaryT.htmtmp

I know Mark is looking at this problem, this update is to inform that increasing the update time had no effect on the overall issue.

[freddie]

Thanks for giving it a try.

[mcrossley]

Are you now running ver 3.6.6?

If so can I see the full log file please - preferably with debug and and ftp logging enabled.

The real time uploads should be kept to a minimum, only the current wind values really change much at that sort of rate.

To reduce overhead, why are you uploading the beteldatasummary at the real time interval. Isn't that a file that only changes once a day, therefore really should be in extra files with the EOD option selected not real time? Or even... my understanding of the datasummary script was that it read the dayfile for its data, why does it need processing and uploading by MX at all? My copy reads from my MySQL database - so the db needs updating once a day, but the PHP file never updates.

[Vegit8]

Hi Mark
Yes running 3082.

I restarted yesterday (03/06) with debugging running.
As you mentioned previously FTP logging doesnt work with SFTP, so no FTP log
I also disabled realtime uploading of the betelsummary stuff.


After 20 hours MX crashed - same error

Code: Select all

2020-06-04 06:23:49.156 Realtime FTP[15]: Error during update: Sharing violation on path /home/geoff/CumulusMX/web/realtimegaugesT.txttmp

I have attached the diags file.
This laptop does nothing else other than run MX, its almost as if there is some sort of file locking in place.

[mcrossley]

Thanks for the log file. The problem
There is some underlying issue with the file transfer though - it seems very slow, and is exacerbating the problem...

Code: Select all

06:23:15.104 SFTP: Starting Upload of web/realtimegauges.txt
...
06:23:37.051 SFTP: Completed uploading web/realtimegauges.txt

That's 22 seconds to transfer the real time file, typically with SFTP it would be of the order of < 0.5 seconds, the previous transfers in the log took around 0.2 seconds

Code: Select all

06:23:37.051 SFTP: Starting Upload of /home/geoff/CumulusMX/web/realtimegaugesT.txttmp
... then
06:23:49.156 Realtime FTP[15]: Error during update: Sharing violation on path /home/geoff/CumulusMX/web/realtimegaugesT.txttmp

At which point it disconnects the SSH connection and reconnects. But this does not abort the previous file transfer of realtimegaugesT.txttmp, which in turn leaves the file locked, seemingly forever, despite the SSH connection having a timeout of 30 seconds. It appears disconnecting the session before the timeout occurs causes the transfer to hang rather abort.

New release required...