Page 1 of 2

Automating the upload of dayfile.txt

Posted: Thu 10 Mar 2011 3:41 pm
by daj
I've written a little article in the Wiki on how to automatically upload dayfile.txt to your website once per day.

It's a simple method, using the Windows scheduler task. I have tested it on Windows XP and Windows 7

Enjoy

http://wiki.sandaysoft.com/a/Upload_Dayfile

Re: Automating the upload of dayfile.txt

Posted: Thu 10 Mar 2011 3:57 pm
by Gina
That looks very useful, thank you :) I must get round to improving my Cumulus web pages - it's all very basic ATM.

Re: Automating the upload of dayfile.txt

Posted: Thu 10 Mar 2011 9:14 pm
by Super-T
GMTA David
I was doing exactly the same thing yesterday but ran into a problem.
I could get all the way to the PUT command, the file is created on the server but then it just sits there showing 150 Ok to send data. ctrl C will abort and it will exit.
I think something is blocking at the server as the file has been created but with 0 content. I haven't got back to it yet to find the problem but any suggestions would be gratefully received. :-)

Re: Automating the upload of dayfile.txt

Posted: Thu 10 Mar 2011 10:20 pm
by mcrossley
Terry, do you need to upload 'put' to a temporary filename, 'delete' dayfile.txt, 'rename' temp to dayfile?

Cumulus uses two different mechanisms for its FTPs because of variations in server support for rename/overwrite etc

ultimate fall back would be 'delete', 'put' I guess.

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 1:14 am
by Super-T
Hi Mark
My Cumulus settings are "Auto update" & "Use FTP Rename" ticked.

Delete and put sounds good....i'll give it a try.

OK....tried adding delete although that probably wasn't a problem because I had tried removing the file from the server before running the script.
Partial result:-
230 Login successful.
ftp> debug
Debugging On .
ftp> lcd c:\!Downloads
Local directory now C:\!Downloads.
ftp> quote PASV
---> PASV
227 Entering Passive Mode (202,174,112,121,240,209)
ftp> binary
---> TYPE I
200 Switching to Binary mode.
ftp> cd www/weather/data
---> CWD www/weather/data
250 Directory successfully changed.
ftp> delete www/weather/data/test.txt
---> DELE www/weather/data/test.txt
550 Delete operation failed.
ftp> put c:\!Downloads\test.txt
---> PORT 10,1,1,200,216,157
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.

Hangs on that last line and ctrl C to get out of it.
Creates the file but no content.

Also ran the delete line initially as delete test.txt as I was already in the correct directory, but with no joy.


Later in the day. My Host says there is a problem with PASV mode not supported in Windows FTP programme. http://support.microsoft.com/kb/271078/en-us. I've also tried "MoveMeFreely FTP programme that does support PASV but still no joy.

It looks like there is a problem when the host runs PASV mode which most probably do.

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 7:49 am
by tjaliwalpa
daj wrote:I've written a little article in the Wiki on how to automatically upload dayfile.txt to your website once per day.

It's a simple method, using the Windows scheduler task. I have tested it on Windows XP and Windows 7

Enjoy

http://wiki.sandaysoft.com/a/Upload_Dayfile
Thanks for this information. It is just what I was looking for. I have a little problem with it however.

I've set up the script and batch file as per the wiki instructions. I have checked and rechecked the variables and am sure I have them correct.

The dayfile.txt is deleted on the server and replaced with another version which is empty. The ftp process times out without uploading the complete file.

What am I doing wrong?

Attached is a screen shot of the cmd.

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 9:13 am
by daj
Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 9:31 am
by Super-T
Should be doing LCD e:\cumulus\data perhaps but if you are using put e:\cumulus\data\dayfile.txt then you don't need to do the lcd e:\cumulus\data I think?

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 9:50 am
by tjaliwalpa
daj wrote:Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed
I see now. There is no directory as my weather site is in the root directory of the webspace. In other words, the Cumulus directory is blank.

I've changed this line to cd / with the following result:

ftp> cd /
250 OK. Current directory is /
ftp> put e:\cumulus\data\dayfile.txt
200 PORT command successful
425 Could not open data connection to port 52978: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

I have no idea where to go next.

Edit:
I tried leaving out the cd / command from the ftp script and found it was not necessary.

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 9:56 am
by tjaliwalpa
tjaliwalpa wrote:
daj wrote:Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed
I see now. There is no directory as my weather site is in the root directory of the webspace. In other words, the Cumulus directory is blank.

I've changed this line to cd / with the following result:

ftp> cd /
250 OK. Current directory is /
ftp> put e:\cumulus\data\dayfile.txt
200 PORT command successful
425 Could not open data connection to port 52978: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

I have no idea where to go next.

Bob
Ah :oops: Got it now :oops:
:bash: :bash: :bash: :bash:

It is a firewall problem at my end with PC Tools Internet Security. I shall have to write a rule to allow port 52978 access.

When I disable the firewall, everything works perfectly.

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 10:02 am
by Super-T
I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
Your error may be specific to your host as I don't get that problem.

My result:- This is using "moveItFreely" ftp prog.
C:\WeatherAnalysis>ftps -s:c:\weatheranalysis\daily.ftp linux11.digiweb.net.nz
220 (vsFTPd 2.0.5)
Connected to linux11.digiweb.net.nz.
User: 331 Please specify the password.
Password: 230 Login successful.
ftp> Debugging On .
ftp> Local directory now C:\WeatherAnalysis.
ftp> ---> PASV
227 Entering Passive Mode (202,174,112,121,226,207)
ftp> ---> TYPE I
200 Switching to Binary mode.
ftp> ---> CWD www/weather/data
250 Directory successfully changed.
ftp> ---> PORT 10,1,1,200,192,164
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.
ftp: 0 bytes sent in 5.07Seconds 0.00Kbytes/sec.
ftp> ---> QUIT
221 Goodbye.

Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 10:10 am
by tjaliwalpa
Super-T wrote:I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
Your error may be specific to your host as I don't get that problem.

My result:- This is using "moveItFreely" ftp prog.
C:\WeatherAnalysis>ftps -s:c:\weatheranalysis\daily.ftp linux11.digiweb.net.nz
220 (vsFTPd 2.0.5)
Connected to linux11.digiweb.net.nz.
User: 331 Please specify the password.
Password: 230 Login successful.
ftp> Debugging On .
ftp> Local directory now C:\WeatherAnalysis.
ftp> ---> PASV
227 Entering Passive Mode (202,174,112,121,226,207)
ftp> ---> TYPE I
200 Switching to Binary mode.
ftp> ---> CWD www/weather/data
250 Directory successfully changed.
ftp> ---> PORT 10,1,1,200,192,164
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.
ftp: 0 bytes sent in 5.07Seconds 0.00Kbytes/sec.
ftp> ---> QUIT
221 Goodbye.

Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us
Actually, my issue very similar. When I have the firewall enabled, to dayfile.txt is deleted and created as a blank file. The process hangs for a time and the replacement file is not transferred. Eventually, it gives up and the following is displayed:

425 Could not open data connection to port 53496: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

The port number is not always the same, so I can't figure out the rule to create.

Edit:
Don't know if I've done the right thing, but if I allow Outbound and Inbound for the the File Transfer Program in my Firewall, everything works beautifully.

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 12:13 pm
by mcrossley
Super-T wrote:I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
...
200 PORT command successful. Consider using PASV.
...
Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us
So the MS FTP does not support PASV mode, you may have to use something like WinSCP - it's nice to have it around anyway!

Or as Bob has done, open up your firewall for two way sessions with the FTP client (computer and possibly router).

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 12:56 pm
by daj
Yeah, the MS FTP tool is a very simple implementation, but it is one that everyone has on their system so is a good starting point.

Some useful reading in this topic, so I will add a link from the Wiki article to this for other who may have problems.

Re: Automating the upload of dayfile.txt

Posted: Fri 11 Mar 2011 7:52 pm
by Super-T
Bob
I did try turning off firewall at one time but your experience means I will re-visit that today.
My Host support seemed to think it was all to do with them requiring PASV as that is meant to avoid the firewall problem but in my case, there is something else involved.