Page 2 of 3
Re: Trouble Uploading NOAA Reports
Posted: Fri 04 Feb 2022 11:37 am
by SJG1976
kevinTDWX wrote: ↑Fri 04 Feb 2022 11:09 am
kevinTDWX wrote: ↑Fri 04 Feb 2022 11:04 am
I had a similar problem several years ago. My problem turned out to be that I was using "Reports" with a capital "R" as the folder name in CUMX as the FTP folder, instead of the actual folder name on my web site which was "reports" with a small "r". If you go to your web site using this link:
https://cm2weather.co.uk/reports/NOAAMO0122.txt you will see your NOAAMO0122.txt in the web folder "reports" (small "r"). My set up is: public_html/Reports (www also works) so renaming your web site folder and and using public_html/ or www/ before the "Reports" web folder name should work for you after renaming your web site folder.
Bump
That's my Live Site and yes that one is set to use the Reports folder with a "a lower case l"
The test Server I'm using is structured with an Uppercase R
so would be
http://localhost/Reports/NOAAM00122.txt
In-fact if I browse to that from where I have manually copied it works fine.
So I'm sure permissions are set-up fine.
Or do you think I should just change the Directory Stucture and FTP to use both lowercase l?
Re: Trouble Uploading NOAA Reports
Posted: Fri 04 Feb 2022 11:38 am
by SJG1976
I'll update the WebServer to be /reports and change MX to upload to /reports
Will let you know how it goes.
Re: Trouble Uploading NOAA Reports
Posted: Fri 04 Feb 2022 12:40 pm
by kevinTDWX
SJG1976 wrote: ↑Fri 04 Feb 2022 11:38 am
I'll update the WebServer to be /reports and change MX to upload to /reports
Will let you know how it goes.
Sorry...I misunderstood you were testing a server. I'm not sure going lower case is going to matter in that case.
Re: Trouble Uploading NOAA Reports
Posted: Fri 04 Feb 2022 1:22 pm
by SJG1976
I'll see if it make a difference.
The Live one as you say point to /reports.
I just set the test one to point to /Reports as in the MX Menu for NOAA that was the default.
Will see what happens tonight.
Re: Trouble Uploading NOAA Reports
Posted: Fri 04 Feb 2022 2:51 pm
by PaulMy
I am sure you know, but whatever you put in NOAA settings FTP Options > FTP folder: must exist on your webserver - i.e. if you use reports in the settings then reports folder must be on your site.
If the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:
Enjoy,
Paul
Re: Trouble Uploading NOAA Reports
Posted: Fri 04 Feb 2022 5:55 pm
by SJG1976
PaulMy wrote: ↑Fri 04 Feb 2022 2:51 pm
I am sure you know, but whatever you put in NOAA settings FTP Options > FTP folder: must exist on your webserver - i.e. if you use reports in the settings then reports folder must be on your site.
If the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:
Enjoy,
Paul
Hi Paul,
That's the odd thing...
I have it set up on the Live Site as /reports and the directory as /reports.
on the Test site i have it as /Reports and the Directory as /Reports.
All the other bits work (like my realtime.txt m,y vars.php and some other customer extra ones. It just seems like the NOAA reports.
If i manually FTP tot he Server it works, so I can only think it's a Directory issue on the Test Server.
I've now changed this to all be lower case (same as the Live Server) as that was the only difference I could see.
Thanks to everyone for all the suggestions
Re: Trouble Uploading NOAA Reports
Posted: Fri 04 Feb 2022 6:08 pm
by SJG1976
Here is the URL for the Test Server
Testserver_noaa.php.PNG
Here is the the Directory Path on the test Server
noaa_dir.PNG
And this is now how I have MX Configured to Upload.
main_ftp.PNG
Everything to me seem to be in order and is now the same as the live MX Install going to the Live Server
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 9:33 am
by SJG1976
Still No update for me again last night.
Code: Select all
2022-02-05 00:00:01.001 WLL Broadcast: {"did":"001D0A71606E","ts":1644019201,"conditions":[{"lsid":484096,"data_structure_type":1,"txid":1,"wind_speed_last":1.00,"wind_dir_last":217,"rain_size":2,"rain_rate_last":0,"rain_15_min":0,"rain_60_min":0,"rain_24_hr":27,"rain_storm":27,"rain_storm_start_at":1643963881,"rainfall_daily":0,"rainfall_monthly":27,"rainfall_year":70,"wind_speed_hi_last_10_min":11.00,"wind_dir_at_hi_speed_last_10_min":319}]}
2022-02-05 00:00:01.059 Done creating graph data files
2022-02-05 00:00:01.060 SFTP[Int]: Connecting using password authentication
2022-02-05 00:00:01.060 SFTP[Int]: CumulusMX Connecting to 192.168.0.3 on port 22
2022-02-05 00:00:01.089 SFTP[56]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\web\cumulusT.xmltmp
2022-02-05 00:00:01.089 SFTP[56]: Renaming /var/www/html/data/cumulus.xmltmp to /var/www/html/data/cumulus.xml
2022-02-05 00:00:01.103 SFTP[56]: Renamed /var/www/html/data/cumulus.xmltmp
2022-02-05 00:00:01.103 SFTP[56]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\web\cumulusT.xmltmp to /var/www/html/data/cumulus.xml
2022-02-05 00:00:01.103 Realtime[56]: Uploading extra web file[1] C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.php
2022-02-05 00:00:01.103 SFTP[56]: Uploading C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.phptmp
2022-02-05 00:00:01.164 SFTP[56]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp
2022-02-05 00:00:01.164 SFTP[56]: Renaming /var/www/html/data/vars.phptmp to /var/www/html/data/vars.php
2022-02-05 00:00:01.182 SFTP[56]: Renamed /var/www/html/data/vars.phptmp
2022-02-05 00:00:01.182 SFTP[56]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.php
2022-02-05 00:00:01.182 Realtime[56]: End cycle
2022-02-05 00:00:02.236 SFTP[Int]: Uploading NOAA reports
2022-02-05 00:00:02.236 SFTP[Int]: Uploading Reports\NOAAMO0222.txt to /reports/NOAAMO0222.txttmp
2022-02-05 00:00:02.258 SFTP[Int]: Error uploading Reports\NOAAMO0222.txt to /reports/NOAAMO0222.txttmp : No such file
2022-02-05 00:00:02.261 SFTP[Int]: The SFTP object is disposed - skipping upload of Reports\NOAAYR2022.txt
I think I'll try the Full Remote Server path name and see if that makes any difference, but I thought the NOAA Directory was relative to the FTP location?
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 9:47 am
by freddie
Did you try out the suggestion from @PaulMy (repeated below)?
PaulMy wrote: ↑Fri 04 Feb 2022 2:51 pmIf the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:
If this method also fails to upload then it is likely a permissions problem on the remote reports directory.
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 10:48 am
by SJG1976
freddie wrote: ↑Sat 05 Feb 2022 9:47 am
Did you try out the suggestion from @PaulMy (repeated below)?
PaulMy wrote: ↑Fri 04 Feb 2022 2:51 pmIf the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:
If this method also fails to upload then it is likely a permissions problem on the remote reports directory.
Sorry Paul,
Missed the last bit.
I've just tried that.
Code: Select all
2022-02-05 10:46:06.381 SFTP[80]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.php
2022-02-05 10:46:06.381 Realtime[80]: Uploading extra web file[2] C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txt
2022-02-05 10:46:06.381 SFTP[80]: Uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txttmp
2022-02-05 10:46:06.397 SFTP[80]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp
2022-02-05 10:46:06.398 SFTP[80]: Renaming /var/www/html/reports/NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txt
2022-02-05 10:46:06.405 SFTP[80]: Renamed /var/www/html/reports/NOAAYR2022.txttmp
2022-02-05 10:46:06.405 SFTP[80]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txt
2022-02-05 10:46:06.405 Realtime[80]: Uploading extra web file[3] C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txt
2022-02-05 10:46:06.405 SFTP[80]: Uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txttmp
2022-02-05 10:46:06.414 SFTP[80]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp
2022-02-05 10:46:06.415 SFTP[80]: Renaming /var/www/html/reports/NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txt
2022-02-05 10:46:06.421 SFTP[80]: Renamed /var/www/html/reports/NOAAMO0222.txttmp
2022-02-05 10:46:06.421 SFTP[80]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txt
2022-02-05 10:46:06.421 Realtime[80]: End cycle
So that works fine.
So I'm guessing it's not a Permissions Issue?
It's just not processing the file on the rollover.
I removed the Files from the Web Server as well, just to make sure they did Actually write the data, and it was fine.
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 11:05 am
by flort
Just looking at this and where it's worked successfully using the CMX interval FTP it is copying NOAAMO0222.txttmp to NOAAMO0222.txt.
On the other hand at rollover it is trying to copy NOAAMO0222.txt to NOAAMO0222.txttmp which is why it's not finding the file. One would think it is a configuration error causing this.
Regards,
Trevor
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 11:36 am
by SJG1976
flort wrote: ↑Sat 05 Feb 2022 11:05 am
Just looking at this and where it's worked successfully using the CMX interval FTP it is copying NOAAMO0222.txttmp to NOAAMO0222.txt.
On the other hand at rollover it is trying to copy NOAAMO0222.txt to NOAAMO0222.txttmp which is why it's not finding the file. One would think it is a configuration error causing this.
Regards,
Trevor
Yes I'm not sure.
I thought it picked up the txttmp when it was set to rename and upload.
I'm not sure however...
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 11:42 am
by freddie
SJG1976 wrote: ↑Sat 05 Feb 2022 11:36 am
flort wrote: ↑Sat 05 Feb 2022 11:05 am
Just looking at this and where it's worked successfully using the CMX interval FTP it is copying NOAAMO0222.txttmp to NOAAMO0222.txt.
On the other hand at rollover it is trying to copy NOAAMO0222.txt to NOAAMO0222.txttmp which is why it's not finding the file. One would think it is a configuration error causing this.
Regards,
Trevor
Yes I'm not sure.
I thought it picked up the txttmp when it was set to rename and upload.
I'm not sure however...
It should create the txttmp file, then transfer the data, then rename the file to whatever you specify in your interface. Sounds to me as if it is failing at the first step for some reason - and that is generally down to permissions. But MX can transfer the file through extra files, so that isn't the problem here. It must be due to the path specification in the interface.
EDIT: have you tried looking in the log file on your server? That will show you where the MX login is trying to create the file to and probably the reason why it can't.
EDIT 2: looking at the messages in your log from the MX SFTP client, it appears to be trying to copy the file to the following location:
.. which isn't under /var/www/html
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 12:07 pm
by SJG1976
I'm going to try a few things.
I've added the Full Path to the MX NOAA Seetings rather than having it as /reports (I'm sure i've alread tried this however)
On the Test Server I'm also going to create a new folder /reports (in the root of the Raspberry Pi) and see if anything goes here.
Failing that I'm really not sure where it's going wrong.
Thanks to everyone who has made suggestions.
Re: Trouble Uploading NOAA Reports
Posted: Sat 05 Feb 2022 2:17 pm
by mcrossley
I've tried to read all of this, but may have missed the reply to the request to upload to "reports" - no leading slash?
The path is relative to your FTP root folder. It depends on the FTP server software what effect a leading slash has. Some will treat it as an absolute path so it will refer to the root of your filesystem. Other servers will take it to mean your FTP root folder. So it is best not to use a leading slash.
Where are the FTP root folders on your test and live servers?
When you tested adding to extra files it looks like you used a full absolute path from the root for remote?
Also when you used Extra files you must have ticked "Process" as it was trying to upload a local file with "tmp" on the end. The NOAA reports do not contain any web tags and should not be processed.