Trouble Uploading NOAA Reports

[SJG1976]

I had a similar problem several years ago. My problem turned out to be that I was using "Reports" with a capital "R" as the folder name in CUMX as the FTP folder, instead of the actual folder name on my web site which was "reports" with a small "r". If you go to your web site using this link: https://cm2weather.co.uk/reports/NOAAMO0122.txt you will see your NOAAMO0122.txt in the web folder "reports" (small "r"). My set up is: public_html/Reports (www also works) so renaming your web site folder and and using public_html/ or www/ before the "Reports" web folder name should work for you after renaming your web site folder.

Bump

That's my Live Site and yes that one is set to use the Reports folder with a "a lower case l"
The test Server I'm using is structured with an Uppercase R

so would be

http://localhost/Reports/NOAAM00122.txt

In-fact if I browse to that from where I have manually copied it works fine.
So I'm sure permissions are set-up fine.

Or do you think I should just change the Directory Stucture and FTP to use both lowercase l?

[SJG1976]

I'll update the WebServer to be /reports and change MX to upload to /reports

Will let you know how it goes.

[kevinTDWX]

I'll update the WebServer to be /reports and change MX to upload to /reports

Will let you know how it goes.

Sorry...I misunderstood you were testing a server. I'm not sure going lower case is going to matter in that case.

[SJG1976]

I'll see if it make a difference.
The Live one as you say point to /reports.

I just set the test one to point to /Reports as in the MX Menu for NOAA that was the default.
Will see what happens tonight.

[PaulMy]

I am sure you know, but whatever you put in NOAA settings FTP Options > FTP folder: must exist on your webserver - i.e. if you use reports in the settings then reports folder must be on your site.

If the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:

Enjoy,
Paul

[SJG1976]

I am sure you know, but whatever you put in NOAA settings FTP Options > FTP folder: must exist on your webserver - i.e. if you use reports in the settings then reports folder must be on your site.

If the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:

Enjoy,
Paul

Hi Paul,

That's the odd thing...
I have it set up on the Live Site as /reports and the directory as /reports.

on the Test site i have it as /Reports and the Directory as /Reports.

All the other bits work (like my realtime.txt m,y vars.php and some other customer extra ones. It just seems like the NOAA reports.
If i manually FTP tot he Server it works, so I can only think it's a Directory issue on the Test Server.

I've now changed this to all be lower case (same as the Live Server) as that was the only difference I could see.

Thanks to everyone for all the suggestions

[SJG1976]

Here is the URL for the Test Server

Testserver_noaa.php.PNG

Here is the the Directory Path on the test Server

noaa_dir.PNG

And this is now how I have MX Configured to Upload.

main_ftp.PNG

Everything to me seem to be in order and is now the same as the live MX Install going to the Live Server

[SJG1976]

Still No update for me again last night.

Code: Select all

2022-02-05 00:00:01.001 WLL Broadcast: {"did":"001D0A71606E","ts":1644019201,"conditions":[{"lsid":484096,"data_structure_type":1,"txid":1,"wind_speed_last":1.00,"wind_dir_last":217,"rain_size":2,"rain_rate_last":0,"rain_15_min":0,"rain_60_min":0,"rain_24_hr":27,"rain_storm":27,"rain_storm_start_at":1643963881,"rainfall_daily":0,"rainfall_monthly":27,"rainfall_year":70,"wind_speed_hi_last_10_min":11.00,"wind_dir_at_hi_speed_last_10_min":319}]}
2022-02-05 00:00:01.059 Done creating graph data files
2022-02-05 00:00:01.060 SFTP[Int]: Connecting using password authentication
2022-02-05 00:00:01.060 SFTP[Int]: CumulusMX Connecting to 192.168.0.3 on port 22
2022-02-05 00:00:01.089 SFTP[56]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\web\cumulusT.xmltmp
2022-02-05 00:00:01.089 SFTP[56]: Renaming /var/www/html/data/cumulus.xmltmp to /var/www/html/data/cumulus.xml
2022-02-05 00:00:01.103 SFTP[56]: Renamed /var/www/html/data/cumulus.xmltmp
2022-02-05 00:00:01.103 SFTP[56]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\web\cumulusT.xmltmp to /var/www/html/data/cumulus.xml
2022-02-05 00:00:01.103 Realtime[56]: Uploading extra web file[1] C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.php
2022-02-05 00:00:01.103 SFTP[56]: Uploading C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.phptmp
2022-02-05 00:00:01.164 SFTP[56]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp
2022-02-05 00:00:01.164 SFTP[56]: Renaming /var/www/html/data/vars.phptmp to /var/www/html/data/vars.php
2022-02-05 00:00:01.182 SFTP[56]: Renamed /var/www/html/data/vars.phptmp
2022-02-05 00:00:01.182 SFTP[56]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.php
2022-02-05 00:00:01.182 Realtime[56]: End cycle
2022-02-05 00:00:02.236 SFTP[Int]: Uploading NOAA reports
2022-02-05 00:00:02.236 SFTP[Int]: Uploading Reports\NOAAMO0222.txt to /reports/NOAAMO0222.txttmp
2022-02-05 00:00:02.258 SFTP[Int]: Error uploading Reports\NOAAMO0222.txt to /reports/NOAAMO0222.txttmp : No such file
2022-02-05 00:00:02.261 SFTP[Int]: The SFTP object is disposed - skipping upload of Reports\NOAAYR2022.txt

I think I'll try the Full Remote Server path name and see if that makes any difference, but I thought the NOAA Directory was relative to the FTP location?

[freddie]

Did you try out the suggestion from @PaulMy (repeated below)?

If the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:

If this method also fails to upload then it is likely a permissions problem on the remote reports directory.

[SJG1976]

Did you try out the suggestion from @PaulMy (repeated below)?

If the files exist locally in CumulusMX\Reports then a quick way to test is to add it to your Extra web files for FTP only and check your server after the CMX interval FTP. Once that is uploading then you will know what settings for NOAA File folder:

If this method also fails to upload then it is likely a permissions problem on the remote reports directory.

Sorry Paul,

Missed the last bit.
I've just tried that.

Code: Select all

2022-02-05 10:46:06.381 SFTP[80]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\web\varsT.phptmp to /var/www/html/data/vars.php
2022-02-05 10:46:06.381 Realtime[80]: Uploading extra web file[2] C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txt
2022-02-05 10:46:06.381 SFTP[80]: Uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txttmp
2022-02-05 10:46:06.397 SFTP[80]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp
2022-02-05 10:46:06.398 SFTP[80]: Renaming /var/www/html/reports/NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txt
2022-02-05 10:46:06.405 SFTP[80]: Renamed /var/www/html/reports/NOAAYR2022.txttmp
2022-02-05 10:46:06.405 SFTP[80]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAYR2022.txttmp to /var/www/html/reports/NOAAYR2022.txt
2022-02-05 10:46:06.405 Realtime[80]: Uploading extra web file[3] C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txt
2022-02-05 10:46:06.405 SFTP[80]: Uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txttmp
2022-02-05 10:46:06.414 SFTP[80]: Uploaded C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp
2022-02-05 10:46:06.415 SFTP[80]: Renaming /var/www/html/reports/NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txt
2022-02-05 10:46:06.421 SFTP[80]: Renamed /var/www/html/reports/NOAAMO0222.txttmp
2022-02-05 10:46:06.421 SFTP[80]: Completed uploading C:\Users\sjg19\Desktop\WLL Testing\Reports\NOAAMO0222.txttmp to /var/www/html/reports/NOAAMO0222.txt
2022-02-05 10:46:06.421 Realtime[80]: End cycle

So that works fine.
So I'm guessing it's not a Permissions Issue?

It's just not processing the file on the rollover.

I removed the Files from the Web Server as well, just to make sure they did Actually write the data, and it was fine.

[flort]

Just looking at this and where it's worked successfully using the CMX interval FTP it is copying NOAAMO0222.txttmp to NOAAMO0222.txt.

On the other hand at rollover it is trying to copy NOAAMO0222.txt to NOAAMO0222.txttmp which is why it's not finding the file. One would think it is a configuration error causing this.

Regards,
Trevor

[SJG1976]

Just looking at this and where it's worked successfully using the CMX interval FTP it is copying NOAAMO0222.txttmp to NOAAMO0222.txt.

On the other hand at rollover it is trying to copy NOAAMO0222.txt to NOAAMO0222.txttmp which is why it's not finding the file. One would think it is a configuration error causing this.

Regards,
Trevor

Yes I'm not sure.
I thought it picked up the txttmp when it was set to rename and upload.
I'm not sure however...

[freddie]

Just looking at this and where it's worked successfully using the CMX interval FTP it is copying NOAAMO0222.txttmp to NOAAMO0222.txt.

On the other hand at rollover it is trying to copy NOAAMO0222.txt to NOAAMO0222.txttmp which is why it's not finding the file. One would think it is a configuration error causing this.

Regards,
Trevor

Yes I'm not sure.
I thought it picked up the txttmp when it was set to rename and upload.
I'm not sure however...

It should create the txttmp file, then transfer the data, then rename the file to whatever you specify in your interface. Sounds to me as if it is failing at the first step for some reason - and that is generally down to permissions. But MX can transfer the file through extra files, so that isn't the problem here. It must be due to the path specification in the interface.

EDIT: have you tried looking in the log file on your server? That will show you where the MX login is trying to create the file to and probably the reason why it can't.

EDIT 2: looking at the messages in your log from the MX SFTP client, it appears to be trying to copy the file to the following location:

Code: Select all

/reports/NOAAMO0222.txttmp

.. which isn't under /var/www/html

[SJG1976]

I'm going to try a few things.

I've added the Full Path to the MX NOAA Seetings rather than having it as /reports (I'm sure i've alread tried this however)

On the Test Server I'm also going to create a new folder /reports (in the root of the Raspberry Pi) and see if anything goes here.

Failing that I'm really not sure where it's going wrong.
Thanks to everyone who has made suggestions.

[mcrossley]

I've tried to read all of this, but may have missed the reply to the request to upload to "reports" - no leading slash?

The path is relative to your FTP root folder. It depends on the FTP server software what effect a leading slash has. Some will treat it as an absolute path so it will refer to the root of your filesystem. Other servers will take it to mean your FTP root folder. So it is best not to use a leading slash.

Where are the FTP root folders on your test and live servers?

When you tested adding to extra files it looks like you used a full absolute path from the root for remote?

Also when you used Extra files you must have ticked "Process" as it was trying to upload a local file with "tmp" on the end. The NOAA reports do not contain any web tags and should not be processed.