NOAA reports-Reports

[HansR]

I have some issues:

My local NOAA directory is Reports.
My remote NOAA directory is reports.

With the most recent version suddenly CMX tries to move the daily NOAA report to the remote directory Reports.

1)
That is strange as I definitely did not change that setting. It also disturbs the subsequent uploads for the FTP frequency.
I also noticed that the moon image was switched on for generation and FTP copy while I was sure I switched that off.
So some settings seem to be reset to their default.
Are you aware of any change in CMX here?

2)
If I look in the logfile I notice the Daily Graph files are send every minute (my logging frequency) and not every FTP frequency (10 minutes). Is that correct?
NOTE: this frequency changes when DEBUG is switched on (look for Uploading daily graph data files from bottom up)

3)
FTP now! does not seem to transfer the NOAA files (after I corrected the destination)

Code: Select all

2021-09-18 00:00:56.584 SFTP[Int]: Error uploading Reports/NOAAMO0921.txt to /.../public_html/Reports/NOAAMO0921.txt : No such file
2021-09-18 00:00:56.603 SFTP[Int]: The SFTP object is disposed - skipping upload of Reports/NOAAYR2021.txt
2021-09-18 00:00:56.603 SFTP[Int]: The SFTP object is disposed - skipping upload of web/HWActueelT.txttmp
2021-09-18 00:00:56.603 SFTP[Int]: The SFTP object is disposed - skipping upload of web/HWActueelT.txttmp
2021-09-18 00:00:56.603 SFTP[Int]: The SFTP object is disposed - skipping upload of web/websitedata.json
2021-09-18 00:00:56.604 SFTP[Int]: The SFTP object is disposed - skipping upload of web/tempdata.json
2021-09-18 00:00:56.604 SFTP[Int]: The SFTP object is disposed - skipping upload of web/pressdata.json
2021-09-18 00:00:56.604 SFTP[Int]: The SFTP object is disposed - skipping upload of web/winddata.json
2021-09-18 00:00:56.604 SFTP[Int]: The SFTP object is disposed - skipping upload of web/wdirdata.json
2021-09-18 00:00:56.604 SFTP[Int]: The SFTP object is disposed - skipping upload of web/humdata.json
2021-09-18 00:00:56.604 SFTP[Int]: The SFTP object is disposed - skipping upload of web/raindata.json
2021-09-18 00:00:56.604 SFTP[Int]: The SFTP object is disposed - skipping upload of web/dailyrain.json
2021-09-18 00:00:56.605 SFTP[Int]: The SFTP object is disposed - skipping upload of web/dailytemp.json
2021-09-18 00:00:56.605 SFTP[Int]: The SFTP object is disposed - skipping upload of web/solardata.json
2021-09-18 00:00:56.605 SFTP[Int]: The SFTP object is disposed - skipping upload of web/sunhours.json
2021-09-18 00:00:56.605 SFTP[Int]: Uploading daily graph data files
2021-09-18 00:00:56.605 SFTP[Int]: The SFTP object is disposed - skipping upload of web/alldailytempdata.json
2021-09-18 00:00:56.605 SFTP[Int]: The SFTP object is disposed - skipping upload of web/alldailypressdata.json
2021-09-18 00:00:56.605 SFTP[Int]: The SFTP object is disposed - skipping upload of web/alldailywinddata.json
2021-09-18 00:00:56.605 SFTP[Int]: The SFTP object is disposed - skipping upload of web/alldailyhumdata.json
2021-09-18 00:00:56.606 SFTP[Int]: The SFTP object is disposed - skipping upload of web/alldailyraindata.json
2021-09-18 00:00:56.606 SFTP[Int]: The SFTP object is disposed - skipping upload of web/alldailysolardata.json
2021-09-18 00:00:56.606 SFTP[Int]: Done uploading daily graph data files
2021-09-18 00:00:56.606 SFTP[Int]: Uploading Moon image file
2021-09-18 00:00:56.606 SFTP[Int]: The SFTP object is disposed - skipping upload of web/moon.png
2021-09-18 00:00:56.606 SFTP[Int]: Done uploading Moon image file
2021-09-18 00:01:01.867 DoLogFile: Writing log entry for 18-9-2021 00:01:00
2021-09-18 00:01:01.868 DoLogFile: log entry for 18-9-2021 00:01:00 written

[Mapantz]

I'm sure FTP Now! doesn't upload the NOAA reports?! You'll have to wait until the rollover time, unless you do it manually.

[HansR]

I remarked this here some time ago and Mark said the FTP now! would now also upload the NOAA reports. Maybe I confuse things with something else.

[mcrossley]

Hi Hans,

Regarding FTP Now!, I did add the upload of the daily graph files to this a while back, but not the NOAA reports.

Re your settings changing, the NOAA settings are not changed automatically either in the CMX code, or the settings page JS code. I tried to reproduce your issue and cannot get my FTP folder to change.

Your Daily graph files are failing to upload because the FTP process is aborting due to the NOAA "Reports" folder not being found.

This needs fixing.
You can see in your posted log extract "/public_html/Reports/NOAAMO0921.txt : No such file" and the SFTP dies (nice!).
So all the subsequent uploads fail.

Only once a daily graph has been uploaded successfully is the upload required flag cleared. As the files are skipped, the upload is re-attempted at each period until success.

[mcrossley]

If you switch on FTP logging you should have seen the SFTP failures

[HansR]

Hi Hans,

Regarding FTP Now!, I did add the upload of the daily graph files to this a while back, but not the NOAA reports.

Yes that was it

Re your settings changing, the NOAA settings are not changed automatically either in the CMX code, or the settings page JS code. I tried to reproduce your issue and cannot get my FTP folder to change.

So we have a riddle. I did not change the NOAA setting for the directory.

Your Daily graph files are failing to upload because the FTP process is aborting due to the NOAA "Reports" folder not being found.

This needs fixing.
You can see in your posted log extract "/public_html/Reports/NOAAMO0921.txt : No such file" and the SFTP dies (nice!).
So all the subsequent uploads fail.

Yes, i said daily graphs in the confusion but also the normal FTP transfer fails on the minute frequency. It should not be on the minute frequency. And it really is if you look in the non-debug part of the log.

Only once a daily graph has been uploaded successfully is the upload required flag cleared. As the files are skipped, the upload is re-attempted at each period until success.

At the beginning the daily log is transferred, at 9h05 cmx is restarted. Why is it trying the upload anyway? The daily has been uploaded at rollover! (00h00)

[HansR]

If you switch on FTP logging you should have seen the SFTP failures

Yes, but the normal log tells me too.

[mcrossley]

At the beginning the daily log is transferred, at 9h05 cmx is restarted. Why is it trying the upload anyway? The daily has been uploaded at rollover! (00h00)

It doesn't actually try to transfer them at all...

• Each section of code (apart from NOAA**) that checks each file type has the "Uploading xxx", "Done uploading xxx" log entries around it.
• Each file has an independent FTP required flag - this allows MX to only transfer files that are actually required. And for example if an upload of daily files fails mid-point, the files that were transferred successfully are not sent again at the next interval.
• If none of the files have the upload flag set, then nothing happens inside that block.

I hope that is a clear explanation?

** the NOAA reports have a global FTP required flag for both files, so if that is not set, the section does not execute at all and no messages are logged.

[HansR]

@mcrossley: ok that explains it.