Generating Files Without Uploading

[radilly]

I am looking to generate the essential files for upload - using an "External program". The "Program" script is working well. I seem to be having trouble getting realtimegauges.txt to generate ... without attempting to FTP it.

It seems like "Enable realtime FTP" is required to generate the file, but that seems to also attempt to FTP it - though it is getting updated. In the MXdiags log I see:

Code: Select all

2021-01-08 19:45:13.949 Completed writing Cumulus.ini file
2021-01-08 19:45:29.205 Realtime[133]: Error reconnecting ftp server - Connection refused
2021-01-08 19:45:29.216 Realtime[133]: Uploading - realtimegauges.txt
2021-01-08 19:45:29.219 FTP[133]: Error uploading web/realtimegauges.txt to realtimegauges.txt : Connection refused
2021-01-08 19:45:45.204 Realtime[134]: Error reconnecting ftp server - Connection refused
2021-01-08 19:45:45.213 Realtime[134]: Uploading - realtimegauges.txt
2021-01-08 19:45:45.216 FTP[134]: Error uploading web/realtimegauges.txt to realtimegauges.txt : Connection refused
2021-01-08 19:46:01.202 Realtime[135]: Error reconnecting ftp server - Connection refused
2021-01-08 19:46:01.207 Realtime[135]: Uploading - realtimegauges.txt
2021-01-08 19:46:01.209 FTP[135]: Error uploading web/realtimegauges.txt to realtimegauges.txt : Connection refused
2021-01-08 19:46:17.205 Realtime[136]: Error reconnecting ftp server - Connection refused

So this is working after a fashion, but clogging up the log. To be clear, I removed the FTP information at the top of the "Internet settings" because I'm not using that feature. When I untick "Enable realtime FTP" the file does not update

Is there a way to generate realtimegauges.txt without the attempt to upload it?

In case it's of interest, I'm using rsync via ssh, which seems to be very fast and efficient in my application. I run the same rsync for realtime, but it assesses the minimum set of files to be updated, and just updates those, i.e. 1 file.

[radilly]

I'm a little unclear on how the options in this section interact. This is a stab at documenting how things are configured. Best I can tell "Enable realtime FTP" is required to generate the file, but it also wants to FTP it.

[PaulMy]

I may be out in left field on this, but couldn't you just add realtimegaugesT.txt in Extra web files, local side as \web\realtimegaugesT.txt, and on the remote side enter the path to where you want it xxxxxx/realtimegauges.txt or is it xxxxxx\realtimegauges.txt, tick Process, and Realtime if that is what you want, and that's it (do not tick FTP).

Enjoy,
Paul

[water01]

My guess if I wrote the code if FTP is not set that would be my first check and after that I would check for FTP logging, since it is illogical to log FTP messages if FTP is switched off. Hence nothing would be logged.

[HansR]

Is there a way to generate realtimegauges.txt without the attempt to upload it?

I would switch off Realtimegauges.txt FTP in the Web/FTP settings.
Then I would create the realtimegaugesT.txt from the webtags and put that in the Extra web files section without FTP.

Are you trying to find loopholes? I mean, what do you want to achieve with creating the realtimegauges and not upload it? Just asking.

[water01]

what do you want to achieve with creating the realtimegauges and not upload it? Just asking.

Running your own local web server would be my guess!

[HansR]

@water01: If you have a local webserver, just configure your local ftp and you're set.
There are several CumulsUtils users who run Apache on the RPi where Cumulus runs. Ftp is not always standard configured beside but if you do then you simply ftp from cumulus to your local host. Problem solved. Keep your firewall in the modem closed for FTP, only open 80 for the Apache.

I still don't see why you would want to generate the gaugesrealtime and not ftp it.

[mcrossley]

I would switch off Realtimegauges.txt FTP in the Web/FTP settings.
Then I would create the realtimegaugesT.txt from the webtags and put that in the Extra web files section without FTP.

This.

The realtimegaugesT.txt file is already supplied, unlike the realtime.txt which is generated internally by MX, the gauges file is a normal template file that is processed to allow the gauges code to the updated independently of MX.

[water01]

I still don't see why you would want to generate the gaugesrealtime and not ftp it.

Because you want to generate the files on a local filesystem without using FTP which is what Cumulus of any flavour was always designed to do via the Extra Web files (hence why there are 99 of them).

[HansR]

I would switch off Realtimegauges.txt FTP in the Web/FTP settings.
Then I would create the realtimegaugesT.txt from the webtags and put that in the Extra web files section without FTP.

This.

The realtimegaugesT.txt file is already supplied, unlike the realtime.txt which is generated internally by MX, the gauges file is a normal template file that is processed to allow the gauges code to the updated independently of MX.

Aha...
Never knew that, never looked for it either...
How about inventing the wheel

Because you want to generate the files on a local filesystem without using FTP which is what Cumulus of any flavour was always designed to do via the Extra Web files (hence why there are 99 of them).

There are worlds I have never been to...

[HansR]

Check

[radilly]

Thanks folks, for all the comments. I need to read them carefully and respond thoughtfully. For one thing, I have an interest in sharing what I'm learning - perhaps in the Wiki - but I'm still trying to cement my understanding of CMX in some areas.

I can expand this in the original post, but I promoted my experiment with rsync under CMX yesterday to handle the real file transfers. I suspect my hosting environment does not handle FTP as well as some other protocols based on my experience. I use SCP (password-less) for images and other web files I generate with only occasional burps. I was seeing a steady stream of FTP errors - like for an hour +/- every few hours for several days. I have seen this in the past, and observed that restarting CMX, or sometimes rebooting the Pi cleaned it up. I've never been able to find the cause. Seemed like it was worth trying rsync via (password-less) SSH. That mechanism is working great by the way, and seems wicked fast.


Mark points to a concept that I think some of the posts may have overlooked...

The realtimegaugesT.txt file is already supplied, unlike the realtime.txt which is generated internally by MX, the gauges file is a normal template file that is processed to allow the gauges code to the updated independently of MX.

From looking at the External programs (Dashboard --> Settings --> Internet Settings --> External programs) there are functionally 3 independent timers - 2 of which you have some control over in the "Web/FTP settings" section above it:

• Upload interval (configurable)
• Realtime interval (configurable)
• Daily ... (not configurable? - the boundary for the "meteorological day")

I believe these map to the 3 sections of the External programs section:

CMX_External_Programs_66.jpg

I hope framing the interaction around these "timers" this way is helpful to others.

While the "Extra web files" does have checkboxes for both "Process" and "Realtime" (which appear to be independent), I've not experimented with using them and External programs together. One would expect, I think, that the latter runs after processing all of these extra web files. (And realtimegaugesT.txt is out there, though I've never needed to alter it.)

In part, the engineer in me wants to minimize extras, but also wants to understand how to leverage the 5 realtime settings just above the External programs section. Seems like one ought to be able with 5 settings to generate, but not FTP. Maybe I've not found the right combination (and testing is further complicated by some settings ... I suspect ... requiring a restart to take effect). Still working through the combos.

[radilly]

I may be out in left field on this, but couldn't you just add realtimegaugesT.txt in Extra web files, local side as \web\realtimegaugesT.txt, and on the remote side enter the path to where you want it xxxxxx/realtimegauges.txt or is it xxxxxx\realtimegauges.txt, tick Process, and Realtime if that is what you want, and that's it (do not tick FTP).

Paul, and others...

YES! This does work! The only thing, which makes sense as I think about it from my messing around, is that one has to tick / check Settings --> Internet settings --> Web/FTP settings --> Enable Realtime (separate settings page). It didn't seem like it was getting processed at all without that (even on the Update interval timer ... which is what the Process tick alone does).

I took a quick look around the Wiki for something on realtime processing. I didn't find anything that treats that subject well for CMX. That's an "opportunity".

Interestingly there's comparatively little being written to the MXdiags log with this configuration relative to other things I had partly working. I prefer less l info written to the log so that abnormal conditions of concern are easier to spot.

Thanks all! I'm humbled by your collective knowledge!

[radilly]

My guess if I wrote the code if FTP is not set that would be my first check and after that I would check for FTP logging, since it is illogical to log FTP messages if FTP is switched off. Hence nothing would be logged.

Well, I guess that was my bad ... I had it in my head it would be written to the MXdiags directory ... since it is a diagnostic. I found it using find with -mmin ... a nice way on Linux to find all files recently written to... the "root" CumulusMX directory.

To me there is some complexity (or a lack of clarity on my part) around properly using the FTP/SFTP options. I don't find a checkbox (or boxes) to allow direct enablement or disablement of FTP, or SFTP. It feels to me like CMX has to sort of deduce my intentions. So it could be useful to turn on FTP logging independently if only to explain how CMX is interpreting the combination of fields filled in at the top of the Internet settings page.

Thanks!

[radilly]

I would switch off Realtimegauges.txt FTP in the Web/FTP settings.
Then I would create the realtimegaugesT.txt from the webtags and put that in the Extra web files section without FTP.

As above, that looks like the best solution. In reflecting on what I was thinking, I suppose I was thinking that checking "Enable Realtime" would generate realtimegauges.txt. I hadn't noticed that the Extra web files page has a column for REALTIME. That provides a lot more flexibility. I also tended to avoid that facility because I was getting a number of .txttmp as an artifact, though that appears to have stopped with the last build.

Are you trying to find loopholes? I mean, what do you want to achieve with creating the realtimegauges and not upload it? Just asking.

Fair question! For several reasons:

• Periodically I was getting lots of FTP errors. From a number of threads I've gotten the impression that support of FTP varies by hosting provider. I gather SFTP is provided because of that (but I could never get it to work).
• I resist storing passwords as clear text as a rule. I've removed it from all my own code using password-less SCP instead.

I'm post some detail on this later, but I've found rsync (via ssh) is extremely efficient and fast. I have some screen shots illustrating how quick it seems to be, but none from when I was using the built in FTP. I may put that back just to document the difference, but from my recollection the difference is quite significant. I may do that in a separate thread so others can leverage the approach if they wish.

Thanks!!