Page 1 of 1

Issues with short intervals for Realtime.txt and FTP user ri

Posted: Mon 26 Aug 2013 12:28 am
by Warrandyte
disclaimer.
I am in the bush (country), and so only have 3G mobile internet - which is unreliable at best (even using a 12db external antenna!)

however when i set the realtime.txt to anything under the approximate time it takes to ftp the regular file upload, i get issues.

common issues are - already connected / disconnected, halts before standard files and images get updated, (i.e. custom pages are updated but not the index.htm) miscellaneous ftp errors and virtual memory issues (from sessions being held open)

I timed the standard upload including all the extra files (0 through xx in cumulus ini - currently 14 files) and got just under 3 minutes from start to finish - primarily because upload speeds on mobile (and indeed other internet connections) are usually much slower than download speeds

at 2 minutes (120 sec) realtime.txt intervals I get occasional failures

at 1 minute it fails almost reliably after the first 6-8 files

at 10 seconds, the laptop has virtual memory issues because it opens new sessions before another has shut down etc.

if I have a really short settings (5 seconds) and the ftp user for realtime.txt has only single session privileges then any upload fails.

I am having to use the ftp <root> user and password with unlimited sessions to do any automated uploads, which means nothing else can be logged in.

(cumulus toolbox operates on its own ftp user account to get around the timing issues also)

Anyone else experiencing these sorts of issues?

We dont have access to ADSL - not even basic as the exchange only has 2 cans and one string, dialup, satellite or cable internet, despite being only 10 km out of town, so sadly those are not solutions.

is there a way to use a sepearate account for the realtime.txt ftp?
Does the toolbox also handle standard (non PHP) pages - I tried this with no success)

Re: Issues with short intervals for Realtime.txt and FTP use

Posted: Mon 26 Aug 2013 7:37 am
by steve
Cumulus uses a separate thread for realtime uploads, so they wouldn't normally affect 'normal' uploads (assuming no other limitations or issues with the transfers). It also checks whether there is already an upload in progress each time, and doesn't attempt to start a new upload, so it's normally self limiting.

So the problems you are seeing are presumably limitations with your ftp server and/or your connection to it.

Cumulus currently only allows one set of ftp details to be entered, so realtime and 'normal' use the same user, and separate connections. If that doesn't work for you, I think your only option is to get Cumulus to generate the files and use something like Fling to do the uploads.

It seems to me that from the way you describe your internet connection that perhaps you are expecting too much of it to be doing 'realtime' uploads and should stick to 'static' pages uploaded over a longer interval. Your weather station data only updates every 50 seconds, after all.

Re: Issues with short intervals for Realtime.txt and FTP use

Posted: Mon 26 Aug 2013 8:00 am
by gluepack
With my prior experience of that type of environment, over a three to four year period prior to the setup I currently have, Steve's last sentence is the pertinent one. And, I don't think switching to WinSCP or alternative will help you.

I see that you are transmitting 14 files. I assume, perhaps incorrectly, that you are submitting files (e.g. website content, like gauges) generated from the Cumulus data, as opposed to just raw data. In that case, perhaps a website re-design is in order so that content is generated at your server, so you transmit fewer and smaller files.

Re: Issues with short intervals for Realtime.txt and FTP use

Posted: Mon 26 Aug 2013 1:53 pm
by Warrandyte
Thank you both. Yes I suspect my web hosting and internet combo.
I am seeing a high number of drop outs so there is an issue there.

I saw the article on FO and 50 seconds a short while after so will change that.
Im more interested in realtime over internet and keeping tje rest locally served so I may well do that.

Now if only there was a webserver component in Cumulus.... that would open up a whole can od worms with isps and port blocking. But an internak server for the lan/ office would be good.

Did anything ever happen with the sql version 2?

Re: Issues with short intervals for Realtime.txt and FTP use

Posted: Mon 26 Aug 2013 3:39 pm
by steve
Warrandyte wrote:Did anything ever happen with the sql version 2?
Do you mean Cumulus 2? See https://cumulus.hosiene.co.uk/viewtopic.php?f=17&t=5427

The version of Cumulus I'm currently working on has a web server built in. But as I've said elsewhere in the forum, progress is slow as I'm doing very little Cumulus development at the moment.

Re: Issues with short intervals for Realtime.txt and FTP use

Posted: Mon 26 Aug 2013 11:41 pm
by Warrandyte
Thanks Steve, keep up the great work it says Im not authorised to view that link which is probably why Ive not seen it.
Ill have to send more fun tickets your way when im a little more flush, as this product is one of the best supported donation wares around.
Of course thanks to all the others such as Mark Crossley and all (sorry for not recalling all of the names) who seem to spend hours on here supporting us, and answering dumb questions.

There is so much content to read here that sometimes topics get lost in the numerous search results.

when I am more up to speed Id be happy to help maintain some of the faqs and so on.

Re: Issues with short intervals for Realtime.txt and FTP use

Posted: Tue 27 Aug 2013 7:32 am
by steve
Warrandyte wrote:Thanks Steve, keep up the great work it says Im not authorised to view that link which is probably why Ive not seen it.
There was a problem with the permissions on the Cumulus 2 section of the forum - now fixed.