Cumulus Support

I have been getting errors in MXdiag about a failure to upload realtimegaugesT.txttmp to realtimegauges,txt. It doesn't happen every time, but approximately every other time. I have tried to manually FTP the file every few seconds and it never fails so I am wondering if there is sometimes a problem with the way MX uses the FTP connection. Could you take a look please.

MXdiag and ftplog attached.

daveh wrote:I have been getting errors in MXdiag about a failure to upload realtimegaugesT.txttmp to realtimegauges,txt. It doesn't happen every time, but approximately every other time. I have tried to manually FTP the file every few seconds and it never fails so I am wondering if there is sometimes a problem with the way MX uses the FTP connection. Could you take a look please.

MXdiag and ftplog attached.

Hi daveh,

You need to upload the realtimegaugesT.txt file not the realtimegaugesT.txttmp file. The latter file is a temporary file hence "tmp" and is produced when Cumulus process an upload of the other file.

Cumulus doesn't actually do anything with the ftp connection, it knows nothing about ftp at all. It's all handled with a third party component. It looks like the connection keeps getting dropped - you seem to be using a very long realtime interval - 10 minutes? Perhaps the connection being idle for so long means that it is being timed out. Cumulus 1 has an option to disconnect and reconnect the realtime connection after N uploads, but I haven't yet implemented that in MX. In your case, setting it to 1 would probably be a good idea when I get around to doing it.

Grimers wrote:You need to upload the realtimegaugesT.txt file not the realtimegaugesT.txttmp file. The latter file is a temporary file hence "tmp" and is produced when Cumulus process an upload of the other file.

It's how it works. Please stop making up answers - this is part of what got you banned last time.

steve wrote:Cumulus doesn't actually do anything with the ftp connection, it knows nothing about ftp at all. It's all handled with a third party component. It looks like the connection keeps getting dropped - you seem to be using a very long realtime interval - 10 minutes? Perhaps the connection being idle for so long means that it is being timed out. Cumulus 1 has an option to disconnect and reconnect the realtime connection after N uploads, but I haven't yet implemented that in MX. In your case, setting it to 1 would probably be a good idea when I get around to doing it.

Thanks Steve. The realtime interval is set to 10 minutes. I want the gauges to work but since I only record data from my sensors every 15 minutes it seemed pointless to update too frequently. However, out of interest I will try reducing that time and see if the errors go away.

steve wrote:

Grimers wrote:You need to upload the realtimegaugesT.txt file not the realtimegaugesT.txttmp file. The latter file is a temporary file hence "tmp" and is produced when Cumulus process an upload of the other file.

It's how it works. Please stop making up answers - this is part of what got you banned last time.

Ok, I understand that. But, it looked to me like he was trying to upload the txttmp file not the actual txt file. I am not making up any answers, I am stating a fact.

Grimers wrote:it looked to me like he was trying to upload the txttmp file not the actual txt file. I am not making up any answers, I am stating a fact.

He has correctly configured Cumulus to process the realtimegaugesT.txt file. It's the contents of the realtimegaugesT.txttmp file which gets uploaded, not the realtimegaugesT.txt file. The latter file has unprocessed web tags in it, and the gauges need actual data, so if that file was uploaded as you suggest, the gauges wouldn't work.

steve wrote:

Grimers wrote:it looked to me like he was trying to upload the txttmp file not the actual txt file. I am not making up any answers, I am stating a fact.

He has correctly configured Cumulus to process the realtimegaugesT.txt file. It's the contents of the realtimegaugesT.txttmp file which gets uploaded, not the realtimegaugesT.txt file. The latter file has unprocessed web tags in it, and the gauges need actual data, so if that file was uploaded as you suggest, the gauges wouldn't work.

I understand, sorry. I thought he was trying to upload the tmp file from his local machine which wouldn't work as Cumulus wouldn't have found any web tags in the file as it had already been processed.

Grimers wrote:

steve wrote:

Grimers wrote:it looked to me like he was trying to upload the txttmp file not the actual txt file. I am not making up any answers, I am stating a fact.

He has correctly configured Cumulus to process the realtimegaugesT.txt file. It's the contents of the realtimegaugesT.txttmp file which gets uploaded, not the realtimegaugesT.txt file. The latter file has unprocessed web tags in it, and the gauges need actual data, so if that file was uploaded as you suggest, the gauges wouldn't work.

I understand, sorry. I thought he was trying to upload the tmp file from his local machine which wouldn't work as Cumulus wouldn't have found any web tags in the file as it had already been processed.

I don't think you do understand. The tmp file is the result of Cumulus processing the txt file and replacing the web tags with real data - which is precisely what is required for display on the web site. So, yes, he DOES want to upload the tmp file.

freddie wrote: I don't think you do understand. The tmp file is the result of Cumulus processing the txt file and replacing the web tags with real data - which is precisely what is required for display on the web site. So, yes, he DOES want to upload the tmp file.

Right, I'm not going on. But, I certainly do not upload the temp file, I upload the txt file so that Cumulus can process the web tags inside the file...

Grimers wrote:I upload the txt file so that Cumulus can process the web tags inside the file...

The processing is done before it is uploaded to your web space, once uploaded it is out of reach of Cumulus which is local on your machine - not your web space!

steve wrote:It looks like the connection keeps getting dropped - you seem to be using a very long realtime interval - 10 minutes? Perhaps the connection being idle for so long means that it is being timed out. Cumulus 1 has an option to disconnect and reconnect the realtime connection after N uploads, but I haven't yet implemented that in MX. In your case, setting it to 1 would probably be a good idea when I get around to doing it.

Steve, I reduced the realtime update to every 5 minutes and have only seen 4 errors in the last 24 hours. These were all consecutive attempts so may have been a temporary problem with connection to my hosting. I think you are right about the connection timing out with it set at 10 minutes.

Dave

I guess the Cumulus doesn't send NOOP 'keep-alive' packets periodically to keep the connection open. FTP clients such as Filezilla send them every 30 seconds by default.

Looking at the documentation, the client supposedly does a Poll every 15 seconds on an idle connection. But it's not clear to me whether this is supposed to stop the server timing out the connection, or just to allow the client to detect that the connection has been closed by the server.

But specifying long realtime intervals (several minutes) was not really what the realtime facility was intended for, which is why it keeps the connection open. If you're going to do that, then it seems to me that you may as well just upload it with the other files (i.e. don't use realtime at all). At some point I'll look at implementing the closing of the connection that Cumulus 1 does, I'll see how hard this will be to do in MX - I don't imagine it will be difficult.

steve wrote:But specifying long realtime intervals (several minutes) was not really what the realtime facility was intended for, which is why it keeps the connection open. If you're going to do that, then it seems to me that you may as well just upload it with the other files (i.e. don't use realtime at all).

I am so pleased you wrote that . I just had one of those "light bulb" moments when you realise you've gone down the wrong track. Being completely new to Cumulus I just blindly followed the instructions at http://wiki.sandaysoft.com/a/Simple_Website_setup and set up the realtimegauges.txt as it describes. That combined with the name of "REALTIME"gauges just led me down the realtime route. Of course, I now see that all I had to do was untick realtime on the extra web files settings page and let the file upload with all the others. Thanks Steve I'm much happier now.