few issues with cumulus and cumulus toolbox

[ace2]

Been running cumulus for over a year without issue, supplying my website data, ssgauges realtime.txt , weather underground and a XML for a weather app using my data.
Then I installed cumulus toolbox a few weeks ago, now my XML file doesn't always upload, ss gauges realtimegauges.txt(realtime) currupts on site and now cumulus is not processing/sending weather underground data.
I've had to disable the XML to make realtimegauges.txt work and when it corrupts I have to manually delete it off site!!!
Has another had this happen with cumulus toolbox????

[water01]

I am doing all of the above with no problems at all. Toolbox is uploading every minute with my near realtime data and Cumulus is doing the rest.

Only thing I can think of is that you must make sure that the two programs are not uploading the same files, otherwise they could tramp all over each other and cause corruption problems.

[ace2]

I thought about that, but toolbox is only uploading every 10 mins and major at 12am, strange it doesn't stuff up all the time....

[ace2]

Well it wasn't toolbox, its cumulus, oh well, changed the station offline limit with gauges and added a 5 min update scripts to take care of the realtime.txt

[steve]

its cumulus

I'm not aware of any issues with processing or FTP of custom templates, or with WU updates. But you've clearly gathered evidence to allow you to say definitively that "it's Cumulus". If you would like to show me that evidence, I can take a look at it and hopefully fix the bugs that you've found.

[ace2]

Mm, where do I start, I can turn on ftp logging, but it only happens sometimes!
When it does happen, it either refuses to upload only the realtime.Txt or it uploads a corrupted 0kb one That can't be overwritten. The only way to get it working again is to delete the file on web, then it fine.
Cumulus is processing the file as the temp one is in directory.

I do have some ftp logs that show a upload error, but all my other pages work fine, no upload errors at all.

I then thought is was due to my scripts uploading via ftp. That way I have a script that uploads the file every 5 mins.

[steve]

I must have misunderstood; as you said "it's Cumulus", I thought you had proof that it was a bug in Cumulus, rather than the much more likely situation that it's a problem with your ftp server or your router/firewall, which is where 99.9% of the problems with ftp occur; particularly as you say it was working for you over a year. It's so long since the Cumulus FTP code changed that I can't even remember when it was.

[hills]

I use to experience strange issues uploading files when I had several different apps all trying to upload at the same time. It turned out my ISP only allows a set number of ftp connections at once. I've stopped all but Cumulus and the toolbox from uploading and it hasn't happened since.

[ace2]

I thought that might have been the case Phil, so I changed my schedule:
Cumulus every 30 mins on the dot, several files + images
Cam photo upload every 10 mins offset starting 12:03, 12:13 etc
Major video upload 11:35pm

This still caused the realtime.txt to corrupt every now and again, never any other file.

So I then set the cam shot and video upload to also upload the temp realtime.txt thinking those might have been effecting it. But it still corrupted that one file.

So its either my system or a cumulus bug

[steve]

I thought that might have been the case Phil, so I changed my schedule:
Cumulus every 30 mins on the dot, several files + images
Cam photo upload every 10 mins offset starting 12:03, 12:13 etc
Major video upload 11:35pm

But are you not also uploading 'realtime' files at a more frequent interval?

This still caused the realtime.txt to corrupt every now and again, never any other file.

You don't seem to be uploading realtime.txt at all - http://www.users.on.net/~ace2/realtime.txt gives a 404

Screen shots of your Cumulus internet settings would be useful. I really can't follow what your setup looks like at all. And it's impossible to even guess at what or where the problem might be without the ftp log.

[ace2]

My bad, realtimegauges.txt
I haven't had any issues before with this, been running thecss gauges for many months without issue.

Its no big deal really. I'll just get cumulus to process the file for my script to upload every 5 mins.

[water01]

I think, if you don't mind me saying so, you have missed the point.

Steve is saying that given what he has requested he would like to investigate whether or not it is a problem. Since you have the logs that have the error in them and presumably you can post pictures of your Internet and Files windows it would be worth doing just in case.

Bugs (if it is a bug) are usually difficult to find, since they only happen in certain circumstances that are created by a particular situation, especially in software such as Cumulus that is in such wide use, so it would be worth letting him take a look.

[ace2]

Sorry, am at work typing on a mobile phone between customers, I can upload my ftp logs and post a happy snap of settings if it'll help.

[ace2]

Ok, posting some shots of my cumulus setting.
have enabled ftp logging and will post that later, i'll give it a a while to see if it plays up

station setting


internet setting


upload files


Have to see if the upload starts playing up again, i shall leave ftp logging on....

[steve]

So you are uploading realtimegauges.txt every 30 seconds, so it will clash with your other uploads from other software. By 'clash' I mean occur at the same time, which shouldn't be a problem, but as Phil has found, with some ftp servers it might be.

Using ftp rename is a good idea, if your ftp server supports it (which it should).

As the only files you are having problems with are XML and JSON, and given that Cumulus doesn't know or care what the content of a file is, it might be worth setting binary mode for those files, even though they are text. This stops the server doing CRLF/LF conversion on the contents of the file, which it doesn't need to do anyway for those types of file. It may stop the corruption if the cause is the server doing that conversion.