Cumulus Support

I know there have already been several posts about error 103 and 32. I have read them, but I am still struggling. I am getting these errors very regularly. The vast majority of the time it is on the extraLog file, but everyone once in a while it is on the monthly file as well. I checked all of the usual suspects. I made sure to tell my anti-virus to leave the cumulus folder alone. I cross referenced the back-up times and they don't match and there is only one back up per day but very many errors per day. I told windows not to index the content of the files. I was concerned that perhaps if the system bogged down maybe cumulus couldn't open, write, save and close in within the one min interval before it tried again, so I changed the interval to 5 min. But I am still getting the same problem. I have no clue where else to look for the issue. I started using cumulus earlier this month because of the awesome customizability of the web/database stuff. This has been ongoing since I started using it and I have been trying to figure it out ever since. To complicate matters, I am running cumulus on two separate computers, and they are both having issues with the same file. Sometimes the times of the errors match between the two computers but often they don't, and computer one by far gets more of them than computer two. The computers do not talk to each other.

Here are the latest errors from each computer since I cleared the error log around 6pm:

Computer 1
11/20/2014 6:25:00 PM : 11/20/2014 6:25:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 6:25:00 PM : 11/20/2014 6:25:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 6:25:00 PM : 11/20/2014 6:25:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 6:45:00 PM : 11/20/2014 6:45:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 6:45:00 PM : 11/20/2014 6:45:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 6:45:00 PM : 11/20/2014 6:45:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 7:05:00 PM : 11/20/2014 7:05:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 7:05:00 PM : 11/20/2014 7:05:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 7:05:00 PM : 11/20/2014 7:05:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 7:25:00 PM : 11/20/2014 7:25:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 7:25:00 PM : 11/20/2014 7:25:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 7:25:00 PM : 11/20/2014 7:25:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 7:45:00 PM : 11/20/2014 7:45:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 7:45:00 PM : 11/20/2014 7:45:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 7:45:00 PM : 11/20/2014 7:45:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 8:05:00 PM : 11/20/2014 8:05:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 8:05:00 PM : 11/20/2014 8:05:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 8:05:00 PM : 11/20/2014 8:05:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103

Computer 2
11/20/2014 6:25:00 PM : 11/20/2014 6:25:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 6:25:00 PM : 11/20/2014 6:25:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 6:25:00 PM : 11/20/2014 6:25:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 6:45:00 PM : 11/20/2014 6:45:00 PM Error writing data to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 32
11/20/2014 6:45:00 PM : 11/20/2014 6:45:00 PM Error writing EOL to data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103
11/20/2014 6:45:00 PM : 11/20/2014 6:45:00 PM Error closing data file: C:\Cumulus\data\ExtraLog201411.txt. I/O error 103

Has anyone encountered anything like this (even after checking virus scans/backups/indexing/etc)? Can anyone think of anything I may be missing?

Brandon

A permissions problem possibly? Is Cumulus running as admin or a user with permissions to write to the folders?

I/O Error 32 is specifically a file sharing violation, i.e. another process is accessing the file at the time that Cumulus wants to write to it. The causes of this could be:

1. Anti-virus or similar software accessing the file (this is the cause in almost all cases that I've seen).
2. Some other software that a user is running which is processing the file in some way.
3. More than one instance of Cumulus running.
4. An issue with the system clock on the PC.
5. A bug in Cumulus where it doesn't close the file down.

If it's the last one, I don't know what the bug is, particularly regarding the extra log file. Cumulus only accesses the file to write the current data to it, at the interval specified, when the system clock changes so that the current minute indicates that it is time to write a new entry. It opens the file, writes to it, and closes it, all in the same procedure. If it was unable to close the file for whatever reason, the first error logged would be "Error closing data file...".

Microsoft have a utility available which will tell you (amongst other things) which process has a file open - http://technet.microsoft.com/en-gb/sysi ... 96653.aspx - you might be able to get some useful information from that.

If you zip up the diags folder and attach it, I'll see if I can see anything unusual.

uncle_bob wrote:A permissions problem possibly? Is Cumulus running as admin or a user with permissions to write to the folders?

Both computers are running cumulus on an administrator account, but not explicitly "run as administrator." Just to rule it out, I did shut down cumulus and opened it back up using "run as administrator" I will monitor it and post whether or not it makes a difference.

steve wrote:I/O Error 32 is specifically a file sharing violation, i.e. another process is accessing the file at the time that Cumulus wants to write to it. The causes of this could be:

1. Anti-virus or similar software accessing the file (this is the cause in almost all cases that I've seen).
2. Some other software that a user is running which is processing the file in some way.
3. More than one instance of Cumulus running.
4. An issue with the system clock on the PC.
5. A bug in Cumulus where it doesn't close the file down.

1. seems to be ruled out
2. this is a mystery, possible though
3. only one instance of cumulus is running so this is ruled out
4. I'm not having any issues with the clock.
5. possible

steve wrote:Microsoft have a utility available which will tell you (amongst other things) which process has a file open - http://technet.microsoft.com/en-gb/sysi ... 96653.aspx - you might be able to get some useful information from that.

I have already used this tool but did not come up with anything interesting or useful.

steve wrote:If you zip up the diags folder and attach it, I'll see if I can see anything unusual.

I have attached the diags folder for the computer that has been having the error more often.

Thanks for your help!

Brandon wrote:uncle_bob wrote:
A permissions problem possibly? Is Cumulus running as admin or a user with permissions to write to the folders?

Both computers are running cumulus on an administrator account, but not explicitly "run as administrator." Just to rule it out, I did shut down cumulus and opened it back up using "run as administrator" I will monitor it and post whether or not it makes a difference.

The errors are still occurring even after explicitly running as administrator.

I can't see anything odd in the logs apart from this error, and a lot of ftp errors (which may or may not be significant). You're getting the error every 20 minutes, and perhaps significantly, you are apparently getting them at the same times on both machines.

Can you confirm that the entries corresponding to those times that you posted earlier are not in ExtraLog201411.txt?

I've just noticed this in one of your previous posts:

I am currently uploading realtime, dayfile, monthly, and extra

What do you mean by that, exactly?

steve wrote:I can't see anything odd in the logs apart from this error, and a lot of ftp errors (which may or may not be significant). You're getting the error every 20 minutes, and perhaps significantly, you are apparently getting them at the same times on both machines.

Can you confirm that the entries corresponding to those times that you posted earlier are not in ExtraLog201411.txt?

The FTP errors don't end up being an issue because as long as the data makes it to the log, it will all catch up the next time it sends to the server. The error writing to the log is a much bigger issue though because it will never be there then. I have confirmed that the entries are not there for those times. Here is the time span that corresponds with the errors I posted above for computer one:

(truncated for ease of viewing)
20/11/14,18:20,38.0,42.0,39.0,...
20/11/14,18:30,38.0,42.0,39.0,...
20/11/14,18:35,38.0,42.0,39.0,...
20/11/14,18:40,38.0,42.0,39.0,...
20/11/14,18:50,38.0,42.0,39.0,...
20/11/14,18:55,38.0,42.0,39.0,...
20/11/14,19:00,38.0,42.0,39.0,...
20/11/14,19:10,38.0,42.0,39.0,...
20/11/14,19:15,38.0,42.0,39.0,...
20/11/14,19:20,38.0,42.0,39.0,...
20/11/14,19:30,38.0,42.0,39.0,...
20/11/14,19:35,38.0,42.0,39.0,...
20/11/14,19:40,38.0,42.0,39.0,...
20/11/14,19:50,38.0,42.0,39.0,...
20/11/14,19:55,38.0,42.0,39.0,...
20/11/14,20:00,38.0,42.0,39.0,...
20/11/14,20:10,38.0,42.0,39.0,...

steve wrote:I've just noticed this in one of your previous posts:

I am currently uploading realtime, dayfile, monthly, and extra

What do you mean by that, exactly?

I have cumulus set to upload those 4 logs via ftp. The month file, dayfile, and extraLog file are sent up using the configuration found here: configuration->internet->files. And the realtime is sent automagically because that box is check under web settings. Once it is on the server, I am using various php files and cron jobs to handle all of the data mostly for insertion into a mySQL database. I'm not sure exactly what you are asking, or if I properly answered your question. Let me know if I can provide more information.

Brandon wrote:I have cumulus set to upload those 4 logs via ftp

Ah, right. Might have been worth mentioning that

That's the cause of the problem, particularly as you are having ftp problems. The ftp code runs in a separate thread so as not to interfere with operations like updating the screen - or writing to the logs. If the ftp thread has the log file open when the main thread needs to open it, it will fail. I'm sorry, but there's nothing I can do about that.

steve wrote:Ah, right. Might have been worth mentioning that

That's the cause of the problem, particularly as you are having ftp problems. The ftp code runs in a separate thread so as not to interfere with operations like updating the screen - or writing to the logs. If the ftp thread has the log file open when the main thread needs to open it, it will fail. I'm sorry, but there's nothing I can do about that.

Sorry for over looking that detail in my description of the problem. In retrospect, I can see that is a very important bit of knowledge. I just assumed that since it was cumulus doing the sending not an outside program that was all just handled. It makes perfect sense though, and explains why the error's happen at the exact same time on both computers sometimes (because they both have the same settings) and then sometimes they don't error because the thread must give it up in time but that is unpredictable. It also make sense why there would be more errors on computer one because it has some more processor intensive things going on that computer two doesn't have.

So now that the mystery has been solved... do others who use the ftp feature run into this? Is there a scheduling "sweet spot" to reduce this? Before, I had everything set to one minute so it would be a conflict much of the time. For the past 2 days I have web settings set to 3 min, realtime at 240 seconds and the log interval at 5 min. So it makes sense that when it is erring out that it is happening every 20 minutes because 4 min and 5 min should meet up every 20 minutes. Is there any way to offset them (ie have both run every two minutes, but on a one minute offset so each minute one or the other is going but not both?)

Thanks again for all of your help.

Brandon wrote:do others who use the ftp feature run into this?

Very few people upload their data logs to their web site, as far as I know. Those who operate SQL databases tend to feed them using the current data from realtime.txt, or via a 'custom' file generated using web tags. Apart from avoiding the problem of interfering with the writing of the logs, it's also much more efficient, as the same data isn't being uploaded over and over.

If you wanted to continue uploading the entire log each time, you could mitigate the synchronising effect by choosing an ftp interval that isn't a factor of 60. There would likely still be a clash from time to time, particularly with the ftp problems you are having, as this is likely to be causing the file to be held open for extended periods. And of course towards the end of the month, as the file gets bigger, it will take longer to upload.

steve wrote:Those who operate SQL databases tend to feed them using the current data from realtime.txt, or via a 'custom' file generated using web tags. Apart from avoiding the problem of interfering with the writing of the logs, it's also much more efficient, as the same data isn't being uploaded over and over.

The main reason I did not rely on the realtime file for sql uploading is because I require the extra sensors, and I was not yet aware of the tags. I did end up finding the tags with your help. The tags are a very awesome feature and I am using them for a realtime warning system via twilio. By time I discovered the tags I had the other system already set up. I considered reconfiguring it, but I kind of started to like the idea that my server could be down for a period of time and as soon as it comes back up all of the missing rows will be written because the whole log will be there TTBOMK that is not possible with realtime or tags.

I might end up just using the tags for all of the uploading, or maybe I will write a small app that makes a quick copy then shoots the copy up to the server independent of cumulus but on an exact offset from cumulus. The copy should be a lot quicker than an upload so it would be less likely to interfere as long as the offset is done right. Regardless, I wouldn't have any of these options without knowing what the problem is so thank you for helping me sort it out.

Brandon

You could try increasing the interval between uploads; I suspect your uploads are taking a very long time and clashing with the next log update. It occurs to me that Cumulus writes to the log files before it initiates the ftp, so I think the problem may be occurring because the previous upload is still in progress.

One point to note - It's not clear to me whether you are uploading the files with the 'normal' uploads or the 'realtime' uploads, given the intervals you've quoted. The 'normal' uploads are synchronised with the logging (assuming the ftp interval is a factor of 60), but the logging is done before the ftp. The 'realtime' uploads operate from a separate timer, so there is always the possibility of a clash. So I think you could mitigate or eliminate the problem by using the 'normal' upload interval, and setting the ftp and logging intervals large enough to allow the ftp to complete within the selected interval.

Or use Cumulus to generate the files locally and upload them with Cumulus Toolbox which I what I do.

Sorry for the delay in my response, I've been buried in work so I haven't had a chance to work on this project for a couple of days.

steve wrote:One point to note - It's not clear to me whether you are uploading the files with the 'normal' uploads or the 'realtime' uploads, given the intervals you've quoted. The 'normal' uploads are synchronised with the logging (assuming the ftp interval is a factor of 60), but the logging is done before the ftp. The 'realtime' uploads operate from a separate timer, so there is always the possibility of a clash. So I think you could mitigate or eliminate the problem by using the 'normal' upload interval, and setting the ftp and logging intervals large enough to allow the ftp to complete within the selected interval.

I'm not sure how to answer the question exactly because I'm not sure. So I will just mention how I have it set up. Under configuration there is an option for FTP logging I have check this a few times, but it seems to uncheck itself (or possibly gets uncheck when closing and opening cumulus?). Anyway, I have not noticed a difference in any uploads or errors whether it is checked or not so I am not sure what it does. Under web settings in the internet configuration site/option tab I have the following boxes checked: Auto Update, Use FTP Rename, UTF-8 encode, Interval 3 minutes, Enable Realtime, Enable Realtime FTP, Realtime Interval 240 seconds. Then under the internet configuration files tab, it is set to sent the month file, the day file, and the extra log file. For each of those the Realtime, FTP, and UTF-8 boxes are checked.

water01 wrote:Or use Cumulus to generate the files locally and upload them with Cumulus Toolbox which I what I do.

It doesn't seem like this would eliminate the problem because I will still have the error if there is a slow upload. That is, unless the Cumulus tool box makes a copy then uploads the copy like I mentioned I might do with a custom application. Does it do that? If not, I'm not really seeing how it would benefit to do that rather than the way it currently is.

I appreciate all of the help/advice on the matter!