Nearly 3 weeks of data lost in log file

[RayProudfoot]

Steve,

I was going through some of the select-a-graphs for July and noticed a straight line joing two lots of data. On closer inspection of my log file for July the 5-minute entries end at 21:20 on 7 July and only start again at 23:36 on 26 July. It would appear I've lost nearly 3 weeks of data.

Can you shed any light on how this could have happened and secondly, is there any way of retrieving it? I am also running WeatherLink and that appears unaffected.

I notice when I restart Cumulus the downloading data box doesn't show any activity and disappears after a few seconds. I'm guessing that's linked to the TimeStamp date/time in today.ini. If I set that back to 7 July is there any chance of recovering the lost data?

[steve]

It's hard to see how Cumulus could have been running all that time without logging anything. Presumably the data was there at some point, you would have noticed if it wasn't? What does you dayfile.txt look like? A corresponding gap?

Could you have done anything that might have caused Cumulus to log to a different location? You could try searching your computer for jul09log.txt.

I doubt that you'll be able to get anything back from the logger by resetting the date; at 5 minute intervals that's 9 days worth of data. Do you normally run Cumulus most of time, or stop it overnight? What have you got in the backup folder?

[steve]

Interestingly, those dates are both a day after a new beta was released. Could you have installed the one from 6th July (831) to a different location?

[RayProudfoot]

It's hard to see how Cumulus could have been running all that time without logging anything. Presumably the data was there at some point, you would have noticed if it wasn't? What does you dayfile.txt look like? A corresponding gap?

It's baffling. Cumulus was stopped on the evening of the 7th (probably to upgrade to a new version) and again on 26th July probably for the same reason. It's run on a netbook 24/7 solely for wx use. The dayfile.txt has an entry for all the missing dates which is even odder. During this period Cumulus looked perfectly okay and even recorded a new record but crucially the 15 minute saves weren't happening.

Could you have done anything that might have caused Cumulus to log to a different location? You could try searching your computer for jul09log.txt.

I've searched the entire netbook drive and the only jul09log.txt files were those in the backup and data folders.

I doubt that you'll be able to get anything back from the logger by resetting the date; at 5 minute intervals that's 9 days worth of data. Do you normally run Cumulus most of time, or stop it overnight? What have you got in the backup folder?

Pity I've lost those records. I don't suppose there's any way of extracting them from WeatherLink is there? Assuming you use that program. Cumulus runs non-stop including overnight. I only stop it when I update the exe. It seems as though after installing the new version it never started logging until the next one overwrote it. I don't keep old versions so it wouldn't be possible to say what version was in use on what date.

In the backup folder I have several folders and two correspond to the two dates involved. Unfortunately the log for the second date is missing all records from the two dates mentioned.

Oh well, put it down to experience. Worse things happen at sea.

[RayProudfoot]

Interestingly, those dates are both a day after a new beta was released. Could you have installed the one from 6th July (831) to a different location?

Good suggestion but the setup always suggests C:\Program Files\Cumulus and I just go with that. I suppose if anyone else reports a similar problem then it might be linked to that beta but otherwise it has to be down to something at my end but I really wouldn't have a clue what caused it.

C'est la vie!

[steve]

[quote="RayProudfoot"Cumulus was stopped on the evening of the 7th (probably to upgrade to a new version) and again on 26th July probably for the same reason. It's run on a netbook 24/7 solely for wx use. The dayfile.txt has an entry for all the missing dates which is even odder. [/quote]

If Cumulus was running uninterrupted between the times where the data is missing, and given that the dayfile is OK, then I have to admit that the most likely explanation is that it simply failed to write to the log file during that run.

Could you zip up the contents of your Diags folder and upload them, please?

Pity I've lost those records. I don't suppose there's any way of extracting them from WeatherLink is there?

It's certainly possible, and probably not too difficult. I think Weatherlink has an option to export a set of data between two dates as text? Could you do one for me for the missing period, and upload it, please?

[RayProudfoot]

Cumulus was stopped on the evening of the 7th (probably to upgrade to a new version) and again on 26th July probably for the same reason. It's run on a netbook 24/7 solely for wx use. The dayfile.txt has an entry for all the missing dates which is even odder.

If Cumulus was running uninterrupted between the times where the data is missing, and given that the dayfile is OK, then I have to admit that the most likely explanation is that it simply failed to write to the log file during that run.

Could you zip up the contents of your Diags folder and upload them, please?

Thanks for your quick response on this Steve. I've zipped up the entire Diags folder and it's attached.

Pity I've lost those records. I don't suppose there's any way of extracting them from WeatherLink is there?

It's certainly possible, and probably not too difficult. I think Weatherlink has an option to export a set of data between two dates as text? Could you do one for me for the missing period, and upload it, please?

I'll start that now and upload shortly - many thanks!

[RayProudfoot]

Hi again Steve,

Here's the data from WeatherLink for the affected period - 7-26 July. I exported the whole of July and removed the data Cumulus already has.

These records are every 15 mins compared to 5-min with Cumulus. That hardly matters if I can fill in that big gap. Thanks once again for brilliant support.

PS. Also attaching dayfile.txt for info.

[steve]

Yes, sadly the logs are full of:

2009-07-17 22:40:00 : 17/07/2009 22:40:00 Error writing to data file: C:\Program Files\Cumulus\data\Jul09log.txt. I/O error 32
2009-07-17 22:40:00 : 17/07/2009 22:40:00 Error writing to data file: C:\Program Files\Cumulus\data\Jul09log.txt. I/O error 103

Error 32 is a file sharing violation - i.e. the file was in use by another process. Error 103 is just a knock-on from that (failed to close the file because it hadn't managed to open it).

From the logs, it looks like there was some sort of problem closing Cumulus down on the evening of the 6th, so I suspect the other process accessing the file was Cumulus itself - the error may have resulted in two copies of Cumulus running at the same time. There's an option in the station settings to warn when this happens - assuming you haven't set this already, it might be worth doing it.

Perhaps Cumulus ought to shout out if it fails to write to the log file; but occasional transient errors may occur, so I wonder if an error pop-up might be over the top?

[steve]

Could I have your July log as well, please?

The conversion won't be perfect, due the differences in the things that WL and Cumulus logs, but most things will be correct and the rest not far off.

[RayProudfoot]

Yes, sadly the logs are full of:

2009-07-17 22:40:00 : 17/07/2009 22:40:00 Error writing to data file: C:\Program Files\Cumulus\data\Jul09log.txt. I/O error 32
2009-07-17 22:40:00 : 17/07/2009 22:40:00 Error writing to data file: C:\Program Files\Cumulus\data\Jul09log.txt. I/O error 103

Error 32 is a file sharing violation - i.e. the file was in use by another process. Error 103 is just a knock-on from that (failed to close the file because it hadn't managed to open it).

From the logs, it looks like there was some sort of problem closing Cumulus down on the evening of the 6th, so I suspect the other process accessing the file was Cumulus itself - the error may have resulted in two copies of Cumulus running at the same time. There's an option in the station settings to warn when this happens - assuming you haven't set this already, it might be worth doing it.[/quote

I didn't have that option set but it is now. It seems I have just been unlucky with the previous instance not terminating. So should that happen again your warning flag should at least alert me to a potential problem.

Perhaps Cumulus ought to shout out if it fails to write to the log file; but occasional transient errors may occur, so I wonder if an error pop-up might be over the top?

Would a warning message interfere with Cumulus outputting data to the log? If so then that might be best avoided. How about an option to force an immediate log write when Cumulus first starts? If that works then it's probably okay but in a situation like mine it could well fail and at least the user would be immediately alerted to the problem. We definitely need a warning for failure to write to the log. It's just how we go about it.

[RayProudfoot]

Could I have your July log as well, please?

The conversion won't be perfect, due the differences in the things that WL and Cumulus logs, but most things will be correct and the rest not far off.

July log attached. Beggars can't be choosers! I'm just grateful you can give me most of what's I've lost back.

[steve]

I suppose it could pop up an error message if the first write fails - which will be straight away if there is data downloaded from the logger, or within the log interval if not. The message wouldn't stop anything else working. It's a bit messy to force a write if one isn't needed.

[RayProudfoot]

I suppose it could pop up an error message if the first write fails - which will be straight away if there is data downloaded from the logger, or within the log interval if not. The message wouldn't stop anything else working. It's a bit messy to force a write if one isn't needed.

Am I the first person to report this problem? If so then it's a pretty rare occurence so I don't think you need to do too much. I'll certainly monitor the software after installing a new version so a message box seems fine.

[steve]

I'm not aware of anyone else having the problem; occasionally data has gone 'missing', but I don't think it has ever been down to a problem like this in Cumulus.