Cumulus Support

I noticed that the graphs of today's data looked oddly 'stepped'. I've seen this a couple of times beforewhere Cumulus encountered a dubious block when downloading data from the station. As I've done before, I deleted the duff data and tweaked the timestamp in Today.ini, which allowed it to skip the bad record and grab the subsequent data. However, looking back over the log I find that there are lots of earlier chunks where readings are suspiciously static. They then change, and stay at the new values for several minutes at a time and so on. It seems OK now, but I have about 72hrs worth of 'dirty' data like this.

Does anyone have any suggestions as to what might be causing the problem? If I understand correctly, it can happen as 'one-off' problems where Cumulus ignores a bad block, but I thought that everything following that was completely static until it was manually purged and re-fetched, not stepwise changes like this.

Secondly, is there a reasonable way of re-building and 'sanitising' the data log? I could set the Timestamp to the first of the errors and get Cumulus to read data from then, but if each step is a bad block then it wouldn't be realistic to do that so many times.

Which items are showing this? All of them?

If I understand correctly, it can happen as 'one-off' problems where Cumulus ignores a bad block, but I thought that everything following that was completely static until it was manually purged and re-fetched, not stepwise changes like this.

I don't understand what you mean by that. When Cumulus ignores a set of readings, the existing values continue to be used, until it decides that the data is OK.

Could you let me have an example of a data log (Aug09log.txt) with this data in it?

Thinking about it - it would be useful to see grabs of the graphs too

Sorry Steve, it's not easy to describe and I didn't want to blast a large chunk of data into the forum uninvited. File now attached. If you take a look around 23/08/09 19:20 you'll see that it's pretty much all of the fields, but wind speed and direction are the ones that really stand out - it just doesn't seem feasible for these to be exactly the same on so many successive readings, especially with the pattern repeating.

I don't understand what you mean by that. When Cumulus ignores a set of readings, the existing values continue to be used, until it decides that the data is OK

When I've seen similar things before, it appeared to have been caused by a single bad record from the weather station, with all subsequent readings being duplicates until I forced a download by getting Cumulus to skip the bad record. I don't think I've seen it decide that the data is OK and get back on track by itself, so I didn't realise it could - I just assumed that the Fine Offset hardware is temperamental and difficult to work with.

Yes, it's certainly suspicious for the wind speed to be showing 13.6 (for example) for several minutes continuously. I can only assume that Cumulus is detecting a problem with one of the readings (rightly or wrongly) and as a result discarding all of the data (which I should probably make optional, really). I think the best thing is to turn on the debug log; it logs a message when it discards the data, with the reason for it. It seems to be happening often enough that we won't have to wait long!

When I've seen similar things before, it appeared to have been caused by a single bad record from the weather station, with all subsequent readings being duplicates until I forced a download by getting Cumulus to skip the bad record. I don't think I've seen it decide that the data is OK and get back on track by itself, so I didn't realise it could - I just assumed that the Fine Offset hardware is temperamental and difficult to work with.

Right, I understand. I think there may be situations where it wouldn't manage to recover, but normally once the duff reading has gone back to normal, it should just carry on again, as it appears to be doing in this case.

It looks like it's behaving quite well at the moment, but I'll try to keep a closer eye on it for the next few days and turn on debugging if it starts doing it again. I've now deleted all the data from the start of the oddities until it settled again and forced an update to back-fill from the weather station, and it looks more or less OK.


On a matter that is (I assume) completely unrelated, while poring over the graphs I noticed a number of temperature readings that seem spurious, e.g. a single reading that's several points of a degree different from its neighbours (in one case it was 3.1deg!). Is this typical of the Fine Offset units (or indeed home weather stations generally) or dies it indicate a problem that I should be investigating?

Blackmyre wrote:On a matter that is (I assume) completely unrelated, while poring over the graphs I noticed a number of temperature readings that seem spurious, e.g. a single reading that's several points of a degree different from its neighbours (in one case it was 3.1deg!). Is this typical of the Fine Offset units (or indeed home weather stations generally) or dies it indicate a problem that I should be investigating?

When you say "neighbours", how far apart in time? Looking at the output from my Fine Offset, the temperature does fluctuate quite a lot, but so does my Davis. But if you're talking about a difference of 3.1 degrees between two 'real time' points on the graphs - one minute apart, that does sound a bit extreme.

When you say "neighbours", how far apart in time

One minute. The 3.1 difference was the biggest I noticed, but there are several spikes of a degree or so, with no corresponding significant changes (rain, pressure, wind direction etc.).

By the way, I just noticed the Cumulus Wiki and, from that, that an initial C2 might be looming. Sounds like interesting stuff. Having a DBMS behind it should make it rather easier to separate client & server functionality (assuming that's your intention?).

Blackmyre wrote:One minute. The 3.1 difference was the biggest I noticed, but there are several spikes of a degree or so, with no corresponding significant changes (rain, pressure, wind direction etc.).

Given that there's no corresponding oddity with the other data, that sounds like a problem with your temperature sensor to me.

By the way, I just noticed the Cumulus Wiki and, from that, that an initial C2 might be looming. Sounds like interesting stuff. Having a DBMS behind it should make it rather easier to separate client & server functionality (assuming that's your intention?).

Well... yes... that was going to be one of my goals, but I have to admit that at the moment it is just a single piece of software like the current version. At some point I will have to decide exactly how I'm going to tackle that, and whether to split the functionality completely, or whether to just allow a station type of 'Cumulus', where two copies of the same program can communicate using tcp/ip. I'm leaning towards the latter at the moment.

sounds like a problem with your temperature sensor to me.

I guess so. I thought I'd mention it in case it turned out to be a common problem with cheap home units, insects crawling on the sensor or whatever.

whether to split the functionality completely, or whether to just allow a station type of 'Cumulus', where two copies of the same program can communicate using tcp/ip. I'm leaning towards the latter at the moment.

Good idea. Thinking about it, perhaps it could be simpler still, with the 'master' instance updating the database and the 'slave' instance(s) just re-querying the database periodically. If want to avoid the slaves having to poll, it might be possible to have a stored procedure fire a piece of code that notifies subscribed clients of the change (though I don't know if SQL-Lite supports such things). Or simply use something like FileSystemWatcher to monitor updates. Either way, there would be no need for the master to communicate directly with slaves.

Blackmyre wrote:I guess so. I thought I'd mention it in case it turned out to be a common problem with cheap home units, insects crawling on the sensor or whatever.

Sun on the sensor?

Good idea. Thinking about it, perhaps it could be simpler still, with the 'master' instance updating the database and the 'slave' instance(s) just re-querying the database periodically. If want to avoid the slaves having to poll, it might be possible to have a stored procedure fire a piece of code that notifies subscribed clients of the change (though I don't know if SQL-Lite supports such things). Or simply use something like FileSystemWatcher to monitor updates. Either way, there would be no need for the master to communicate directly with slaves.

One problem with having the slaves just using the database data is the update frequency. Typically, you would only add an entry to the database every few minutes (or possibly once a minute if you really needed that much data), while the data from the station might be updating every few seconds, and you might want the slave to update that frequently also.

Sun on the sensor?

I thought the screens were supposed to largely prevent such fluctuations. And some of the spikes were in the wee small hours.

the data from the station might be updating every few seconds

Hmmm... You could have a 'realtime' table, of course, but I suppose that would be quite a lot of querying, especially if there were multiple slaves. Do you have UDP in mind?

Blackmyre wrote:I thought the screens were supposed to largely prevent such fluctuations.

They're not particularly good. A number of people have made their own screens from things like plant pot saucers (hmm... that needs to go in the wiki).

And some of the spikes were in the wee small hours.

Ah, right.

Do you have UDP in mind?

I hadn't actually thought about it too much, but I'd imagined it would use a TCP connection. Or perhaps the master could broadcast using UDP (like the home automation xAP protocol).

perhaps the master could broadcast using UDP (like the home automation xAP protocol).

I'm not familiar with xAP, but thinking along the lines of reducing traffic made me think UDP broadcast might be attractive: fast, simple, no registration needed.

Look forward to seeing it anyway. Is the switch from Delphi very recent, or have you been using Viual Studio for a while now?

Blackmyre wrote:Look forward to seeing it anyway. Is the switch from Delphi very recent, or have you been using Viual Studio for a while now?

About a year now. I started off with the free 'express' edition, then Microsoft were kind enough to give me a free three year subscription to MSDN, so I now have the full version and everything else too. It's all been a bit of a learning curve: C#, WPF, the Telerik controls, lots of new stuff to learn. But it keeps it interesting, even if it does mean I make very slow progress at times.