Problems with station communication: USB locks up completely

[rwilliam99]

While determining the 'station clock' time, it presumably read the data on the 'sensor clock' interval. I suppose I could try to make it avoid doing that, but that then has the potential to miss a 'station clock' change and have to wait for the next one. I don't know if that's what pywws does.

And it seems to me that there's always going to be the chance of hitting the station clock change while determining the sensor clock time, and I don't think there's any way around that.

Also, it looks like your sensor clock is drifting by six seconds a day, and your station clock by four seconds a day, and I don't currently have any code in to cater for that (a drift of more than three seconds):

Steve:

A few questions for you (clarifications really)

- Station clock is the console, correct?
- Sensor clock is the clock on the sensor array, correct?
- In your opinion (since you have experience with a wide variety of systems) is that drift considered really bad? Should I see if I can return it for a replacement?
- Lastly, on the topic of the clock drift - This station is supposed to have a Radio Controlled Clock (RCC) that I thought set the clock based on a signal from an atomic clock somewhere. Maybe it only does it so often?

Thank you so much for your work and help on this. My postings are only intended to help you improve Cumulus. It is a great piece of software.

[steve]

- Station clock is the console, correct?
- Sensor clock is the clock on the sensor array, correct?

Yes; the 'station clock' is theoretically when the minute changes on the station, and the 'sensor clock' is when the data in memory is updated as a result of new data arriving from the sensors.

In your opinion (since you have experience with a wide variety of systems) is that drift considered really bad? Should I see if I can return it for a replacement?

I don't know. I've never looked at clock drift on any weather station before, and I don't even know what is considered good or bad with electronic clocks in general.

Lastly, on the topic of the clock drift - This station is supposed to have a Radio Controlled Clock (RCC) that I thought set the clock based on a signal from an atomic clock somewhere. Maybe it only does it so often?

Note that the drift is actually between the PC and the console, so maybe the console is actually correct and the PC is off. I don't know how often the console updates.

The following might be a useful exercise:

Stop Cumulus and edit today.ini to delete the station clock and sensor clock values (to force a resync). After resync is complete, stop Cumulus and delete the clock values again, but this time force your PC to get the correct time from the internet. Start Cumulus again, and then after resync is complete again we can compare the clock values from before and after your PC has the correct time set by looking at the diags files.

[steve]

Just to be clear, though - clock drift is nothing to do with the problem you've just hit, as it happened immediately after a re-sync.

[rwilliam99]

OK, I think I did what you asked. I stopped Cumulus, edited today.ini to remove the following two values:

FOSensorClockTime=41287.4797675116
FOStationClockTime=41287.479884537

I restarted, and waited for both of those values to be populated in today.ini (which seems like it should be sync). I recorded those two values from today.ini, stopped Cumulus, used the internet time update function in windows 7 to force a clock update (it is enabled to syncronize periodically via time.nist.gov). Once that was complete, I deleted the values from today.ini again and restarted cumulus. Once I restarted, I waited until those values were populated.

So here is what it looks like to me:

Values on first go round
FOSensorClockTime=41287.4587728935
FOStationClockTime=41287.468772338

After resync:
FOSensorClockTime=41287.4797675116
FOStationClockTime=41287.479884537

Log entries from the diags files:

First sync:
1/13/2013 11:00:33.850 : 11:00:33 AM Start Synchronising
1/13/2013 11:15:00.185 : Writing today.ini, LastUpdateTime = 1/13/2013 11:15:00 AM raindaystart = 19.1102352142334 rain counter = 19.1102352142334
1/13/2013 11:15:00.185 : Latest reading: 320: Data: 0E 2A C8 00 51 1B 80 57 28 00 00 00 04 52 06 00
1/13/2013 11:15:01.930 : Sensor clock 11:00:37 AM
1/13/2013 11:15:01.930 : Station clock 11:15:01 AM
1/13/2013 11:15:06.930 : 11:15:06 AM Stop Synchronising

Second Sync:
1/13/2013 11:30:49.287 : 11:30:49 AM Start Synchronising
1/13/2013 11:30:49.883 : EWUSB: Rain total count from station = 1618
1/13/2013 11:31:02.024 : Sensor clock 11:30:51 AM
1/13/2013 11:31:02.024 : Station clock 11:31:02 AM
1/13/2013 11:31:07.025 : 11:31:07 AM Stop Synchronising

Attached is the diags folder zipped up.

One interesting thing - the display on the console (station) has not been set yet from the RCC. So I'm a little unclear as to where the station clock is coming from - the PC? Too many clocks involved!

[steve]

The 'station clock' is actually the PC clock at the time the station performs the action that it performs when its clock reaches the minute. This synchronisation process doesn't involve actually reading any times from the station.

Your 'station clock' times look consistent:

1/13/2013 11:15:01.930 : Station clock 11:15:01 AM
1/13/2013 11:31:02.024 : Station clock 11:31:02 AM

Effectively one second different (modulo 60), which could be accounted for by the PC clock setting.

However, the 'sensor clock' looks odd.

1/13/2013 11:15:01.930 : Sensor clock 11:00:37 AM
1/13/2013 11:31:02.024 : Sensor clock 11:30:51 AM

I make that 1814 seconds difference, which I think is 38 seconds different, modulo 48. Do you agree?

So I think perhaps there is a problem with the 'sensor clock' code. Or possibly, your data is appearing to change when it actually hasn't. I'll see if I can see anything obviously wrong with the code, but I may have to add some extra logging during the synchronisation process.

[rwilliam99]

However, the 'sensor clock' looks odd.

1/13/2013 11:15:01.930 : Sensor clock 11:00:37 AM
1/13/2013 11:31:02.024 : Sensor clock 11:30:51 AM

I make that 1814 seconds difference, which I think is 38 seconds different, modulo 48. Do you agree?

I'm not sure what you are comparing it to? 38 seconds different than what? The timestamps (PC clock)? The thing that looks weird to me is the PC clock (timestamp) is 11:15:01, but the sensor clock says 11:00:37.

I see 1814 seconds on the sensor clock settings (11:30:51 - 11:00:37 - 30 mins 14 seconds), but if you compare the timestamp (PC Clock) settings, it is only 901 seconds (11:31:02 - 11:15:01 - 15 mins 1 second).

I guess I'm just slow - sorry to be so dense.

Rob

[steve]

I'm not sure what you are comparing it to? 38 seconds different than what?

38 seconds (or 10 seconds) out of sync with the previous synchronisation.

The thing that looks weird to me is the PC clock (timestamp) is 11:15:01, but the sensor clock says 11:00:37.

The timestamp at the start of the entry is the time that it was logged, the same as with all entries in the diags logs. It just logs both 'clocks' at the same time, when it's finished synchronising.

I've found the bug in the synchronisation code. Basically, the 'sensor clock' sync only works if it needs to be done immediately Cumulus starts up. I'll upload a new build later today.

[steve]

I've added a couple of extra things in build 1058. Firstly, the diags log messages should show by how much the PC and station clocks have drifted apart since the last sync. Secondly, you can adjust the 'avoidance period' from the default 3 seconds, by adding a line to the [Station] section in cumulus.ini (while Cumulus is stopped):

FOReadAvoidPeriod=N

where N is the number of seconds each side of the station's memory updates during which Cumulus should avoid reading the data. Obviously, you can't afford to make these much bigger than the default, as there is potentially a total period of 2*N seconds each minute and 2*N seconds in each 48-second period, where Cumulus can't read the data.

[ejay]

I've been using build 1055 since mid December 2012. Unfortunately my station's initialisation bytes became corrupt a few days ago. So it's no better than before.

I am about to try build 1058, after duly resetting the station for the 6th time in a year.

I'm wondering whether the memory reading clash is even the cause of these problems. Fine offset may simply have a nasty firmware implementation that doesn't read/write the memory safely regardless of what the USB communication is doing.

[steve]

I've been using build 1055 since mid December 2012. Unfortunately my station's initialisation bytes became corrupt a few days ago. So it's no better than before.

That's not the problem that this change is intended to work around. I have no idea what those bytes mean (other than the expected 'OK' values), or by what mechanism they get corrupted. The 'USB lockup' problem only occurs with relatively recent stations. The 'corrupt bytes' problem has been occurring from the beginning with some stations.

It's still too early to say for certain, but it does look like this change works (ignoring my bug), for the problem for which it was intended.

[VTHokie74]

My observation is that the USB lockups on my WS-1090 seem to occur when the internal memory buffer of the unit is full and is overwriting older data. After my last USB lockup I started exiting Cumulus and using EasyWeather to clear the memory in the 1090 when it gets full . I have not had a lockup since I started doing that. I am using an older Dell Laptop with Windows XP to run the station. I was averaging a USB lockup every few weeks before.

I have not tried the beta version that attempts to fix the problem.

[steve]

I can't tell you that what you're doing can't work, because your evidence is to the contrary, but 'clearing' the memory just resets the display on the console, it doesn't actually do anything to the memory; the station continues to overwrite old data.

[ddouglass]

Ok, maybe I am missing something but where is the 1.9.3 build 1058 beta version for download?

[water01]

It is here https://cumulus.hosiene.co.uk/viewtopic.p ... 1&start=25

[Spider-Vice]

My FO continues acting up like hell, it went through a big storm surprisingly well but after that it just started failing, well, recently I've seen that the station seems to update constantly most of the times but then it just seems to lose sensor contact all of a sudden, sometimes recovering it, others not recovering it, like today. Thinking of the USB issue, I restarted it today and it did seem to "work"; until it started doing it again. Now the mysterious thing: It lost its signal about an hour ago (20:46), at 21:00 it goes get the DCF signal... And that got detected without the sensor data So might this be an USB issue or...? I'm really lost, transmits one part but not the other.