what if there was a faster update than 'real-time'

[mcrossley]

Only really relevant for Davis stations, but say you could get the wind speed and direction out of Cumulus as it is read from the station, so a 2.5 second update rate. What would be the best way of doing it?

Write to a file and monitor for file changed?
Write it to a network socket?
COM? (A bit Windows only)
Or...?

[mcrossley]

BTW I am thinking Cumulus TNG here, or just how best to do it from a separate program accessing the station in parallel.

[prodata]

BTW I am thinking Cumulus TNG here,

TNG??

or just how best to do it from a separate program accessing the station in parallel.

Which would mean using VVP (for a Davis station) - no other robust option.

[mcrossley]

Well Steve is working in the background on a new version that separates interface and background service.

Yes VVP (or equivalent) would be required. VVP performs a very similar function to ASCOM that we use to share a telescope controller amongst numerous client programs. I am familiar with the .net framework used in Ascom 'drivers'. If I were doing one for the VP2 though I would not bother with the virtual serial ports as they are a pain on 64 bit systems as you need signed drivers - and they normally cost money for every deployment.

I've said it before but weather stations need their own equivalent of ascom so each client program doesn't have to reinvent how to talk to reach different station they support.

[BCJKiwi]

I guess there is the question of how robust VVP actually is?
I know many have it working well. There are others who don't.

I have run some tests. As the requirement was very simple - just Cumulus & WL both in IP mode (to avoid all the problems associated with the serial emulators) - I fully expected it to be robust. Unfortunately I found there were regular lockups and resulting loss of data.

Also, as VVP has not been updated for some time (and appears to now be unsupported) it does not properly support the current Davis gear.

So Mark, a VVP replacement would appear to be a high priority - you could actually make money from that!

[steve]

The version I'm working on has a web server based interface for extracting data, so you could read the data as often as you wanted (e.g. using XMLHttpRequest). Is that what you mean?

[mcrossley]

Steve I guess that could work, but rather than the overhead of using an Ajax type interface for such rapid data updates it would be better to use something like web sockets, or 'long polling'. More of a push technology than a pull. The server end sending the data when it has a update - after the initial session set-up of course.

A VVP replacement probably isn't on the cards - not enough time on my part, and I expect any money to be made would be pretty small. Especially if Davis release a VP3 and change the rules.

[steve]

Steve I guess that could work, but rather than the overhead of using an Ajax type interface for such rapid data updates it would be better to use something like web sockets, or 'long polling'. More of a push technology than a pull. The server end sending the data when it has a update - after the initial session set-up of course.

We could look at adding other interfaces for the data. Anything that is supported by Mono could be added. But my tests using this interface haven't shown any performance issues.

A VVP replacement probably isn't on the cards - not enough time on my part, and I expect any money to be made would be pretty small. Especially if Davis release a VP3 and change the rules.

I thought the same. I effectively already have a lot of the code, so it would just (!) be a case of adding the 'client' side. And like you, I would do it via TCP/IP only.

[Dave]

Only really relevant for Davis stations, but say you could get the wind speed and direction out of Cumulus as it is read from the station, so a 2.5 second update rate. What would be the best way of doing it?

Write to a file and monitor for file changed?
Write it to a network socket?
COM? (A bit Windows only)
Or...?

The loop packet from the VP is the limiting factor for faster "real time" updates. 2.5 seconds is the VP native max speed of "real time" serial data output.

[prodata]

The loop packet from the VP is the limiting factor for faster "real time" updates. 2.5 seconds is the VP native max speed of "real time" serial data output.

Strictly speaking, you can get LOOPs faster than this. They seem to be available about every 2 secs or even more often, though it does depend on what else the console CPU is being asked to do and so 2 secs is a reasonable average figure. And the time to get a single LOOP via the DLL (at least for a previous version DLL) I find is around 0.16secs. This is a little test presentation program I wrote some years ago which uses the DLL (plus some WDL, but no WD!):

http://www.elyweather.co.uk/wdl/

and this consistently reports a LOOP return time of 0.16 secs. (NB1 This is the time to get a single LOOP, the timer for consecutive loops is set longer at 2.5secs. NB2 This URL is often not live - the FTP control is flakey and I need to replace it with something better, if I ever get time.)

But, this is just a pedantic detail. I agree that there's no point with a Davis station in trying to get data more often than every 2.5secs (or actually every 2.5625secs on current Vantage models using channel #1, other channels are marginally longer) because there's no new wind data available to the console at any shorter time interval.

A couple of other points in passing:

Steve (H) definitely has been working on a new version of VVP in the past year or so. But I understand that he has had a very busy work schedule plus other important commitments and so VVP has been very much on the back burner recently. I'm pretty sure that he would still like to release a new VVP version at some point, but whether other pressures will allow him time to do so is pretty much impossible to know.

Second, just to note that the new Instromet systems do provide a 1 sec data update interval.

[steve]

And the time to get a single LOOP via the DLL (at least for a previous version DLL) I find is around 0.16secs.

My experience is that it depends on the firmware version, hence all these recent issues with having to enforce a delay in the Cumulus reading loop. When Cumulus asks for the data on most firmware versions, it takes 2 to 3 seconds for the DLL to return the data. On some of the recent firmware versions, it returns almost immediately. It probably also depends on the DLL version; I'm stuck on version 2.4, the later ones take around 9 seconds (IIRC) to return the data (I was using firmware 1.90 when I last tried it, I think). I think the problem was that it was reading the firmware version before every data read.

[steve]

I should add that it also seems to depend on the actual hardware. I get the 2-3 second response with fw 3.12, but some people with the same fw but more recent consoles/loggers are getting the almost instant response.

[Dave]

Strictly speaking, you can get LOOPs faster than this. They seem to be available about every 2 secs or even more often, though it does depend on what else the console CPU is being asked to do and so 2 secs is a reasonable average figure. And the time to get a single LOOP via the DLL (at least for a previous version DLL) I find is around 0.16secs. This is a little test presentation program I wrote some years ago which uses the DLL (plus some WDL, but no WD!):

http://www.elyweather.co.uk/wdl/

and this consistently reports a LOOP return time of 0.16 secs. (NB1 This is the time to get a single LOOP, the timer for consecutive loops is set longer at 2.5secs. NB2 This URL is often not live - the FTP control is flakey and I need to replace it with something better, if I ever get time.)

But, this is just a pedantic detail. I agree that there's no point with a Davis station in trying to get data more often than every 2.5secs (or actually every 2.5625secs on current Vantage models using channel #1, other channels are marginally longer) because there's no new wind data available to the console at any shorter time interval.

I probably wasn't real clear with my comment. I really wasn't referring to the "overhead" of the methodology used to process a LOOP command to the VP console from external software, but rather the native sampling and processing rate of the loop packet within the VP console itself. Wind vector measurements are the most demanding, in terms of time sensitivity, of the various meteorological elements. One of the compromises of using a digital (contact closure or photometric) wind velocity sensor (the VP uses contact closure), is that they require an interval of sample time. In essence the wind velocity reading is derived as a function of logic circuit frequency output over time. In contrast, an analog sensor output, such as an alternator or magneto has a voltage output that is directly proportional to wind velocity and can be read in real time on an analog voltmeter with no sampling delay, but of course can't be used on a computer without a/d conversion (there goes that sampling delay time again! ).

How the designer of the VP chose to sample and calculate the sensor contact closures to derive wind velocity will determine how often "new" wind data will be present within a LOOP command array regardless of how fast external software can read and parse the string. With the VP, that native limit is around 2.5 seconds. Remember that as wind velocity decreases, the frequency of the sensor pulses will also decrease. If a simple reed switch circuit is used as the pulse generator, as is the case with the VP, there will only be one pulse per revolution, requiring much longer sample times to measure low wind velocities. Sample times can be decreased with a photo chopper optical design, which could be designed to give more pulses per revolution (higher frequency sensor output, hence less time interval needed per sample), but is a much more expensive sensor to manufacture. It boils down to a compromise to reach a price point of a mass marketable product vs absolute accuracy.

I hope I didn't ramble on too much!

[prodata]

I probably wasn't real clear with my comment.

I think we've all been addressing subtly different aspects of the original post. But yes I agree - there's a minimum sample period for wind speed on (current) Davis anemometers and there's no value in any more frequent sampling of the data.

With the VP, that native limit is around 2.5 seconds.

I've always suspected that it's actually 2.25secs. I'm not sure that Davis have ever disclosed the timing details of the ISS but my guess would be that they carried over the timing from the WMII stations to VP/VP2, which used a wind speed calibration number of 1600 (and which is actually still quoted in the spec sheet for the 6410 VP2 anemometer along with a suggested 'sample period' of 2.25secs) to convert revs per hour to mph.

But conceivably they do adjust the calibration number to a 2.5625sec gating period (otherwise about one wind gust value in 12 would be lost) and if they were being really careful they might even adjust it further according to which wireless channel was being used, but I doubt it).

And, as I mentioned, certain other stations like the Instromet do allow a shorter sampling period like 1sec, but it's slightly difficult to know where that fits in. As far as I know, the WMO definition of a wind gust is still the mean wind speed over a 2 to 3 second sample period (ie and not the maximum instantaneous wind speed, however that might be measured) and so a standard wind gust report ought arguably to respect that definition. So, from a definition-based perspective, a 2-second refresh rate might be the minimum that a designer needs to accommodate.

Of course, this is still not addressing Mark's original question about the best mechanism for inter-process communication. I'd have guessed that saving to a file and then watching that file is probably not the optimum solution, though it is possibly a fairly robust one. Would this be a good idea on something like a Raspberry Pi where the file saving is probably all going to be done to an SD card? But given that the preference would presumably be for a cross-platform solution then I'm not sure what other options there are beside a file or a sockets-based approach. But maybe there are other alternatives.

One final point on the VVP aspect: VVP does have many features. If all you required of a VVP-like feature in a new program was that it buffered and echoed LOOP packets to one or more TCP/IP ports then that might be a much simpler proposition. You don't have to think in terms of writing a complete VVP clone.