New user - Vantage Vue - More frequent wind data?

[sunray]

Hi folks! First, MANY THANKS for creating this fine software and making it available!

Last week I finally got my weather station back on line after a four year pause. I have installed a Vantage Vue with Weatherlink IP at the cottage, and installed Cumulus on my PC at home. Setup was surprisingly easy and now I get the data from the station logged on the PC .

To my question: I would like to use the Vantage Vue for wind gust analysis and thus would need to log every wind data sample. The Vantage Vue samples with 2.5 sec interval and 1 degree resolution, and the Cumulus main window also shows these individual data samples in its main window (Latest and Bearing fields) albeit with some jitter in the time stamps. So far so good.

But my problem is that there seems to be no way of logging these individual data samples to a data file. Even using the 1 minute data log interval is too coarse to look at individual gusts.
Is there any solution for this problem? I realize it would be a lot of data to log, but hard disks are inexpensive these days

EDIT: I tried to search if this topic has come up earlier, but no luck...

[steve]

You would need to do something like taking the realtime.txt file (which you can configure to be created at any interval down to 1 second), and creating your own log from that. You don't need to have it uploaded to a web site (which is its normal function) you could just process it locally. Some people do this kind of thing to create a 'realtime.log' which they use for creating graphs or whatever on a web site, or feed the file into an SQL database. Instead of using the realtime.txt file, you could create a 'custom' file using web tags, and get Cumulus to process it at whatever interval.

Note that because of the way the Davis DLL works (the code which Cumulus uses to read the data from the console) there is no guarantee that every 2.5 second update will be seen by Cumulus, and in practice you may find that quite a few are 'missed'.

[mcrossley]

It is possible to get (almost - see Steve's caveat) every 2.5 second wind speed update irrespective of your realtime interval.

For instance my realtime interval is set to 5 seconds, but my Climatica display updates every 2.5 seconds by reading the wind arrays available via the web tags. Cumulus adds a wind speed and direction entry to these arrays every time it gets data from your station. You just need to keep track of what the pointer into the array was last time you read it, and read all the entries since then. Your interpolated time stamps may a little out if Cumulus has missed a reading, but that may be acceptable?

[steve]

Ah - very cunning, Baldrick.

[sunray]

Many thanks Steve for the quick response! I had a look in the realtime.txt FAQ. This looks like a good solution!


Unfortunately my coding skills are very rusty. I guess a simple solution would be to have a program that reads the realtime.txt file at (say) 2 sec intervals, checks if the timestamp has changed and if yes appends it to a larger data log file which I can then read into Excel for analysis. If someone has already done such a program/script, I'd be grateful for any support

[WX_Frank]

It is possible to get (almost - see Steve's caveat) every 2.5 second wind speed update irrespective of your realtime interval.

For instance my realtime interval is set to 5 seconds, but my Climatica display updates every 2.5 seconds by reading the wind arrays available via the web tags. Cumulus adds a wind speed and direction entry to these arrays every time it gets data from your station. You just need to keep track of what the pointer into the array was last time you read it, and read all the entries since then. Your interpolated time stamps may a little out if Cumulus has missed a reading, but that may be acceptable?

Very impressive Instrument setup/layout, "mcrossley" .....can I use this with my Vantage Vue + Cumulus setup?

Regards,
Frank.

[mcrossley]

Very impressive Instrument setup/layout, "mcrossley" .....can I use this with my Vantage Vue + Cumulus setup?

You could in theory, but it belongs to John at Prodata Systems, you would need his permission.

[WX_Frank]

Very impressive Instrument setup/layout, "mcrossley" .....can I use this with my Vantage Vue + Cumulus setup?

You could in theory, but it belongs to John at Prodata Systems, you would need his permission.

OK...is he here on the forum or where do I contact him for permission and what to do?
Would be great...love it

[mcrossley]

John is on the forum as 'prodata' or via his website http://weatherstations.co.uk/

I don't think he has made a decision yet on the future of the display, so I would not expect him to allow distribution until that future has been decided. I know he has other priorities at the moment, so I would not pester him to make it available.

[sunray]

Hello again,
Once again thanks for your efforts with Cumulus.

I have now made progress and been able to log data from the realtime.txt output. Before starting my actual wind gust analysis project I thought it useful to have a look at the data stream. So I wrote a small logging app. It reads the realtime.txt file’s data row each second and checks if either wind speed or wind direction has changed since the previous row. If yes, the new row is appended to the log file.

Then I made some analysis of the resulting log file. The average time between each row is approx 4 sec, so on average slightly more than every 2nd of the Vantage Vue’s total data samples are registered. (VVue output rate is specified as 2,5 to 3 seconds). This is for a day with light to medium wind conditions. Two days ago when it was almost calm the average became longer, as the wind data from the VVue remained unchanged for longer periods.

Here is a frequency distribution table of the intervals between data rows, where either wind speed or direction has changed. Note that the "Bin" data on e.g. the 00:00:05 row means the number of measured intervals larger than 4 but smaller or equal than 5 seconds.

cumulustest06distrib.GIF

However (and this is the reason for my post), the log file shows curious large gaps in the data stream, see chart.

cumulustest06 graph.GIF

I had a closer look at the gaps and it turns out that they repeat quite regularly at approx. 36 minutes intervals (average 36:20 in this particular dataset) and have a duration of approx. 6.5 minutes (average 6:31 in this dataset).

cumulustest06gaps.GIF

The portion of the gaps to the total time logged is approx. 17 percent, so a significant amout of data gets lost.
So now I am wondering where the reason might be for this behavior? My system consists of the remote Vantage Vue with Weatherlink IP, connected over an ADSL link and the internet to my PC at home running Cumulus. So there are multiple points where a data gap can occur.

The Cumulus error log actually also reports this problem in the form of a number of “Data input appears to have stopped - check your station and connections” messages with similar timing. Also, the "Last data read at" clock in lhe lower left corner of Cumulus stops during these data gaps.

cumulustest06clock.GIF

(I did also observe that there is a regular gap in the data after each full minute, i.e. after the :00 second tick it takes typically 6 seconds until the next data row with new data is logged.)

I would appreciate if someone else who has experienced similar problems, could advise what I can do to get rid of those gaps!

[steve]

(I did also observe that there is a regular gap in the data after each full minute, i.e. after the :00 second tick it takes typically 6 seconds until the next data row with new data is logged.)

There are various things that Cumulus does once a minute, some of which are optional. Because of the way the Davis DLL operates, each of these can take up a whole 2.5 second data slot. The operations are:

1. Disconnect to allow the WLIP to do its uploads to weatherlink.com. Controlled by the 'discon period' in the station settings (set to zero to disable).

2. Read extra barometer data from the console so that Altimeter Pressure can be calculated. This is done mainly for CWOP uploads. You can turn this off by stopping Cumulus and editing cumulus.ini, adding a line

DavisCalcAltPress=0

to the [Station] section.

3. Force the update of the barometer. This is not needed for newer stations, and is turned off by default. Make sure it isn't selected in the station options.

4. Read today's high gust data from the console (if rollover at midnight is selected, rather than 0900). This is to make sure that the high gust is not missed as not every 2.5 second update will normally be captured. This cannot currently be turned off.

In addition, every 15 minutes it reads the reception stats from the console. This cannot currently be turned off.

With regard to your six minute gaps, that's either a communication issue (quite likely, given your extended communication path via the internet), or (perhaps more likely, given the regularity) the WLIP is 'busy' doing something, most likely uploading (or trying to upload) to weatherlink.com. Do you have that enabled?

[sunray]

Hi Steve!

My current settings are:
- discon period : 0
- DavisCalcAltPress : not added
- Force update barometer: unchecked

Regarding the six minute gaps, I do have weatherlink.com enabled. Unfortunately, I have misplaced my original Weatherlink CD so any configurations done with it are those done in 2009 on my original installation (on another computer)

So I'm at the moment unable to change anything that way. (Davis has an upgrade downloadable for the WL software, but they have dire warnings about installing it without the original software installed..... I guess I have to try anyway?)

I went to weatherlink.com, but my account page only allows me to disable the Wunderground service (which I understand uses WL.com data anyway and thus should not put additional load on the comms link?).

While I'm at it, would you have any suggestion for a slight problem I've observed: The realtime.txt file appears to work so that it is written to every second (or any chosen interval) regardless of if there is a new update from the station or not, and in the latter case just repeating the previous data. This makes it tricky to know if Cumulus has missed an update from the station, or if the station did update but all weather data has just remained stable since last update.
I'm not yet sure if this does matter for me, but I have a feeling this has some effect on the accuracy of averaging calculations. I.e. if the wind is reported stable during, say, 20 seconds, that wind value should be used in the averaging. But if there is a 20 seconds gap in the updates it might be prudent to not let that last recorded "hanging" value influence the average?
As a humble suggestion, one solution might be to add a "new update" flag to the realtime.txt data, indicating that this is the first data row written since the arrival of a new update from the station. Then on subsequent repeats this flag is cleared, until the next update arrives.

Cheers,

[steve]

Regarding the six minute gaps, I do have weatherlink.com enabled. Unfortunately, I have misplaced my original Weatherlink CD so any configurations done with it are those done in 2009 on my original installation (on another computer)

As I understand it, you configure the WLIP uploads via its web interface (i.e. put its IP address into your browser). You would have to open up the necessary ports if you wanted to do that over the internet, of course.

While I'm at it, would you have any suggestion for a slight problem I've observed: The realtime.txt file appears to work so that it is written to every second (or any chosen interval) regardless of if there is a new update from the station or not, and in the latter case just repeating the previous data. This makes it tricky to know if Cumulus has missed an update from the station, or if the station did update but all weather data has just remained stable since last update.

Yes, that's basically how Cumulus works, sorry. It works with several different types of weather station, not all of which update at the same frequency as Davis stations, and in some cases it's not easy to tell if or when the station has updated its data. Much of Cumulus is therefore a compromise.

[sunray]

Thanks!

I finally found out how to config the WLIP to stop uploading to Weatherlink.com. This thread shows it: https://cumulus.hosiene.co.uk/viewtopic.php?f=6&t=3331

After exiting Cumulus I was able to access my WLIP and got a similar screen as in the above thread.
So now I can remove the tick mark from the Weatherlink.com box.

But a question just for safety's sake: Should I also change to a static IP address? So far it uses the "automatic" (dynamic) option. What are the pros and cons? The current IP address has remained the same for one month now. What if there is a change of (dynamic) address - will I be able to contact the WLIP again? I found the currently used IP address from the My Account page of Weatherlink.com, but will a new IP address be uploaded to Weatherlink.com if the box is unchecked? ?

My station is quite a distance away from where I live, so I 'd rather not do anything that will require a site visit, therefore the question.

[sunray]

Hello again!

I am still trying to find the cause of these regular “six minute gaps” in the output from Cumulus. The problem still persists, so I have to turn to you for cues.

I have tried the following:
- Unchecking the “upload to weatherlink.com” box on the Weatherlink IP web configuration page. No effect on the gaps, so I left it active.

- Discon period set to 0. No effect on the gaps.

- Stop the reading of extra barometer data from the station (DavisCalcAltPress=0). This had a visible impact on the pause that Cumulus makes after each minute tick. It went to 2-3 sec from the previous around 6 sec. This was welcome, since it allows at least one more wind sample to be registered per minute, so I will leave this on. Unfortunately there is no impact on the “six minute gaps”.

- Next I increased the Discon period in order to allow Weatherlink.com to successfully fetch data, and also observe how Weatherlink.com acts during the “six minute gaps”. Increasing the Discon period to 5 sec had the result that Weatherlink.com was able to get updated every minute, as it should. Interestingly, during the “six minute gaps” also Weatherlink.com did not update ! So this confirms that also Weatherlink.com is affected by the gap.

- I then reduced the Discon period to 3 sec. The effect was Weatherlink.com got updated only approx. every alternate minute, so this value seems slightly low if I would want to serve both Cumulus and Weatherlink.com in parallel. But remember, my goal is to catch each wind sample, so for that Discon period needs to be 0.

- I then checked the “Restart on data loss” box in Cumulus settings to see if a restart would have effect, and waited until the next “six minute gap”. After one minute, Cumulus did restart but then displayed error 32701 i.e. no contact with station. Further manual restart attempts were also not successful until the six minute gap had ended. Likewise, Weatherlink.com did not get updated until after the six minutes had passed. So, restarting Cumulus during the gap had no effect.

- Then, a check with Cumulus inactive during the next “six minute gap”. I manually refreshed the Weatherlink.com page, and during the expected time of the next gap it did not respond! So this time the problem was there also without Cumulus!

- However, I also looked at the WLIP web configuration page, and noticed on the last row that it reported DNS:0 and Status:128 which I found strange. A few minutes after the “gap” had ended, upon refreshing the web configuration page the WLIP Status field had reset to 0 !

- I then decided to wait for the next expected data gap, and this time Weatherlink.com performed normally and no gap was to be seen. At this point I had a hunch that the data gap and the status code 128 have something to do with each other.

- Next trial: start Cumulus with a Discon period of 10 sec, to allow access to the WLIP configuration page, and check the status code. So I did until the next “six minute gap”, when both Cumulus, Weatherlink com stopped updating and the WLIP configuration page stopped responding. So the problem came back when Cumulus came on again.

However, now the WLIP configuration page still reported status code 0.

So a summary of the facts so far:

The data gap somehow starts to occur when Cumulus comes on.

When Cumulus is turned off the data gap seems to cease, i.e all is OK when Weatherlink.com is running.

WLIP may display status code 0 or 128 but the data gap may occur in both cases.

I’m puzzled… any helping advice of what to try next? Thanks for any ideas!