Weather Underground Manual Update suggestion

[aussiewmr]

Hi Steve,
Twice this month I lost my internet connection for 12 and 8 hrs each. This meant that Weather Underground had a big gap in the data. They way I fixed it was to shutdown Cumulus, delete the data from the month log file (e.g. Mar13log.txt), open today.ini and adjust time stamp to the start period of the data loss. restart Cumulus and then let it do its stuff (download data from VP2, do catchup etc). Not sure if that's right, but it worked on both occasions.
So it led me to ask the question. would it be possible for cumulus to have a manual update feature for WU (and PWS etc) where a user can request an update with a start time, end time and per reading interval and it used Cumulus log files instead of downloading data from the station to send the data to the web (WU etc). The interval would be great, because I currently have the VP2 set to record data once a minute, but update WU every 5 mins. Using my procedure above causes WU to get 1 minute readings during catch-up instead of 5 minute readings.
If you say its a realistic possibility, then I will submit a feature request.
Cheers
Phil
(PS I did search for the terms "Weather" "Underground" "Update", with no relevant hit so my apologies if this has been asked before).

[aussiewmr]

OK,
Before everyone beats me up, I was looking at the VP2 forum and found this topic: https://cumulus.hosiene.co.uk/viewtopic.php?f=6&t=6672 and it basically answers what I requested above.
I think it should be easy enough to tackle outside of cumulus.
I don't like the idea of running weather link with cumulus because I have read a few times that the Virtual VP software is a bit temperamental and stops working now and then.
I plan to do some research and if I sort anything out I will let you all know.
Cheers

[steve]

So it led me to ask the question. would it be possible for cumulus to have a manual update feature for WU (and PWS etc) where a user can request an update with a start time, end time and per reading interval and it used Cumulus log files instead of downloading data from the station to send the data to the web (WU etc). The interval would be great, because I currently have the VP2 set to record data once a minute, but update WU every 5 mins. Using my procedure above causes WU to get 1 minute readings during catch-up instead of 5 minute readings.
If you say its a realistic possibility, then I will submit a feature request.

I do keep thinking about having a go at this, but if I did it, it would probably be a stand-alone facility, and as such it's something that someone else could do anyway. I accept that this sort of thing isn't as 'sexy' as fancy web site stuff and mobile apps, so people don't tend to want to do add-ons like this and prefer to leave me to do the more mundane stuff.

If I did come up with something, it would almost certainly just upload every log entry in the specified interval (like the 'catch up' facility does).

There's also this - http://www.hilltop-cottage.info/blogs/adam/?p=187 - as referenced in the thread you've mentioned, but there seems to be an issue with the timestamps.

[BCJKiwi]

I had a look at this yesterday as well as you may recall that I have also asked about this in the past.

An option which occurred to me was that one of our PHP / js experts could write a routine that could sit on the webserver and be manually run as required.
To get the 'missing' data onto the web server would involve opening the relevant monthly log with a proper text editor and copy/paste the block of data to be sent and ftp-ing it to the webserver.
Once it is there, then the server side script could be run.
Hopefully this server side routine could send data to each of the services that handle catchup, not just WU.

I think the issue with the dates on the Hilltop Cottage stuff is that the writer did not make it clear that UTC was required, nor how he achieved it. His approach was simply to add the time offset to shift his local time to UTC and change the format of the string sent.

However Steve, I really do think it would be best if you could write a routine that leverages off all the code, passwords and settings that are already in Cumulus to manage the conversions and formatting, and which destinations to send to.
Again, the user could simply copy/paste the block of 'missing' data from the monthly log(s) to a separate file. A desktop shortcut (or a menu item in Cumulus that calls the separate routine) could then run this routine against a standard filename.

You have previously described the process;

After it's processed each entry from the station's logger and updated its data, it constructs a WU upload URL in the required format using the current data, and stores it away in a list. When it's finished downloading the logger data, it kicks off a thread which processes the list of URLs, sending them to WU one at a time.

Would it be possible to use/replicate this sort of process for this proposed new purpose? I realise the source data will be in a different format which will complicate things.

I feel it is best to leave this data selection manual as it then provides for all manner of reasons to send data be it a new install (as was the case for Hilltop Cottage), an internet outage, or whatever.

I appreciate this is a lot of work but I'm sure you would make a lot of users happy .

[mcrossley]

I'm working on a little something to address this, I'll keep you posted.

[steve]

A fiery horse with the speed of light, a cloud of dust and a hearty Hi-Yo Silver! The Lone Ranger!...

[mcrossley]

... but, work gets in the way today

[steve]

You not worry, Kemosabe; even Lone Ranger need to eat.

[mcrossley]

Either you've been watching too many old films, the pressure of Cumulus is finally getting to you, or you're on the sauce already this morning!

[aussiewmr]

Thanks Mark / Steve.
I think a stand alone app is all that's needed. Its not something that really happens all that often.
I did take a look at the Davis extension that WU put out. I managed to get the config dialog to popup in a basic C# app but with no documentation and no doubt unsuitability of the other DLL exported functions - I gave up.
I started on my own C# app, got as far as reading the cumulus Month log files into a data grid, but sounds like Mark is well on the way to having this sorted. So I might wait.
I guess that sounds a bit lazy, but no point having two solutions do the same thing.
Cheers

[hvalentim]

Alternatively to the manual update option, it would be great if the program was able to acknowledge the failed updates when Internet goes off (instead of just logging them) so as to later resume them.

[gluepack]

aussiewmr said...

(PS I did search for the terms "Weather" "Underground" "Update", with no relevant hit so my apologies if this has been asked before).

Lol, on a technical site like this (not that "Weather Underground Update", is all that technical) it is annoyingly difficult to search and find what it is you are really after. Luckily, I found this on first try.

Living in this god-forsaken country, the internet infrastructure is not much better than the electricity infrastructure so I often have to do this process.

I have used the Hilltop method (http://www.hilltop-cottage.info/blogs/a ... ndergound/), mentioned before, for a while and over the last couple of days have had to do it on at least three occasions, amounting to several hours worth of data.

I think I have sorted it out now (witness my comments on the original blog) but my vote is for some mod to Cumulus to enable (re-)submission based on date/time range.

The only problem (depending upon why you want to (re-)submit) with that is that WU does not update records, you have to delete them first so, if it is mods that you want to make, as opposed to additions, you would have to delete the appropriate records first.

I'm also interested in what other weather services cater for retroactive submission as they either don't allow it or they won't respond to my query.

As far as the Hilltop method is concerned, I use the example spreadsheet but the things that I have found that need changing are...

a) the wind gust (column D on the "Converted for Underground" sheet) gets the wrong data from the "Raw from Cumulus" sheet i.e. it should be from column O and contain
='Conversion Factors'!$B$1*'Raw from Cumulus'!O2
(in cell D2 and drag down the column).

b) the conversion factor for wind speed is incorrect (for me). The raw is in kph not mps so it has to be changed to 0.62137 in order for the mph to be converted back to kph correctly (I've often wondered, but have been too lazy to check, whether there is a WU parameter name for kph as opposed to mph).

c) make sure you change to your weather station id and password in the base URL on the Conversion Factors sheet

d) the text file created from the generated URLs needs to be edited to change the times (and dates, if necessary) to UTC

I haven't bothered to reduce the amount of data to 5 minute intervals (by the time I have done the above my brain is usually scrambled) which makes the graphs look rather interesting sometimes.

Just to reiterate, in case you got bored with the above and skipped over it...

...my vote is for some mod to Cumulus to enable (re-)submission based on date/time range....
I'm also interested in what other weather services cater for retroactive submission as those that I have contacted either don't allow it or don't respond to my query.

[steve]

I'm also interested in what other weather services cater for retroactive submission as they either don't allow it or they won't respond to my query.

ISTR that Weatherbug doesn't. WU clearly does. The others, I don't know.

[mcrossley]

An old thread, but I had to catchup some WU data today after a broadband failure yesterday. SO, I now have an Excel spreadsheet to reads the Cumulus data, selects the period you want and uploads it to WU.

Obviously it worked for me, but I'm not sure how well it will cope with other localisations and time zones. I *think* it will, but it is un-tested.

If anyone wants to test it, shout out and I'll provide a copy. If it works for other people I post it to the Wiki.

[mcrossley]

Oh, I forgot it also does WOW catchup as well.