Weather Underground and Cumulus 1.9.1 beta

[teg]

I'm trying out Cumulus 1.9.1 beta because of its extended uploading to the Weather Underground web site. It appears to upload as far as the last download from the Fine Offset console - well almost because one day had a chunk missing between 0215 and 2000 hours even though all the data was picked up by Cumulus. The earier days were complete. However is it possible to go further back in time and upload all data to Weather Underground?

Trev

[steve]

If Cumulus downloaded the data from the logger, but it didn't make it to WU, there must have been a problem with the upload to WU, and any errors should have been logged to the diags file at the time.

What exactly do you mean by "go further back in time and upload all data"? If you mean all of your historical Cumulus data, then the answer is no. I only added code to upload the station logger data as it was downloaded.

[teg]

Steve, thanks for this information. If I wanted to reload to Cumulus from an earlier date do I just edit the timestamp in the today.ini file or would editing of other files also be required to avoid duplication of data.

Trev

[steve]

Yes, you would have to strip the data logs back to the appropriate point, as otherwise it would just append the old data to the existing files. Don't forget that you can only go back as far as the earliest data in your logger.

[teg]

As I suspected. If there any way to establish the earliest date stored in the logger?

Trev

[steve]

As I suspected. If there any way to establish the earliest date stored in the logger?

Not easily, as they're not timestamped. You would have to calculate it based on your (station) logging interval. There are 4080 entries in the logger (assuming it's currently full). Cumulus ignores a few of the earliest if downloading the entire contents, in case they have been overwritten in the time it's taken to download them.

[teg]

If Cumulus downloaded the data from the logger, but it didn't make it to WU, there must have been a problem with the upload to WU, and any errors should have been logged to the diags file at the time.

It's happend again - blocks of data not getting uploaded to WU. See extract from diags file below.

22/11/2010 22:40:11.620 : An error has occurred during the Wunderground update: HTTP component WUHTTP is busy
22/11/2010 22:40:11.620 : Uploading Wunderground archive #89
22/11/2010 22:40:11.620 : An error has occurred during the Wunderground update: HTTP component WUHTTP is busy
22/11/2010 22:40:11.620 : Uploading Wunderground archive #90
22/11/2010 22:40:11.620 : An error has occurred during the Wunderground update: HTTP component WUHTTP is busy
22/11/2010 22:40:11.620 : Uploading Wunderground archive #91

Would this be a problem with WU?

Trev

[steve]

Would this be a problem with WU?

Hard to say - it looks like perhaps WU is being a bit slow and Cumulus is trying to send the next update before it's had a response from the previous one. I'll check the code.

[steve]

It's a bug introduced somewhere around build 962; the timer for the 'live' updates is getting started too soon. Pending a fix, you may be able to circumvent it by turning off rapid fire (if you're using it) before you close Cumulus and turning it on again after you start again. Then it just depends on what your WU interval is set to compared to how long it takes to send all the logger entries.

[teg]

I'm trying it with rapid fire off and update interval set to 10 minutes (5 minutes before) - see how it goes. The 'missing' updates tended to happen with the longer uploads.

Trev

[robynfali]

I am getting this same problem, but to be honest, i am not overly concerned, yes I do use the rapidfire, but when i go to wunderground it does seem to feed ok, just thought i would mention it

04/12/2010 19:33:50 : An error has occurred during the Wunderground update
04/12/2010 19:48:40 : An error has occurred during the Wunderground update
04/12/2010 20:11:36 : An error has occurred during the Wunderground update
04/12/2010 21:30:43 : An error has occurred during the Wunderground update
04/12/2010 21:33:54 : An error has occurred during the Wunderground update
04/12/2010 21:50:47 : An error has occurred during the Wunderground update
05/12/2010 03:30:58 : An error has occurred during the Wunderground update
05/12/2010 05:01:24 : An error has occurred during the Wunderground update
05/12/2010 05:32:28 : An error has occurred during the Wunderground update
05/12/2010 06:27:41 : An error has occurred during the Wunderground update
05/12/2010 07:51:52 : An error has occurred during the Wunderground update
05/12/2010 09:54:21 : An error has occurred during the Wunderground update
05/12/2010 11:47:49 : An error has occurred during the Wunderground update
05/12/2010 13:30:45 : An error has occurred during the Wunderground update
05/12/2010 14:03:15 : An error has occurred during the Wunderground update
05/12/2010 19:33:07 : An error has occurred during the Wunderground update
05/12/2010 21:23:30 : An error has occurred during the Wunderground update
05/12/2010 21:51:11 : An error has occurred during the Wunderground update
06/12/2010 02:14:01 : An error has occurred during the Wunderground update
06/12/2010 06:15:12 : An error has occurred during the Wunderground update
06/12/2010 06:43:37 : An error has occurred during the Wunderground update
06/12/2010 07:51:01 : An error has occurred during the Wunderground update
06/12/2010 08:50:29 : An error has occurred during the Wunderground update
06/12/2010 10:46:35 : An error has occurred during the Wunderground update

[steve]

I did change the code to make sure it didn't attempt an upload if the previous one was still in progress, so I think these must be genuine occasional errors. When you're uploading every 5 seconds, a few are bound to fail, it's the nature of the internet.

[robynfali]

Oh of course mate, it's why i am not making a big fuss out of it, my site is working, WU is just a "bonus"

[teg]

Latest beta works fine with Weather Underground - i.e no missing data. Question for Steve. The WU uploading takes some time - don't know whether this is due to WU receiving or Cumulus uploading. Therefore is it possible for Cumulus to provide an indication on the level of progress for uploading to WU?

Trev

[steve]

The WU uploading takes some time - don't know whether this is due to WU receiving or Cumulus uploading.

The actual upload should be very quick, it's only a few bytes, but it has to wait for a response before sending the next upload.

Therefore is it possible for Cumulus to provide an indication on the level of progress for uploading to WU?

This is tricky because the upload runs in a separate thread to avoid locking the UI up. Threads can't access the UI directly, there would have to be a synchronisation mechanism with the main thread, and I don't really want to have to do that.