Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.5 (build 3282) - 23 February 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Potential issue with sunshine hours when loading from Archive

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

I have been away for 10 days and my test install of CMX was not left running, so this morning I loaded up the latest update of CMX and started this test install up again at 10:01 today (8th May). It loaded up the archive data from last shutdown on 26th April until today 8th May. All went well but looking at my values for sunshine hours today this shows 4.3 hours sunshine today but my CMX which has been running alongside WD 24x7 shows only 2.3 hours for today.

Now I know leaving a system 10+ days offline is not that usual but I'm sure some folks do if they go on holiday, but even so if one has an outage during the daytime which could last some hours catch up might still over estimate the sunshine hours. I'm guessing that this is caused by the gap between data points which I think is 5 minutes so potentially if only one minute is sunny it could assume that it was 5 minutes actually, over estimating by up to 80% potentially.

Now I'm not sure there is anything which can be done about this since we have to live with the way Ecowitt have defined the data points. Obviously a station with some kind of logging and battery backup would eliminate this (assuming the data points were no more than 1 minute apart), sadly at present this means a much more expensive station like the Davis ones.

I thought I'd post my thoughts on this for information and maybe a bit of discussion so folks will realise the archive at 5 minutes is maybe not perfect for sunshine hours if a local outage is not short.

Stuart
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

Just wanted to say that this issue hits the sunshine hours if I simply shutdown CMX overnight. Turned off at 23:17 and on at 10:02 this morning and it shows 3 hours of sunshine, much in excess of what it should be. So for anyone shutting down overnight it is likely your sunshne hours will be wildly wrong!

Stuart
freddie
Posts: 2421
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by freddie »

broadstairs wrote: Mon 09 May 2022 9:05 am Just wanted to say that this issue hits the sunshine hours if I simply shutdown CMX overnight. Turned off at 23:17 and on at 10:02 this morning and it shows 3 hours of sunshine, much in excess of what it should be. So for anyone shutting down overnight it is likely your sunshne hours will be wildly wrong!

Stuart
Not wishing to argue, but what is wrong with recording 3 hours of sunshine before 10am this time of year? Even here further west sunrise was at 05:25. Looking at the satellite sequence for this morning there doesn't appear to be much cloud around in Kent.
Freddie
Image
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

freddie wrote: Mon 09 May 2022 10:18 am
broadstairs wrote: Mon 09 May 2022 9:05 am Just wanted to say that this issue hits the sunshine hours if I simply shutdown CMX overnight. Turned off at 23:17 and on at 10:02 this morning and it shows 3 hours of sunshine, much in excess of what it should be. So for anyone shutting down overnight it is likely your sunshne hours will be wildly wrong!

Stuart
Not wishing to argue, but what is wrong with recording 3 hours of sunshine before 10am this time of year? Even here further west sunrise was at 05:25. Looking at the satellite sequence for this morning there doesn't appear to be much cloud around in Kent.
The fact is that this is more than double what my other CMX recorded today and that one runs 24x7 on the same sensors!!!!!!!

Stuart
freddie
Posts: 2421
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by freddie »

broadstairs wrote: Mon 09 May 2022 10:33 am
freddie wrote: Mon 09 May 2022 10:18 am
broadstairs wrote: Mon 09 May 2022 9:05 am Just wanted to say that this issue hits the sunshine hours if I simply shutdown CMX overnight. Turned off at 23:17 and on at 10:02 this morning and it shows 3 hours of sunshine, much in excess of what it should be. So for anyone shutting down overnight it is likely your sunshne hours will be wildly wrong!

Stuart
Not wishing to argue, but what is wrong with recording 3 hours of sunshine before 10am this time of year? Even here further west sunrise was at 05:25. Looking at the satellite sequence for this morning there doesn't appear to be much cloud around in Kent.
The fact is that this is more than double what my other CMX recorded today and that one runs 24x7 on the same sensors!!!!!!!
Oh okay, fair enough. I hadn't appreciated that you were comparing with another system.
Freddie
Image
User avatar
mcrossley
Posts: 12641
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by mcrossley »

Hi Stuart, can you confirm that your solar algorithm, percentage, and threshold; and the solar calibration and station location settings are exactly the same across both installs of CMX?

The calculation in CMX is very simple, for each historic record it checks if...
- The solar radiation value (after calibration) is greater than the threshold value
- The solar radiation value (after calibration) is greater than the percentage of calculated maximum
- You don't have a Blake Larsen sensor

If all the above are true, then 5 minutes is added to the days sunshine hours - 5 minutes being the Ecowitt archive interval.

Rinse and repeat for each archive record.
User avatar
mcrossley
Posts: 12641
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by mcrossley »

Hmm, possibly too simple, there should also be a check that the calculated solar max is > 0 to prevent spurious solar readings during the night being added in, though a sensible threshold value would prevent that anyway. But someone may configure the threshold to be zero!
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

mcrossley wrote: Mon 09 May 2022 1:25 pm Hi Stuart, can you confirm that your solar algorithm, percentage, and threshold; and the solar calibration and station location settings are exactly the same across both installs of CMX?
I just checked and there was a difference in transmission factor in that the one which gets turned off over night was 0.8 and the 24x7 one was 0.84 both are the same for June & December. So I'll see how much difference that makes - I suspect not a huge amount. All other settings are the same and they both access the same gateway.

As for how this works I think the issue is setting all 5 minutes as sunny. It is perfectly feasible that the solar value could change quite quickly during the time from sunrise to mid morning especially. I was thinking that perhaps two adjacent 5 minute values could be compared and then the values adjusted to give a smooth rise/fall across the 5 minutes, or another way could be to take the mean between the two and only make it sunny if that value exceeds the threshold.

I know sunshine hours is only a very inaccurate science (more a guestimate :roll: ). I'm still having to compare my WD calculations with CMX to see how well I can get them to match as my gut feeling is WD is closer to the actual than CMX still but I have improved it somewhat.

My main issue at present is the times just after sunrise and before sunset where the max values I see in CMX seem to me to be too low! Actually at present my sunset values are less of an issue as my sensor is shaded by next door from around 6pm, the old VP sensor was mounted on my house roof and did not get shaded! But my wife wont let me climb ladders these days :groan:

Stuart
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

Well today sods law it has been cloudy and overcast since dawn so when I started CMX today the archive catch up has not added any sunshine hours at all which to be honest is what I'd expect. The issue comes when there is some sun to be recorded. So we wait another day to see what happens. Thursday is the first forecast day with some sun here so if the UKMO is correct we wait and see what happens.

Stuart
User avatar
PaulMy
Posts: 3760
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2 Plus 24-Hour FARS
Operating System: Windows8 and Windows10
Location: Komoka, ON Canada
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by PaulMy »

I will post my sunshine data - VP2 Plus:
1. CMX b3183 running 24/7 WiFiLogger - last 4 days sunshine hours: 14.1, 13.2, 13.5, 2.7
2. CMX b3183 running 24/7 Weatherlink Live - last 4 days sunshine hours: 13.9, 13.0, 13.4, 2.5
3. CMX beta running 24/7 Weatherlink Live - last 4 days sunshine hours: 13.9, 13.0, 13.3, 2.6
4. CMX b3180 new start this morning and catch up from 04/30 Weatherlink Live at 5 minute interval - last 4 days sunshine hours: 14.0, 13.0, 13.3, 2.6
Solar settings for all are the same - Ryan-Stolzenback, 75%, Min 0, June 0.8, Dec 0.8

That shows a slight difference in the WiFiLogger (1) setup compared to the WLLs (2,3) that are all running 24/7.
The CMX b3180 (4) had not run since 04/30 when v3.16.0-b3182 was released so that did the catch up when I ran it this morning, and sun shine hours (4) is very close to the CMX running 24/7 (1,2,3). So for me using Davis VP2 Plus, the archive catch-up provides comparable sunshine hours.

Enjoy,
Paul
Davis Vantage Pro2+
C1 www.komokaweather.com/komokaweather-ca
MX www.komokaweather.com/cumulusmx/index.htm /index.html /index.php
MX www.komokaweather.com/cumulusmxwll/index.htm /index.html /index.php
MX www. komokaweather.com/cumulusmx4/index.htm

Image
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

OK so this morning my CMX running 24x7 shows 0.6 (36 minutes) sunshine today but my CMX just restarted today and caught up via the archive shows 1.3 (1 hour 18 minutes) sunshine today. Now both of these installs of CMX run off the same GW1100 so they share the same sensors and both have the same settings for solar and use the WS80 reported solar data. So almost exactly double the sunshine hours recorded after archive catch up from Ecowitt data. Seems like the only time they agree if there is zero sunshine recorded from the archive and zero actual as yesterday

Something is going wrong here, not sure what but at a rough guess probably because it assumes that one 5 minute report give 5 minutes of sunshine.

One further observation which seems to me to show both CMX installs are running identical data streams and settings is that both show 1.1 (1 hour 6 minutes) sunshine for yesterday as all the sunshine happened in the afternoon while both installs were running normally.

Stuart
User avatar
mcrossley
Posts: 12641
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by mcrossley »

You haven't revealed your settings yet, but I suspect most of the problem is down to period around sun rise.

Ecowitt use 5 minute logging, we cannot get around that, and we cannot invent data that doesn't exist, so we have to assume 5 minute blocks of sunshine or no sunshine from a single reading. Catch-up in this case is never going to be very accurate.

However, I have just compared my solar readings as recorded "live" by CMX around dawn this morning and compared them against the values on ecowitt.net.

Code: Select all

Time    Ecowitt  WS80  WS80-avg  WS80-max  Theoretical
05:25         0     0         0         0            0
05:30         0     0         0         0            2
05:35       1.2     0         0         1            5
05:40         0     0       0.8         1            9
05:45       1.2     0       0.4         1           14
05:50       2.5     3       2.2         3           20
05:55       4.9     2       3.8         6           27
06:00       3.6     2       1.6         2           35
06:05       3.6     3       2.6         3           43
06:10       4.9     4       3.4         4           53
06:15       4.9     3       3.6         4           62
06:20       4.9     6       3.8         6           73
06:25       9.8     8       7.2         8           84
06:30      13.4    10       9.0        10           95
06:35      12.2     9      11.0        12          107
06:40      11.0     8       9.0        10          119
Some explanation, the WS80 figures from CMX are both rounded to integers, and they are a snapshot at the time of the one minute interval - the WS80 updates the solar value every 4.75 seconds and it does bounce around a bit when you think the sun is constant, so I think there is some noise in their readings.

The WS80 avg and mx values are from the 5 snapshots in that 5 minute block.

I think I conclude that ecowitt.net is recording the maximum value within the 5 minutes. This is naturally going to skew the figures higher than snapshots which will be randomly distributed above or below the average.

What I do not know is what the GW1000 sends every minute to ecowitt.net. Is it the solar value at the end of that minute, or the maximum during that minute? If the latter then if you get one high reading for a few seconds, the whole 5 minutes is recorded as that high value.

If the figure on ecowitt.net is the maximum, they that will also skew any calculated sunshine hours higher than it should be too.

I think my conclusion is that if you want more accurate sunshine hours then you need to leave CMX running as the data from ecowitt.net is going to give higher values if they are in the region of the sunshine threshold.
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

I completely agree about leaving it running 24x7 which is what I've always done with my live system. However what I see does not completely explain why when Paul posted his catch up sunshine hours data from his VP2/WLL catch up which is also recorded at 5 minute intervals I understand and seems to agree with his system which runs 24x7. Does WLL do something different with the data or is it actually 1 minute intervals (which would make sense since the values are so accurate).

I agree with your comments about sunrise time being very difficult as the values are so low percentages do not help in this case.

Stuart
User avatar
mcrossley
Posts: 12641
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by mcrossley »

The WLL records the solar value at the end of the period rather than the maximum within the period. Which as I said, assuming the value is changing around the threshold value, will average out with some above, some below the threshold. The Ecowitt value will always be above.
broadstairs
Posts: 731
Joined: Thu 14 Aug 2008 7:17 am
Weather Station: Ecowitt GW1003/GW1103/GW2000
Operating System: Windows 7 and Linux
Location: Broadstairs, Kent, UK
Contact:

Re: Potential issue with sunshine hours when loading from Archive

Post by broadstairs »

In which case would it be better to take the mean of two adjacent 5 minute readings and use that to decide? Might it improve the result?

Stuart
Post Reply