Cumulus Support

Posted: **Tue 22 Sep 2020 5:34 am**

Another issue that comes from data in the wild is that CMX does not always write the data to the monthly logs at the prescribed interval.

Below you see the observation of time interval changes of my own data. This is registered while reading the records one by one and checking for interval changes. Any interval change - I set it to one minute in the inifile - leads to a different estimate of solar energy and solar hours. I also observed this in Patricks data and will check some other datasets which I have available (Paul, Javier, Laurent, Steinar). The conclusion is that checking the interval with which the data is written must be checked record by record to actually get a true estimate of solar duration and energy.

There are some questions/conclusions involved here:

How is it possible that the interval is not constant (believe me, I never changed it). I see start/stop as cause but there also other unexplained moments.
The gaps you see can be caused by maintenance, manual editing and power outages etc... and are considered unavoidable
The necessity to check for interval changes has a severe performance - unacceptable? - effect in cutils, I need to think this over and maybe abandon the solar exercise.

Especially the last effect may lead to the conclusion that it is not possible to do the solar graphs with the design of cutils as I made it combined with the irregular datastorage of CMX. And btw: some of the interval changes have to do with regular and automated starts and stops because of backups. That apparently has been much improved because of the use of systemd (no interval changes since 18/8).

Comments are welcome.

Code: Select all

2020-09-22 07:08:11.209 Solar Graphs - CreateListsFromMonthlyLogs start
2020-09-22 07:08:11.946 Solar Graphs - CreateListsFromMonthlyLogs DataLogInterval = 1
2020-09-22 07:08:11.947 Solar Graphs - CreateListsFromMonthlyLogs SunThreshold = 0,75
2020-09-22 07:08:25.199 Solar Graphs - DataLogInterval change at 18-7-2019 09:00:00 - Old: 1 New: 2
2020-09-22 07:08:25.199 Solar Graphs - DataLogInterval change at 18-7-2019 09:02:00 - Old: 2 New: 1
2020-09-22 07:08:29.341 Solar Graphs - DataLogInterval change at 30-7-2019 09:00:00 - Old: 1 New: 2
2020-09-22 07:08:29.341 Solar Graphs - DataLogInterval change at 30-7-2019 09:02:00 - Old: 2 New: 1
2020-09-22 07:08:29.686 Solar Graphs - DataLogInterval change at 31-7-2019 09:00:00 - Old: 1 New: 2
2020-09-22 07:08:29.686 Solar Graphs - DataLogInterval change at 31-7-2019 09:02:00 - Old: 2 New: 1
2020-09-22 07:08:32.775 Solar Graphs - DataLogInterval change at 9-8-2019 06:38:00 - Old: 1 New: 2
2020-09-22 07:08:32.775 Solar Graphs - DataLogInterval change at 9-8-2019 06:40:00 - Old: 2 New: 1
2020-09-22 07:08:34.101 Solar Graphs - DataLogInterval change at 13-8-2019 12:16:00 - Old: 1 New: 2
2020-09-22 07:08:34.101 Solar Graphs - DataLogInterval change at 13-8-2019 12:18:00 - Old: 2 New: 1
2020-09-22 07:08:34.777 Solar Graphs - DataLogInterval change at 15-8-2019 08:25:00 - Old: 1 New: 22
2020-09-22 07:08:34.777 Solar Graphs - DataLogInterval change at 15-8-2019 08:47:00 - Old: 22 New: 1
2020-09-22 07:08:41.526 Solar Graphs - DataLogInterval change at 4-9-2019 11:08:00 - Old: 1 New: 2
2020-09-22 07:08:41.526 Solar Graphs - DataLogInterval change at 4-9-2019 11:10:00 - Old: 2 New: 1
2020-09-22 07:08:52.024 Solar Graphs - DataLogInterval change at 5-10-2019 09:00:00 - Old: 1 New: 2
2020-09-22 07:08:52.024 Solar Graphs - DataLogInterval change at 5-10-2019 09:02:00 - Old: 2 New: 1
2020-09-22 07:08:55.415 Solar Graphs - DataLogInterval change at 15-10-2019 12:17:00 - Old: 1 New: 2
2020-09-22 07:08:55.416 Solar Graphs - DataLogInterval change at 15-10-2019 12:19:00 - Old: 2 New: 1
2020-09-22 07:09:14.098 Solar Graphs - DataLogInterval change at 9-12-2019 13:58:00 - Old: 1 New: 2
2020-09-22 07:09:14.098 Solar Graphs - DataLogInterval change at 9-12-2019 14:00:00 - Old: 2 New: 1
2020-09-22 07:09:24.645 Solar Graphs - DataLogInterval change at 9-1-2020 09:16:00 - Old: 1 New: 2
2020-09-22 07:09:24.646 Solar Graphs - DataLogInterval change at 9-1-2020 09:18:00 - Old: 2 New: 1
2020-09-22 07:09:33.549 Solar Graphs - DataLogInterval change at 4-2-2020 09:00:00 - Old: 1 New: 2
2020-09-22 07:09:33.549 Solar Graphs - DataLogInterval change at 4-2-2020 09:02:00 - Old: 2 New: 1
2020-09-22 07:09:56.761 Solar Graphs - DataLogInterval change at 12-4-2020 09:00:00 - Old: 1 New: 2
2020-09-22 07:09:56.761 Solar Graphs - DataLogInterval change at 12-4-2020 09:02:00 - Old: 2 New: 1
2020-09-22 07:10:00.845 Solar Graphs - DataLogInterval change at 24-4-2020 09:00:00 - Old: 1 New: 2
2020-09-22 07:10:00.845 Solar Graphs - DataLogInterval change at 24-4-2020 09:02:00 - Old: 2 New: 1
2020-09-22 07:10:04.243 Solar Graphs - DataLogInterval change at 4-5-2020 16:49:00 - Old: 1 New: 2
2020-09-22 07:10:04.244 Solar Graphs - DataLogInterval change at 4-5-2020 16:51:00 - Old: 2 New: 1
2020-09-22 07:10:04.583 Solar Graphs - DataLogInterval change at 5-5-2020 08:09:00 - Old: 1 New: 2
2020-09-22 07:10:04.584 Solar Graphs - DataLogInterval change at 5-5-2020 08:11:00 - Old: 2 New: 1
2020-09-22 07:10:06.622 Solar Graphs - DataLogInterval change or data gap too large = 47
2020-09-22 07:10:06.622 Solar Graphs - DataLogInterval change Skip record
2020-09-22 07:10:14.448 Solar Graphs - DataLogInterval change at 3-6-2020 17:10:00 - Old: 1 New: 24
2020-09-22 07:10:14.449 Solar Graphs - DataLogInterval change at 3-6-2020 17:34:00 - Old: 24 New: 1
2020-09-22 07:10:16.504 Solar Graphs - DataLogInterval change or data gap too large = 34
2020-09-22 07:10:16.504 Solar Graphs - DataLogInterval change Skip record
2020-09-22 07:10:20.930 Solar Graphs - DataLogInterval change at 22-6-2020 09:00:00 - Old: 1 New: 2
2020-09-22 07:10:20.930 Solar Graphs - DataLogInterval change at 22-6-2020 09:02:00 - Old: 2 New: 1
2020-09-22 07:10:22.284 Solar Graphs - DataLogInterval change at 26-6-2020 16:06:00 - Old: 1 New: 5
2020-09-22 07:10:22.284 Solar Graphs - DataLogInterval change at 26-6-2020 17:11:00 - Old: 5 New: 1
2020-09-22 07:10:22.284 Solar Graphs - DataLogInterval change or data gap too large = 46
2020-09-22 07:10:22.284 Solar Graphs - DataLogInterval change Skip record
2020-09-22 07:10:22.971 Solar Graphs - DataLogInterval change at 28-6-2020 09:00:00 - Old: 1 New: 2
2020-09-22 07:10:22.971 Solar Graphs - DataLogInterval change at 28-6-2020 09:02:00 - Old: 2 New: 1
2020-09-22 07:10:26.070 Solar Graphs - DataLogInterval change or data gap too large = 35
2020-09-22 07:10:26.079 Solar Graphs - DataLogInterval change Skip record
2020-09-22 07:10:26.424 Solar Graphs - DataLogInterval change at 8-7-2020 17:49:00 - Old: 1 New: 6
2020-09-22 07:10:26.424 Solar Graphs - DataLogInterval change at 8-7-2020 17:55:00 - Old: 6 New: 1
2020-09-22 07:10:26.767 Solar Graphs - DataLogInterval change at 9-7-2020 09:00:00 - Old: 1 New: 2
2020-09-22 07:10:26.767 Solar Graphs - DataLogInterval change at 9-7-2020 09:02:00 - Old: 2 New: 1
2020-09-22 07:10:27.113 Solar Graphs - DataLogInterval change at 10-7-2020 09:00:00 - Old: 1 New: 2
2020-09-22 07:10:27.113 Solar Graphs - DataLogInterval change at 10-7-2020 09:02:00 - Old: 2 New: 1
2020-09-22 07:10:32.310 Solar Graphs - DataLogInterval change at 25-7-2020 19:03:00 - Old: 1 New: 2
2020-09-22 07:10:32.310 Solar Graphs - DataLogInterval change at 25-7-2020 19:05:00 - Old: 2 New: 1
2020-09-22 07:10:33.335 Solar Graphs - DataLogInterval change at 28-7-2020 10:34:00 - Old: 1 New: 2
2020-09-22 07:10:33.335 Solar Graphs - DataLogInterval change at 28-7-2020 10:36:00 - Old: 2 New: 1
2020-09-22 07:10:38.104 Solar Graphs - DataLogInterval change or data gap too large = 37
2020-09-22 07:10:38.112 Solar Graphs - DataLogInterval change Skip record
2020-09-22 07:10:38.453 Solar Graphs - DataLogInterval change at 12-8-2020 14:31:00 - Old: 1 New: 2
2020-09-22 07:10:38.453 Solar Graphs - DataLogInterval change at 12-8-2020 14:33:00 - Old: 2 New: 1
2020-09-22 07:10:38.796 Solar Graphs - DataLogInterval change at 13-8-2020 10:06:00 - Old: 1 New: 2
2020-09-22 07:10:38.796 Solar Graphs - DataLogInterval change at 13-8-2020 10:08:00 - Old: 2 New: 1
2020-09-22 07:10:40.505 Solar Graphs - DataLogInterval change at 18-8-2020 10:34:00 - Old: 1 New: 2
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 10:36:00 - Old: 2 New: 1
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 12:54:00 - Old: 1 New: 4
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 12:58:00 - Old: 4 New: 2
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 13:00:00 - Old: 2 New: 1
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 13:09:00 - Old: 1 New: 2
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 13:11:00 - Old: 2 New: 1
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 13:18:00 - Old: 1 New: 2
2020-09-22 07:10:40.506 Solar Graphs - DataLogInterval change at 18-8-2020 13:20:00 - Old: 2 New: 1
2020-09-22 07:10:48.708 Solar Graphs - DataLogInterval change at 11-9-2020 09:01:00 - Old: 1 New: 0
2020-09-22 07:10:48.708 Solar Graphs - DataLogInterval change at 11-9-2020 09:01:00 - Old: 0 New: 1
2020-09-22 07:10:52.482 Solar Graphs: Timing of CreateListsFromMonthlyLogs = 161272 ms
2020-09-22 07:10:52.482 Solar Graphs - CreateListsFromMonthlyLogs stop

Posted: **Tue 22 Sep 2020 5:45 am**

HansR wrote: ↑Mon 21 Sep 2020 12:10 pm The above may also have an effect on the wind sampling used for the Wind Rose btw, if so I'll get back on that one.

I looked at the effect of the irregular time interval on the wind rose and wind run graphs.
I use counting the occurrence of bearing/speed combinations as a statistical result, so the count results in the percentage of total. It is clear that the irregular sampling distance has some effect, I assume it is not very large given the natural variability of the wind. So I will not change the wind graphs because of the irregularities in the datafiles.

Posted: **Tue 22 Sep 2020 8:20 am**

The performance issue may have been solved for a large part by doing some selection of data on basis of date, a bit (lot

) more clever than I did originally.

Posted: **Tue 22 Sep 2020 1:45 pm**

Apparently the data from the B-L Sunrecorder has not been stored

Cumulus reads the B-L file for whether the sun is shining and the cumulative hours of sunshine in the day. Cumulus does not gather or save any other information from the B-L. The B-L sunrecorder program itself retains its own data files and graphs.

The B-L Sunrecorder was started in June 2012 and the sunhours are logged in Cumulus.
I upgraded from VP2 to VP2 Plus on May 6, 2018, and that time I added logging solar and UV from the VP2 Plus and that is the start of this data in Cumulus. Sunhours were still being logged in Cumulus from the B-L data.
Unfortunately when moving to a new home in June 2019 I was not able to get the B-L installed so have it in a box. From that time sunhours are determined and logged in CumulusMX from the VP2 Plus.

I have gone through different logging interval over the years, starting at 30 minutes when first using Weatherlink and then also Cumulus in 2008, changing to 15 minutes a while later and then 10 minutes, and 5 minutes. When I started to use the WiFiLogger and sending data to Globe (Globe will only accept 15 minute interval) I had to set the logger interval to 15 minutes and kept CumulusMX to 5 minutes. That different logging interval is not ideal and at CumulusMX startup it reads the whole datalogger archive. I have changed CumulusMXutils to 15 minutes so it is the same as the WiFiLogger. I know it has caused some difficulty when Cumulusutils reads the data files with several different intervals, sorry, but it is what it is!

Enjoy,
Paul

Posted: **Tue 22 Sep 2020 2:55 pm**

New version installed, and all seems ok www.komokaweather.com/cumulusutils/index.html
In the top menu tabs select Graphs - Solar Graphs - and pulldown bar for the four new additions.

Enjoy,
Paul

Posted: **Tue 22 Sep 2020 3:01 pm**

@Paul: OK, if data isn't there, there's nothing I can do. Maybe in future I'll take the sunhours from the dayfile iso calculating it through the logfiles but even then: how do I know? Secondary sensors are quite unclear for me, anyway, it will be solved in the future I guess.

PaulMy wrote: ↑Tue 22 Sep 2020 1:45 pm I have changed CumulusMXutils to 15 minutes so it is the same as the WiFiLogger. I know it has caused some difficulty when Cumulusutils reads the data files with several different intervals, sorry, but it is what it is!

Noo need to apologise, indeed, it is what it is. Anyway, I think I made a generic solution which runs fine (I hope). And even if you did not change the interval, my solution would have been necessary anyway because small data gaps appear always apparently, which require a solution. It now is as accurate as possible and the problems with the datafiles are there to be solved.

Posted: **Tue 22 Sep 2020 3:37 pm**

Hi Hans
Everything seems to be working fine. I will look at this in detail in the evening.Thank you.

Regards,
Patrick

http://www.meteosegur.fr/CU_MX/utils/index.html

Posted: **Tue 22 Sep 2020 7:25 pm**

Now if the new version seems to work well:

http://kocher.es/cumulusMX/FWI

Saludos

Posted: **Tue 22 Sep 2020 8:15 pm**

@meteo19, @kocher:

Great, any remarks are welcome.

I must say the result are more surprising than I thought.
Interesting result with all of you with the effects of clouds, and atmospheric status.
More variation than I thought.

Posted: **Wed 23 Sep 2020 8:59 am**

@rogerthn: you publish some energy data on your site.
I see some discrepancy with the actual solar energy data measured by your sensor (the cutils graphs), the values in your table (up to 50 kWh/day) and the values I found on this map (found here). Do I misread your table?
(correct for values/day and values/year)

Annual-solar-irradiation-in-Europe-JRC-2014.jpg

Posted: **Wed 23 Sep 2020 9:29 am**

The values on monthly is from my Inverter using "modbus python".
I do have yearly and daily as well
You can also check on SolarEdge
Hence, something very different from values in cutils

Posted: **Wed 23 Sep 2020 10:39 am**

rogerthn wrote: ↑Wed 23 Sep 2020 9:29 am Hence, something very different from values in cutils

Well, that is why I did the map (the scientific values).
In our region it is roughly average 3 kWh per day (1200/year) while you come up with insolation of 30 kWh/day (September number).
My conclusion is, that the cutils value (average) is pretty good given the science results so my question is: where does that daily number in your table come from?
Is it produced by your solar panel supplier application?

I assume the last link shows your grid on your house? How many square meters are those panels (10m2 my guess)?

Posted: **Wed 23 Sep 2020 11:13 am**

HansR wrote: ↑Wed 23 Sep 2020 10:39 am Is it produced by your solar panel supplier application?

The numbers do come from my very own scripts and are stored in MariaDB

The values on SolarEdge is updated at 15 minute intervals while mine uses 10 second interval

HansR wrote: ↑Wed 23 Sep 2020 10:39 am I assume the last link shows your grid on your house? How many square meters are those panels (10m2 my guess)?

Total on my pages are "bought" energy.
Since I do not have a "Smart Meter" (on the to be fixed list) I do not have real time values on sold energy
30 x Longi 305W Mono PERC ~49m2 in SSW and 20° roof angle

Posted: **Wed 23 Sep 2020 11:57 am**

rogerthn wrote: ↑Wed 23 Sep 2020 11:13 am 30 x Longi 305W Mono PERC ~49m2 in SSW and 20° roof angle

49 m2 btw is quite a surface!

OK, so the solar values given are roughly 1 kWh/m2 (average)

I have to change the y-axis description form "Daily kilo Watt hour" to "Daily kilo Watt hour per m2" for clarity.
Consider it done, next release (it's only the default, your transalation has to be done by you so can be done now).

Posted: **Wed 23 Sep 2020 7:43 pm**

HansR wrote: ↑Wed 23 Sep 2020 11:57 am 49 m2 btw is quite a surface!
OK, so the solar values given are roughly 1 kWh/m2 (average)

Yes, roughly 1 kWh/m2 average
There is one more factor to consider, my inverter is a SE7k i.e. 7kW and at 7.4kW it does "clipp" to prevent inverter overload.
My max production, 67 kWh is from the 14th of June with unknown kWh clipping but "undersized" inverter is considered to be best practices since they are said to have better efficiency at lower production and starts earlier.

However, I am more than happy so far

I did start on the 8th of October and the tender did say, estimated yearly production 8600 kWh and this was reached on the 31th of August

It is also said that more than 1000 kWh / kWp (30x305=9.15 kWp) per year is GOOD, this was reached on the 17th of September

HansR wrote: ↑Wed 23 Sep 2020 11:57 am I have to change the y-axis description form "Daily kilo Watt hour" to "Daily kilo Watt hour per m2" for clarity.
Consider it done, next release (it's only the default, your transalation has to be done by you so can be done now).

My "Swedish" WS do not have any solar

Cumulus Support

v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar

Re: v.4.5.0.0 Solar