Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.11.4 (build 3133) - 25 May 2021
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

If you are interested in supporting Cumulus then maybe you would like to contribute to the maintenance of the Wiki? This need not take hours of your time - just a half hour here and there can make a big difference, particularly if many people are doing this. Any contributions are very welcome, whether they involve new content or editing of existing content. It will be very helpful to current and future users of Cumulus software if the Wiki is kept well-maintained and current. If you are interested then please contact forum user saratogaWX and ask for a Wiki account.

On the ChartsCompiler

Discussion of the Cumulusutils tool and website generator.

Moderator: HansR

Post Reply
User avatar
HansR
Posts: 2199
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

On the ChartsCompiler

Post by HansR »

This post is about the ChartsCompiler.

The idea of the chart compiler came from the many questions for little variations in charts which lead to new data (files / usage), new parameters and additional sometimes configurable code. All very well, but when talking about automation you need to search the reusable parts and see what needs to be done to generalise it. So that is what the ChartsCompiler is about: a too for creating charts for websites which use CumulusMX as a basis in a generalised way.

Beside having a possibility to create charts, it is also a tool to be more efficient with data. Recent charts - e.g. Growing Degree Days - actually only require temperature and not additional data files. Basically every chart which plots derivatives can be calculated where only the basic measurements from the station are uploaded to the site.

Now, calculating all - almost all - charts is an ideal situation and there are many reasons just to upload the datafiles and plot the charts. Nobody feels the pain. But I see the data increasingly going back and forth to and from the website servers and I simply like trying to get that data load down.

So: easy user driven chart definition and reduction of data transfers are the goals.

We'll see how far I get. The basics are here and the tool is laid out in two Wiki articles: the ChartsCompiler and the CDL (Chart Definition Language). The CDL is still under development as is the compiler itself so changes will be fast and many.

It is implemented simply as a module of CumulusUtils and the output can easily be tested just by renaming the .txt output to .html and load it in the browser (make sure the required datafiles are present where you test it). This way it can be used as a stand alone tool (you need to use the parameters GeneratejQueryInclude
DoLibraryIncludes (true to use the output standalone).

Have fun.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3132 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

krash
Posts: 17
Joined: Tue 15 Dec 2015 7:39 am
Weather Station: PCE-FWS 20
Operating System: Raspbian GNU/Linux 10 (buster)
Location: Madrid

Re: On the ChartsCompiler

Post by krash »

Dear Hans...
I have been reading a bit of the CDL and touching things in my charts, first of all I would like to give you my congratulations for all the work you have done. In particular I have seen the SUM function of EVAL, could not another function that is the arithmetic mean be done? It is only a proposition, for now I will continue investigating and testing ..
Oh by the way, I have passed the CompileOnly option and it turns out that if you update the graphics part but it leaves me the web without the wind rose or the wind direction rose it appears blank........ I don't know ... thank you very much, friend.

User avatar
HansR
Posts: 2199
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On the ChartsCompiler

Post by HansR »

@krash:
First, thanks for the appreciation and trying things out. That really helps :) Don't apologise for an inconvenience, it is not!

I will try and answer your questions (as in the post and your follow up PM's) point by point below. As I pointed out: things are very much developing and I need to find a way to use the variables from the datafiles and the way to use those in the functions I intend to offer. So be prepared for changes!
  1. Using the CompileOnly function and using the output into the website means that you are actually using the inclusion libraries of HighCharts and jQuery as taken care of by the Website Generator. That is a very OK way to use the compiler, making it easier to try out the features (the other way is to use the compiler output standalone). But to do so, you need to set the following parameters (section [General]) to false: GeneratejQueryInclude and DoLibraryIncludes. I assume your problems with the windrose will disappear. If not, let me know.
  2. Arithmentic mean: For time series the arithmetic mean lead to just one value and as such it is not plottable, so the arithmetic mean won't be available to the user (though I might need it internally e.g. for calculating the stddev). For creating a chart there does not seem to be a reason to make this available unless otherwise argued. And YES, the standard deviation will become available at some point.
  3. With respect to your observation that the OUTPUT specifier is not working: could you send me your CutilsCharts.def that contains the specification? The one you sent me does not.
  4. I see you discovered the SMA statistic ;) and maybe that is what you want with the arithmetic mean (point 1). However you want to apply that to the RECENT chart for the pressure and that is not available because STATS is currently not available for the RECENT data. I will look into that but I am not sure a generic solution would be available with the RECENT data and if there is, it will require quite some additional work so don't expect that on short term. What I am looking into is - for all MAX/MIN values in the Dayfile - to create an estimator for the average of such parameter. Temperature is the only measurement to store both MAX, MIN and AVG to make that immediately plottable. The other parameters like Pressure, an average needs to be calculated to be able to plot it. That might be an idea for the EVAL construction but currently only one Plotvariable can be used in the EVAL equation. You can of course use SMA on the MIN or the MAX value, but I assume you just want the average line through the column graph of the pressure (which uses the MIN/MAX) just like my demo for temperature.
NOTE: the problem I need to find a solution for is the naming (source, test and validity) of the Plotvariables which are not in the datafiles (so the EVAL or STATS variables).

Thanks for trying and your remarks! It's helpful.
You can check the current work here (I will provide a caption work in progress after a release).

Regards,
Hans

https://meteo-wagenborgen.nl
Cumulus build 3132 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
meteo19
Posts: 33
Joined: Tue 03 May 2016 8:13 pm
Weather Station: Davis VP2
Operating System: Microsoft Windows 7 Pro
Location: Ségur-Le-Château, France
Contact:

Re: On the ChartsCompiler

Post by meteo19 »

Salut hans

I had more time this weekend to work with Utils you did a huge job easy to set up you can look at what I achieve on my site.
http://www.meteosegur.fr/CU_MX/utils/
And on the menu + and look at the new charts.
One question you think you install additional sensors like soil temperature or soil moisture and foliage.
Well done for everything and thank you for your great application.Have a nice day and sorry for my very bad English. :groan:

Well cordially,

Patrick

User avatar
HansR
Posts: 2199
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On the ChartsCompiler

Post by HansR »

meteo19 wrote:
Fri 28 May 2021 6:42 am
Salut hans

I had more time this weekend to work with Utils you did a huge job easy to set up you can look at what I achieve on my site.
http://www.meteosegur.fr/CU_MX/utils/
And on the menu + and look at the new charts.
One question you think you install additional sensors like soil temperature or soil moisture and foliage.
Well done for everything and thank you for your great application.Have a nice day and sorry for my very bad English. :groan:

Well cordially,

Patrick
Salut Patrick,

Many thanks for the appreciation and your English is fine :D
I checked out your site and the + menu and indeed it looks great to see it all operational like this.
One of the finest Utils sites btw. with the colouring and use of almost all possibilities in the menu. Inspiring!

With respect to the additional sensors (and AirQuality of other sensors as well) that is kind of a headache dossier. I have been looking at it but have not been able to find an efficient solution to handle that (in a generic manner). I will keep thinking about it but don't expect something soon. I have to dive deeper in the extra sensor technique and possibilities. Currently the focus is on the ChartsCompiler. With summer coming and the Corona spring releasing (je pense en FR: ressorts de Corona) it will take some time. But yes, some day I will have to I guess ;)

Thanks again for the complements.

Best regards,
Hans

https://meteo-wagenborgen.nl
Cumulus build 3132 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
HansR
Posts: 2199
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On the ChartsCompiler

Post by HansR »

To show a demo of what the compiler may offer, I created a chart, graphing the dewpoint as coming from the Davis and two calculations (aproximation 1 and 2) and Humidity in one chart with this Chart Definition. Approximation 1 is the calculation as done by CMX.

Code: Select all

Equations
; Actual Water Vapour Pressure
  test1 EVAL [ Humidity/100 * 6.112 * EXP(17.62*Temperature/(243.12+Temperature)) ]

; Documentation:
; https://en.wikipedia.org/wiki/Dew_point
; https://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html
;
Chart Dewpoint Title Dewpoint Calculations in CDL
  Plot Humidity
  Plot Dewpoint
  Plot Approximation1 EVAL [ (243.12 * LN(test1) - 440.1) / (19.43 - LN(test1)) ] Axis Temp
  Plot Approximation2 EVAL [ Temperature - ((100 - Humidity)/5) ] Axis Temp
;  STATS Dewpoint SMA colour green
EndChart
Which produces this chart:
Schermafbeelding 2021-06-17 081811.jpg
Nice confirmation of what was to be expected :)
  1. When RH > 50% difference is 0.1 to 0.5 degree for Approximation 2
    When RH < 50% the difference grows fast to 1.5 degree for RH 48%
  2. The Davis produces the same value as the CMX calculation 8-)
You do not have the required permissions to view the files attached to this post.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3132 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

User avatar
HansR
Posts: 2199
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On the ChartsCompiler

Post by HansR »

A remark needs to be made on the columnrange plots. The plotvariables which can be plotted as columnrange have only min/max stored. It is only the temperature which has the average available from the CMX data.

To mimic the average line through the range bars I suggest to use (min + max)/2 estimator. That may not be correct but it may be close in most cases. The chart then becomes:

Code: Select all

CHART DaylyBaro TITLE Dayly Barometer with rangecolumn
  PLOT DAILY MinBarometer colour Crimson As ColumnRange
  PLOT DAILY AverageBarometer EVAL [ (MinBarometer + MaxBarometer)/2 ] colour black As spline Axis Pressure
EndChart
This leads to the following chart:
    Schermafbeelding 2021-06-25 132945.jpg
    You do not have the required permissions to view the files attached to this post.
    Hans

    https://meteo-wagenborgen.nl
    Cumulus build 3132 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

    User avatar
    HansR
    Posts: 2199
    Joined: Sat 20 Oct 2012 6:53 am
    Weather Station: Davis Vantage Pro 2+
    Operating System: Raspbian GNU/Linux 10 (Buster)
    Location: Wagenborgen (NL)
    Contact:

    Re: On the ChartsCompiler

    Post by HansR »

    meteo19 wrote:
    Fri 28 May 2021 6:42 am
    Salut hans

    I had more time this weekend to work with Utils you did a huge job easy to set up you can look at what I achieve on my site.
    http://www.meteosegur.fr/CU_MX/utils/
    And on the menu + and look at the new charts.
    One question you think you install additional sensors like soil temperature or soil moisture and foliage.
    Well done for everything and thank you for your great application.Have a nice day and sorry for my very bad English. :groan:

    Well cordially,

    Patrick
    Salut Patrick,

    Could you send me more specifications of what you would like to see in Cutils with respect to additional sensors?

    As new functionality in CMX is coming up on this it will make this more interesting for Cutils to support additional sensors, so I will dive into this.
    You can it to send me (in Franch) by PM or you can add it to the Change Request thread (in English, preferred)

    Regards,
    Hans
    Hans

    https://meteo-wagenborgen.nl
    Cumulus build 3132 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.17-v7+ ● Mono 5.18.0.240

    Post Reply