Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

On the ChartsCompiler

Discussion of the Cumulusutils tool and website generator.

Moderator: HansR

Post Reply
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

On the ChartsCompiler

Post by HansR »

This post is about the ChartsCompiler.

The idea of the chart compiler came from the many questions for little variations in charts which lead to new data (files / usage), new parameters and additional sometimes configurable code. All very well, but when talking about automation you need to search the reusable parts and see what needs to be done to generalise it. So that is what the ChartsCompiler is about: a tool for creating charts for websites which use CumulusMX as a basis in a generalised way.

Beside having a possibility to create charts, it is also a tool to be more efficient with data. Recent charts - e.g. Growing Degree Days - actually only require temperature and not additional data files. Basically every chart which plots derivatives can be calculated where only the basic measurements from the station are uploaded to the site.

Now, calculating all - almost all - charts is an ideal situation and there are many reasons just to upload the datafiles and plot the charts. Nobody feels the pain. But I see the data increasingly going back and forth to and from the website servers and I simply like trying to get that data load down.

So: easy user driven chart definition and reduction of data transfers are the goals.

We'll see how far I get. The basics are here and the tool is laid out in two Wiki articles: the ChartsCompiler and the CDL (Chart Definition Language). The CDL is still under development as is the compiler itself so changes will be fast and many.

It is implemented simply as a module of CumulusUtils and the output can easily be tested just by renaming the .txt output to .html and load it in the browser (make sure the required datafiles are present where you test it). This way it can be used as a stand alone tool. You need to use the parameters GeneratejQueryInclude and DoLibraryIncludes (true to use the output standalone).

[EDIT 24/9/21]:
Now, after the completion of the ExtraSensors module, the ChartsCompiler has become a pivotal module within the ecology of CumulusUtils as the charts for the ExtraSensors are created by writing out CDL code to the CutilsCharts.def file. This means that you can not only plot the ExtraSensor values but add other sensors to the same chart. So it is possible to chart e.g. Soil Temperature against Environment temperature from the standard weather station. Extra Sensors and regular measurements all have the datarange RECENT and therefore can be combined. This gives great possibilities in combining plot variables.

The important thing to notice is that you can create and use charts with the ChartsCompiler on a standalone basis and use those charts in any website or standalone.

Have fun.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
krash
Posts: 32
Joined: Tue 15 Dec 2015 7:39 am
Weather Station: PCE-FWS 20
Operating System: Raspbian GNU/Linux 10 (buster)
Location: Madrid

Re: On the ChartsCompiler

Post by krash »

Dear Hans...
I have been reading a bit of the CDL and touching things in my charts, first of all I would like to give you my congratulations for all the work you have done. In particular I have seen the SUM function of EVAL, could not another function that is the arithmetic mean be done? It is only a proposition, for now I will continue investigating and testing ..
Oh by the way, I have passed the CompileOnly option and it turns out that if you update the graphics part but it leaves me the web without the wind rose or the wind direction rose it appears blank........ I don't know ... thank you very much, friend.
-----MADRID (SPAIN)--------
http://cuatrotorres.ddns.net/
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: On the ChartsCompiler

Post by HansR »

@krash:
First, thanks for the appreciation and trying things out. That really helps :) Don't apologise for an inconvenience, it is not!

I will try and answer your questions (as in the post and your follow up PM's) point by point below. As I pointed out: things are very much developing and I need to find a way to use the variables from the datafiles and the way to use those in the functions I intend to offer. So be prepared for changes!
  1. Using the CompileOnly function and using the output into the website means that you are actually using the inclusion libraries of HighCharts and jQuery as taken care of by the Website Generator. That is a very OK way to use the compiler, making it easier to try out the features (the other way is to use the compiler output standalone). But to do so, you need to set the following parameters (section [General]) to false: GeneratejQueryInclude and DoLibraryIncludes. I assume your problems with the windrose will disappear. If not, let me know.
  2. Arithmentic mean: For time series the arithmetic mean lead to just one value and as such it is not plottable, so the arithmetic mean won't be available to the user (though I might need it internally e.g. for calculating the stddev). For creating a chart there does not seem to be a reason to make this available unless otherwise argued. And YES, the standard deviation will become available at some point.
  3. With respect to your observation that the OUTPUT specifier is not working: could you send me your CutilsCharts.def that contains the specification? The one you sent me does not.
  4. I see you discovered the SMA statistic ;) and maybe that is what you want with the arithmetic mean (point 1). However you want to apply that to the RECENT chart for the pressure and that is not available because STATS is currently not available for the RECENT data. I will look into that but I am not sure a generic solution would be available with the RECENT data and if there is, it will require quite some additional work so don't expect that on short term. What I am looking into is - for all MAX/MIN values in the Dayfile - to create an estimator for the average of such parameter. Temperature is the only measurement to store both MAX, MIN and AVG to make that immediately plottable. The other parameters like Pressure, an average needs to be calculated to be able to plot it. That might be an idea for the EVAL construction but currently only one Plotvariable can be used in the EVAL equation. You can of course use SMA on the MIN or the MAX value, but I assume you just want the average line through the column graph of the pressure (which uses the MIN/MAX) just like my demo for temperature.
NOTE: the problem I need to find a solution for is the naming (source, test and validity) of the Plotvariables which are not in the datafiles (so the EVAL or STATS variables).

Thanks for trying and your remarks! It's helpful.
You can check the current work here (I will provide a caption work in progress after a release).

Regards,
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: On the ChartsCompiler

Post by HansR »

To show a demo of what the compiler may offer, I created a chart, graphing the dewpoint as coming from the Davis and two calculations (aproximation 1 and 2) and Humidity in one chart with this Chart Definition. Approximation 1 is the calculation as done by CMX.

Code: Select all

Equations
; Actual Water Vapour Pressure
  test1 EVAL [ Humidity/100 * 6.112 * EXP(17.62*Temperature/(243.12+Temperature)) ]

; Documentation:
; https://en.wikipedia.org/wiki/Dew_point
; https://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html
;
Chart Dewpoint Title Dewpoint Calculations in CDL
  Plot Humidity
  Plot Dewpoint
  Plot Approximation1 EVAL [ (243.12 * LN(test1) - 440.1) / (19.43 - LN(test1)) ] Axis Temp
  Plot Approximation2 EVAL [ Temperature - ((100 - Humidity)/5) ] Axis Temp
;  STATS Dewpoint SMA colour green
EndChart
Which produces this chart:
Schermafbeelding 2021-06-17 081811.jpg
Nice confirmation of what was to be expected :)
  1. When RH > 50% difference is 0.1 to 0.5 degree for Approximation 2
    When RH < 50% the difference grows fast to 1.5 degree for RH 48%
  2. The Davis produces the same value as the CMX calculation 8-)
You do not have the required permissions to view the files attached to this post.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: On the ChartsCompiler

Post by HansR »

A remark needs to be made on the columnrange plots. The plotvariables which can be plotted as columnrange have only min/max stored. It is only the temperature which has the average available from the CMX data.

To mimic the average line through the range bars I suggest to use (min + max)/2 estimator. That may not be correct but it may be close in most cases. The chart then becomes:

Code: Select all

CHART DaylyBaro TITLE Dayly Barometer with rangecolumn
  PLOT DAILY MinBarometer colour Crimson As ColumnRange
  PLOT DAILY AverageBarometer EVAL [ (MinBarometer + MaxBarometer)/2 ] colour black As spline Axis Pressure
EndChart
This leads to the following chart:
    Schermafbeelding 2021-06-25 132945.jpg
    You do not have the required permissions to view the files attached to this post.
    Hans

    https://meteo-wagenborgen.nl
    CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
    User avatar
    HansR
    Posts: 5870
    Joined: Sat 20 Oct 2012 6:53 am
    Weather Station: GW1100 (WS80/WH40)
    Operating System: Raspberry OS/Bullseye
    Location: Wagenborgen (NL)
    Contact:

    Re: On the ChartsCompiler

    Post by HansR »

    A nice example of the ChartsCompiler is its use in the context of ExtraSensors.

    By default the ExtraSensors are grouped by CumulusUtils the same way as they are grouped by CMX (see the extra sensor page in the management interface). This means that e.g. Temperatures from sensors on non-related places will be in the same chart while the corresponding dewpoints are in another chart. That makes the charts less useful. As the ChartsCompiler is intrinsically coupled to the ExtraSensors and the charts definitions are written to the CutilsCharts.def, you may modify the charts after the initial generation (or even create the charts from scratch). To do so set the inifile parameter:

    Code: Select all

    UserModificationExtraSensorCharts=true
    This prevents generating/overwriting when the next ExtraSensors command is run. Always remember this if you start changing your ExtraSensor configuration: you either must change the charts yourself or set this to false again and redo everything from the start.

    So as an example I will show what I did in my own configuration (see my site, menu=>Extra=>Extra Sensors). I have two WH31 (T/H) sensors, one WN34 (T, probe version) and one WH51 soil moisture sensor (Ecowitt). They are distributed as follows: one WH31 in the attic and WH31, WN34, WH51 outside beside the weather station. The attic is just for explaining.
    1. The outside gives me the possibility to put in one chart the main temperature, temperature at -30 cm in the soil and +10 cm above the soil together with the corresponding RH and DP.
    2. The outside also gives me the possibility to chart the soil moisture together with main temperature, humidity and rainfall
    3. The attic shows its temperature, RH and DP against the outside temperature
    See the figures below (note the sharp drops are restarts of my CMX resulting in absent values), the corresponding code definitions are below the figures. All terms can be translated through the standard language system or are fixed in the chart definitions (the title). Legends are originally taken from the strings.txt file.
    BodemTemperatuur = Soil Temperature
    BodemVochtigheid = Soil Moisture
    Zolder = Attic

    So, in summary: you can chart any extra sensor with any of the main RECENT parameters from your weather station.
      Schermafbeelding 2021-11-14 111317.jpg
      Schermafbeelding 2021-11-14 111351.jpg
      Schermafbeelding 2021-11-14 111428.jpg
      Automatically generated chart definitions:

      Code: Select all

      Chart Temperature Title Trend grafiek van Extra Temperature Sensors
        Plot Extra Temp1
        Plot Extra Temp2
      EndChart Output extrasensorscharts.txt
      
      Chart Humidity Title Trend grafiek van Extra Humidity Sensors
        Plot Extra Humidity1
        Plot Extra Humidity2
      EndChart
      
      Chart DewPoint Title Trend grafiek van Extra DewPoint Sensors
        Plot Extra Dewpoint1
        Plot Extra Dewpoint2
      EndChart
      
      Chart SoilMoisture Title Trend grafiek van Extra SoilMoisture Sensors
        Plot Extra SoilMoisture1
      EndChart
      
      Chart UserTemp Title Trend grafiek van Extra UserTemp Sensors
        Plot Extra UserTemp1
      EndChart
      
      Manually modified chart definitions:

      Code: Select all

      Chart BodemTemperaturen Title Trend grafiek van Extra Buiten Temperatuur Sensors
        PLOT Temperature Colour #0000cc zindex 90
        Plot Extra Temp1
        Plot Extra Humidity1
        Plot Extra Dewpoint1
        Plot Extra UserTemp1
      EndChart Output extrasensorscharts.txt
      
      Chart BodemVochtigheid Title Trend grafiek van Extra Bodem Sensors
        PLOT Temperature Colour #0000cc zindex 90
        Plot Humidity Colour #64E572 zindex 60
        Plot Extra SoilMoisture1 axis free zindex 40
        Plot RainFall As Area colour #33B4FF opacity 0.2 zindex 10
      EndChart
      
      Chart Zolder Title Trend grafiek van Extra Zolder Sensors
        PLOT Temperature Colour #0000cc zindex 90
        Plot Extra Temp2
        Plot Extra Humidity2
        Plot Extra Dewpoint2
      EndChart
      
      
      You do not have the required permissions to view the files attached to this post.
      Hans

      https://meteo-wagenborgen.nl
      CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
      User avatar
      HansR
      Posts: 5870
      Joined: Sat 20 Oct 2012 6:53 am
      Weather Station: GW1100 (WS80/WH40)
      Operating System: Raspberry OS/Bullseye
      Location: Wagenborgen (NL)
      Contact:

      Re: On the ChartsCompiler

      Post by HansR »

      Graphing the Cloud Base
        The cloud base is only present in realtime.txt (afaik) and cannot be plotted because it is not present in the database (it is not being logged).
        The ChartsCompiler is able to overcome this through the EVAL / EQUATION feature.
        Cumulus calculates the Cloud Base for the realtime.txt file according to this Wikipedia reference.

        So if you define:

        Code: Select all

        Equations
        ; Cloud Base definition for °C and meters:
          CloudBase EVAL [ (Temperature - DewPoint) / 2.5 * 304.8 ]  
        
        [...]
        
        Chart CloudBase Title Wolkenhoogte boven zeeniveau ConnectsTo 20
          PLOT CloudBase EVAL [ CloudBase ] Axis Free
        EndChart
        
        Note that you have to add your station height above sea level to this Cloud Base (which for me is zero).

        For three days the charts looks like this (in NE Netherlands on 24/2/2022, check my site):
          Schermafbeelding 2022-02-24 163225.jpg
          You do not have the required permissions to view the files attached to this post.
          Hans

          https://meteo-wagenborgen.nl
          CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
          AndyKF650
          Posts: 704
          Joined: Thu 20 Aug 2020 8:35 am
          Weather Station: Davis VP2 Plus
          Operating System: RPiv4 64bit Linux 12 Bookworm
          Location: Jersey Channel Islands
          Contact:

          Re: On the ChartsCompiler

          Post by AndyKF650 »

          Hi Hans

          That looks good,

          I have tried it on my site and can get the graph working OK but the axis is showing " Dimensionless". I see you have got the Dutch for " Height above Sealevel in Metres ".

          How do you do that? I expect the same solution will apply for my Windrun chart as well.
          User avatar
          HansR
          Posts: 5870
          Joined: Sat 20 Oct 2012 6:53 am
          Weather Station: GW1100 (WS80/WH40)
          Operating System: Raspberry OS/Bullseye
          Location: Wagenborgen (NL)
          Contact:

          Re: On the ChartsCompiler

          Post by HansR »

          Hi Andy,

          How appropriate that you are the first to try :D

          If an axis is 'free' , it is because it is unknown at compile time what the user is plotting. Therefore the axis title and the legend description need to be created by the user himself after compilation. This is done through the language file which has two entries for each 'free' axis after the first run. You create the text wanted and run again.

          I maybe could have done that in the language but that would have created too much complexity.

          [EDIT] I see you found out yourself ;)

          Regards,
          Hans

          https://meteo-wagenborgen.nl
          CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
          Post Reply