Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.10.2 (build 3117) - 02 March 2021 (please see announcement regarding releases since 3.5.0)
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of the NOAA report and Snow Index drop-down menus to 2030)

Use this Wiki link to Download the Software (Cumulus MX / Cumulus 1 and other related items).

On Graphs, the corresponding JSON's and bandwidth

Discussion of the Cumulusutils tool and website generator.

Moderator: HansR

Post Reply
User avatar
HansR
Posts: 1874
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

On Graphs, the corresponding JSON's and bandwidth

Post by HansR »

With b3115 of CMX this message has become superfluous.
Please for setup see: Notes on the CMX release b3115/b3116

[EDIT NOTE:] As from CMX 3.9.7 b3107 you can set IncludeGraphDataFiles=0 and follow the advise below. The settings will be somewhat revised in future but that will be along the idea that a list of selectable datafiles will be made giving the opportunity to optimise the bandwidth for your website.

I would like to spend a few words on some recent changes in CumulusMX which have some consequences for the use of CumulusUtils, specifically the bandwidth used.

The standard Cumulus Charts (Recent Charts as they are named in the Menu) are also used in CumulusUtils and can be found under the Home button in the menu. Recently in CumulusMX there have been additions for Historic Charts, AQ charts and the most recent Select-a-chart (which uses the JSON's for the Recent Charts).

All these charts have their own set of JSON data files: eg. the Recent Charts need tempdata.json, solardata.json [etc...], the Historic Charts need the alldailyxxxxdata.json and the AQ data require airquality.json (from the point of view of CumulusMX).

The presence of the datafiles on your website is governed by the parameter IncludeGraphDataFiles=1 [section: FTP Site] of the Cumulus.ini file. This parameter is an all or nothing option.

I asked for a break down in parameters to govern which JSON files to actually send but @mcrossley does not want to do that and it is suggested that if you don't want all datafiles you just switch off the IncludeGraphDataFiles parameter and regulate the required presence of specific JSON files by the Extra Web Files feature of CumulusMX. This process has recently been described in this thread.

So, if you don't mind to waste some bandwidth every 10 or 30 minutes or so, you can stop reading here and leave everything as is.
If you do mind about bandwidth, read on.

CumulusUtils assumed setting of CumulusMX as choices of the user which were reflected in CumulusUtils. With the expansion of CMX and the whole JSON dataset transferred to the website that assumption is no longer correct.

So I will advise to:
  1. Set IncludeGraphDataFiles=0 in Cumulus.ini
  2. Put the datafiles for the Recent Charts in the Extra Web Files section, do not tick Process, do not tick Realtime
  3. Historic charts are not used in CumulusUtils and will probably not be added to CumulusUtils. Not on a short term, maybe never. If you use them for your other websites, remember that the FTP send is done in the UpdateInterval (Cumulus.ini, [section: FTP site]) and cannot be changed. Those files refresh only at rollover time so would only require a once per 24 hr refresh, no optimisation is possible (afaik) see reaction of @mcrossley below.
  4. The AirQuality.json as output by CMX is not used by CumulusUtils, so don't send it unless you have use of it soemwhere else (if you have an AirLink device). The AirQuality JSON's required for CumulusUtils are only sent when configured properly and and only carry the data you wish to see.
  5. The new Select-a-Graph does not require additional JSON files. I had already been thinking of a graph (as a HOME page starter) with multiple parameters in it (compile time selectable) and I may still do that. Maybe I will borrow the dynamic Select-a-Graph feature by Mark, I have not decided yet. Whatever choice, this seems not to require additional advise wrt the JSON datafiles.
If you wish to use all JSON files, just leave IncludeGraphDataFiles=1 in Cumulus.ini and forget the whole thing.

Setting the JSON transfer as indicated above you will definitely save a lot of bandwidth, together with the Thrifty qualifier option (with especially a long period for the Wind and Solar graphs - I use 21 days) CumulusUtils has a very optimised performance both in CPU as in bandwidth.

Sorry for the inconvenience and awkward configuration change.
Please NOTE: The CumulusMX version of the AirQuality data only contains the current concentrations of pm2p5 and pm10 and nothing more. As in CumulusUtils you can chart every AirLink data you wish, it requires its own datafiles which are generated at the same frequency as the CumulusMX JSON files and contain only those data you ask for. As the parameters in the JSON are different from the CumulusMX JSON, the files are not interchangeable. So for CumulusUtils you do NOT need the CMX AirQualiy.json
Please NOTE: the AirQuality data (and other 'Extra' sensor data produced by the Ecowitt GW1000 and similar devices) are not supported by CumulusUtils and I doubt they ever will be.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3116 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.11-v7+ ● 5.18.0.240 (Debian 5.18.0.240)

User avatar
mcrossley
Posts: 7963
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Buster Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by mcrossley »

HansR wrote:
Wed 27 Jan 2021 2:20 pm

*Historic charts are not used in CumulusUtils and will probably not be added to CumulusUtils. Not on a short term, maybe never. If you use them for your other websites, remember that the FTP send is done in the UpdateInterval (Cumulus.ini, [section: FTP site]) and cannot be changed. Those files refresh only at rollover time so would only require a once per 24 hr refresh, no optimisation is possible (afaik).
Extra files has the "End of day" option to only transfer once a day, after rollover.

User avatar
HansR
Posts: 1874
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by HansR »

When was that added?
Hans

https://meteo-wagenborgen.nl
Cumulus build 3116 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.11-v7+ ● 5.18.0.240 (Debian 5.18.0.240)

User avatar
mcrossley
Posts: 7963
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Buster Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by mcrossley »

Build 3047 - Feb 2019.

User avatar
HansR
Posts: 1874
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by HansR »

Stunning :lol:
Hans

https://meteo-wagenborgen.nl
Cumulus build 3116 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.11-v7+ ● 5.18.0.240 (Debian 5.18.0.240)

User avatar
HansR
Posts: 1874
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by HansR »

@mcrossley: the JSON files do not seem to get generated with IncludeGraphDataFiles=0 so can't be found and can't be FTP'ed. How to get them generated?
(And oh, I assume they are / should be in the web-directory).
Hans

https://meteo-wagenborgen.nl
Cumulus build 3116 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.11-v7+ ● 5.18.0.240 (Debian 5.18.0.240)

User avatar
HansR
Posts: 1874
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by HansR »

See NOTE at the top of the first message.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3116 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.11-v7+ ● 5.18.0.240 (Debian 5.18.0.240)

User avatar
HansR
Posts: 1874
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by HansR »

See renewed NOTE in the release post. As from CMX 3.9.7 b3107 you can set IncludeGraphDataFiles=0 and follow the advise in that post to optimise bandwidth.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3116 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.11-v7+ ● 5.18.0.240 (Debian 5.18.0.240)

User avatar
HansR
Posts: 1874
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: Davis Vantage Pro 2+
Operating System: Raspbian GNU/Linux 10 (Buster)
Location: Wagenborgen (NL)
Contact:

Re: On Graphs, the corresponding JSON's and bandwidth

Post by HansR »

This thread has become superfluous because of b3115 of CMX.
Please see note in first message.
Hans

https://meteo-wagenborgen.nl
Cumulus build 3116 ● Davis Vantage Pro 2+ ● RPi 3B+ ● Raspbian 5.10.11-v7+ ● 5.18.0.240 (Debian 5.18.0.240)

Post Reply