Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.5 (build 3282) - 23 February 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

CumulusUtils

Discussion of the Cumulusutils tool and website generator.

Moderator: HansR

Locked
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.5.2 is online

This release is required because of a NOAA module bug at a new month.
Replacement of the executable and one complete run fixes it (or a thrifty run on the 2nd of a month).

version 4.5.2 (2/10/2020)
  1. NOAA: Fixed a bug which generated a new file only the 2 day of the year i.s.o. the second day of the month when under thrifty.
  2. Website: Fixed the last HTML errors on the main site but you won't notice. The website generator is now W3C Validator compatible and should not generate errors.
  3. Website: If Google-Analytics is enabled (an ID is defined in GoogleStatsId) it is now possible to give the user (or just yourself) an opt-out possibility (PermitGoogleOptout=true), see the manual.
  4. Solar and UV are now separately configurable both for both dashboards.
  5. It is now a requirement for use of cutils as a module provider, to use UTF8 character encoding of the page the module is used in. Specifically for the character for degrees and other characters outside the standard range like all language specific characters (the website generator has this standard automatically).
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.5.3 is online

I would say this is a stable version.
The next module to add will be support for air quality (the Davis AirLink), which seems to be added to CMX as a sensor.
When more info is there I will probably add it.

I have currently no new ideas and I do not foresee any changes for additional weather info so unless asked, don't expect other additions soon.

version 4.5.3 (3/10/2020)
  • Now all HTML of the individual modules are W3C Validator compatible (which includes some technical changes which I think have no impact |) ).
  • Added the start of observation date to the record pages.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.5.4 is online

This actually is only a patch release for this minor bug assuming no more bugs will follow :lol:

version 4.5.4 (4/10/2020)
  1. Fixed bug for Current PwsFWI (Standard format) value on the menubar.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.5.5 is online

version 4.5.5 (26/10/2020)
  • Website: Fixed bug for DST switch. Station time now is correct (not sure it is for some pacific/Australian situations where minutes start to count).
  • Monthly logfiles: Fixed bug for existence of AirLink logfiles.
    Now those files are ignored while reading the monthly logfiles.
NOTE: @meteosangonera: testing of the dayfile issue still waiting for the new CMX release, we need some patience. And in addition, your cumulusutils.ini file probably contains a section [website] (even if you don't use that) with parameters ShowUV=true and ShowSolar=true. Accidental they cause a generation of the solar graphs and should be set to false.

NOTE: I was a bit confused by the times some of the sites (v 4.5.4) displayed, because they were correct which algorithmically was not possible. This means that the user must have modified either the local time or the timezone on the machine which runs cumulusutils (or modified the resulting code). Worst case is that I do not understand what is going on which is an option. Another option is that it is a difference between RPi and Windows machines (running CMX/CumulusUtils) in which case I am not sure it is the end of the issue. I prefer getting a message if something is wrong.

In summary the times must always be as follows:
  • Station time is always the actual time of the station with or without DST, in my case UTC -1 (or -2 in DST). This time is always more or less equal to the last update time.
  • Browser time always shows the time of viewer of the site. So, if somebody in the UK is now watching my site, a time of one hour less than my time should be observed (UK is now on UTC)
  • UTC is displayed for reference and is what it is.
Times should be correct always without modifying your computer time.
Automatic and correct time setting on the machine which runs CMX/CumulusUtils is assumed.

RESULT:
The clock on the dashboard for Station Time of the generated website did not react well on DST change back to wintertime in some instances. The 4.5.5 bugfix release was installed on both Windows and RPi's and this showed that the cause is in the difference in handling of timezone differences on Windows and RPI (native .NET and Mono).
This original line of code works in Windows (native .NET) only:

Code: Select all

(TZ.GetUtcOffset(DateTime.Now) - TZ.BaseUtcOffset).Hours
And this fixed line works on native .NET and Mono:

Code: Select all

TZ.GetUtcOffset(DateTime.Now).Hours - TZ.BaseUtcOffset.Hours
Now lets see how this behaves when DST is activated again in March.

This had effect, not only on the clock, but also on the rise/set times of sun and moon.
So: Users with CMX/cutils running on RPi really need the 4.5.5 version.

The documentation shows that the calls used are introduced in various versions of .NET (and thus Mono) so the runtime version installed may also affect the result.
Always use .NET/Mono to the latest possible update level.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Please note that when upgrading CMX to version 3.9.1 (build 3096) there has been a change for the JSON library. The CMX software list states to remove «Newtonsoft.Json.dll».
However, that change is not true if you are using CumulusUtils therefore you need to keep «Newtonsoft.Json.dll» in the Cumulus directory.
It will not have side effects on the operation of the software.

In time I will upgrade CumulusUtils to the the new dll.

(thanks to Patrick for the observation, sorry for the inconvenience)
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.6.0 is online

version 4.6.0 (3/11/2020)
  1. Sysinfo: For the Raspberry Pi the CPU temp is now available.
  2. Air Quality: The first release of the Air Quality module, support for the Davis AirLink sensor and my own FakeAirLink system. Version 4.6 may be regarded as a beta to the AirQuality system as many things have to be optimised and maybe even rewritten. It is working fine though.
The Air Quality module connects to the support of CMX for the AirLink sensor of Davis. I took the liberty of writing an AirLink emulator for the RPi which you can find here. If any of you have legacy sensors I might take a look on how to adjust the interface in a method similar to the FakeAirLink. Let me know if you are interested.

The module should be working for two sensors. However, as I don't have a second sensor I was unable to test so be aware. Best to start with one I guess :D .

Graphing will be added but I don''t know when yet. It depends a bit on the road CMX is taking.

The AirQuality module can be used as module in any website even if it has its own realtime system.
Read the manual on how to configure CMX for the airlinkrealtime.txt file.

If any problem arises, let me know.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

version 4.6.1 is online

version 4.6.1 (4/11/2020)
  1. Website/AirQuality: If no sensor is present, the menu choice 'AirQuality' will not appear in the menu.
  2. AirQuality: The CA handling of the AQI is debatable and in my view incorrect. The AQI display will for the time being be like the US en needs to be set accordingly in CMX.
  3. AirQuality: Fixed a bug when generating the outside sensor. Both in the sensor code as in the airlinkrealtime.txt file (a space too many was generated).
This is a release for AirQuality module to account for a bug concerning outside sensor generation.
Major for AirLink users, minor for others.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.6.2 is online

version 4.6.2 (6/11/2020)
  • Website/AirQuality: Realtime files and general report files (like pwsFWI or YADR) are now read with the cache=false qualifier in the ajax call. This should improve user experience.
  • AirQuality: reworked the interface for the order of the first columns.
  • AirQuality: Fixed a bug concerning the normalisation of the AQI to the position of the indicator on the right panel of the table.
  • AirQuality: Changed default country to US.
Please note the post on Air Quality.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Because I am implementing secure FTP protocols I had to switch provider which I did this morning.
That provider switch may have some consequences as the map won't work anymore.
I am not sure how this will work out but I hope to be back online soon (later today) and you will hardly notice.
Well, you will notice: Upgrade to 4.7.0 required to keep functionality.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.6.3 is online

version 4.6.3 (11/11/2020)
  • Maps: Reverted to old FTP system (only plain FTP supported) and corrected for provider change of Map database.
This should correct the failure of the old versions.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.7.0 is online

version 4.7.0 (15/11/2020)
  1. Run: the executable and dlls have been removed from the CumulusMX directory to the utils\bin directory. You have to call CumulusUtils with the following command (being in the cumulus directory) utils\bin\cumulusutils.exe. This avoids conflict and synchronisation of version of dlls. Cumulusutils has now evolved into technical independency.
  2. Internal FTP: Made FTPS available (SFTP will follow);
  3. Internal FTP: Made FTP logging optional (section [FTP site], parameter FtpLog, default off)
  4. Map: Internal working has changed because the Map database stationswithutils.xml underwent a provider change because of the FTP changes.
  5. Technical: CumulusUtils is now using ServiceStack.Text.dll i.s.o. NewtonSoft.Json.dll (You may now remove Newtonsoft dll from the CumulusMX directory is you are on CMX version 3.9.1 and higher)
  6. Added a NOTE at the start of the manual to focus on the parameters CumulusRealTimeLocation and UploadDir to avoid confusion.
I solved the problem of conflicting DLL versions by creating a bin directory under the utils directory. Just copy this from the distribution.
The manual is updated.

REMOVE:
  • CumulusMX/cumulusutils.exe
  • CumulusMX/cumulusutils.exe.config
(The same for Cumulus-1 but then a different dir name)

The command (in the Cumulus directory) to use is

Code: Select all

utils/bin/cumulusutils.exe [action]
All other things like output and logging remain unchanged. I sincerely hope this one is more successful than the previous one. It has been tested on 5 different machines and apart from one pending it is pretty much without problems.

NOTES:
Running this on a Raspberry where you use Apache as your webserver you may run into some questions but if your OS/Mono is up to date then it is definitely a configuration issue.
Running this under Windows 7 may bring you in some problems (especially wrt the map - this appears to already so in 4.6.3 :( ).
The new provider for meteo-wagenborgen.nl has created an additional problem for the lockfile mechanism. In case of failure, warn me and I'll manually release the lock. Then run again.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.7.1 is online (and FTPS should be working)

version 4.7.1 (20/11/2020)
  1. Manual: Further fine tuning info UploadDir and CumulusRealTimeLocation.
  2. Technical: Reported issue to FluentFTP with respect to FTPS. Waiting for feedback/Solution. As a workaround downgrading the FluentFTP library to 32.3.1.0. works. It will remain so until certain upgrade is safe.
  3. Technical: Optimised some parts of the CumulusUtils javascript runtime.
  4. Technical: resolved a naming conflict between CumulusUtils and a library. The effect of this conflict is unknown, but it may account for some weird effects some users encountered.
  5. AirQuality: Bug fix adjustment decimal separator.
  6. AirQuality: When using thrifty, the module is not uploaded (not necessary because the info seen is realtime, no history data so far, this may change in future).
  7. AirQuality: The calculation of the AQI has been fine tuned and should be OK now both in CMX (build 3097) and CumulusUtils.
  8. AirQuality: The table width of AirQuality report must fit within its frame.
  9. Maps: The lockfile mechanism has now been rewritten to take advantage of the FluentFTP library. The algorithm did not change, the use of FTP did. Keep an eye open for anomalies.
After 4.7.0 release (provider change together with FTP library change) the technical issues now hopefully have been left behind. Please report if not so.
So far Windows 7 users may still not be able to run the map.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

version 4.7.2 is online

This version is functionally identical to 4.7.1 but has some technical modifications to accommodate Windows 7 users (they had limits because of technical issues since the introduction of the FluentFTP library).

version 4.7.2 (24/11/2020)
  1. Parameter SslFtp did not exist in Cumulus 1; Made sure the default is correct on C1 / Win7 machines.
  2. Made sure FTP logging works on Win7 machines.
  3. Made sure Maps works on Win7 machines
The next release will have graphing of the AirLink and the addition of the AirQuality module to the translation system.
But that may take some time.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.8.0-beta1 is online

version 4.8.0-beta (29/11/2020)
  • Added graphing for the two AirLink sensors.
  1. This is quite an elaborate upgrade of the airlink module. If you don't have an Airlink there is nothing here for you.
  2. Basically each sensor has two graphs (for 2p5 and 10 ug/m3). The original Table View awaits some fixes in CMX which are coming with build 3097.
  3. The 10 minute refresh (or whatever frequency the user has set for the graphs) has not been implemented yet for the airquality and maybe it never will as currently switching between the views rereads the json
  4. If you want the data to update at some frequency, you have to run the special command:

    Code: Select all

    utils/bin/cumulusutils.exe AirQuality Thrifty
    The Thrifty command assures only the JSON files are updated and send to the website. See the manual.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5831
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CumulusUtils

Post by HansR »

Version 4.8.0 is online

version 4.8.0 (1/12/2020)
  1. Added graphing for the two AirLink sensors.
See the manual and the remarks in the previous post for how it works.
Thanks to Phil and Paul for trying out.

I don't foresee any other new releases in the near future unless disaster strikes or you come up with great ideas.
So for the time being, this is it. 8-)

NOTE: If the data is written to the AirLink datafile on a multiple of 5, 10, 15 30 minutes (or whatever) then please start the AirQuality data gathering on multiple of that frequency. I had assumed a minute frequency and it therefore tries to find an erroneous end time which may take minutes before it reaches that. This will be fixed later.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
Locked