Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4018) - 28 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Weather Diary (C1 and MX Differences) - Resolved already in MX

A Forum to archive Cumulus MX development suggestions that have been rejected or solved by other means.
Post Reply
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Weather Diary (C1 and MX Differences) - Resolved already in MX

Post by sfws »

I moved from Cumulus 1 to MX, and this post has my observations re Weather Diary.

While MX can read most Cumulus 1 log files, it cannot read the Cumulus 1 log.xml file and transfer its contents to diary.db. Luckily, there has not been much snow here since I moved and restarted my Cumulus 1 last January, so in my case I manually copied my 3 entries across. I don't know whether an automated transfer would be useful for anyone else, but for me it might be easy to write a PHP script to do that.

To me the key Weather Diary functionality difference is that in Cumulus 1 you can select the time at which the snow settings rollover (Cumulus.ini, Station section, SnowDepthHour=9), and I am guessing MX does not support that (as there is no mention in build 3046/3047 announcements) although I have not experimented yet to test accurately when the web tag <#snowdepth> is updated in MX.

Another difference is that Steve's design permitted multiple entries on a single day (all edits) to be retained, but Mark's editor design only has a date picker, (all entries are recorded at time of 00:00:00) so any edit overwrites previous entry.

Anyway, my schema for my daily summary table in my MySQL database (written a decade ago for use with Cumulus 1) includes columns for all that is now in the sqlite database as well as reading dayfile.txt and some other logs. For MX, my minor change to my database table update script (that normally runs when Cumulus is processing the rollover) is so it reads the sqlite db when forming the INSERT or UPDATE queries for my daily summary table. I use the"PHP Data Objects (PDO)" commands to access the sqlite database and then "foreach" constructs to read the one row for each date and assign the values to the relevant MySQL columns. I'm guessing the MX SQL facilities can not do this?
Last edited by sfws on Wed 15 Apr 2020 8:56 pm, edited 1 time in total.
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Weather Diary

Post by mcrossley »

Thanks for this. I really don't have much of an idea about the how the diary was used in Cumulus 1. The code is MX was largely put in place by Steve, but commented out. I just implemented a user interface and enabled the code.

So the gist of it is, in Cumulus 1...
  • You can add multiple diary entries per day
  • The values for the web tags <#snowdepth>, <#snowlying>, <#snowfalling> are taken from the last diary entry for today? Today taking the SnowDepthHour into account.
  • The snow "day" can have an independent rollover time from the station met day
OK, the good news, CumulusMX still reads the value "SnowDepthHour" from the Cumulus.ini file, and uses it with the above web tags

Multiple entries per day should be doable, I'll take a look. Adding multiple entries is easy, but I'll have to think about how to handle editing them.
They would have to logged against actual date, but Cumulus internally converts the day using the SnowDepthHour just like it does for met day hour.
My question is, how critical is it to have multiple entries vs. a single editable entry?

Adding snow data via the builtin SQL would be possible, but you would have to use a custom SQL statement rather than the default. I'd have to check when the custom SQL is run in relation to the rollover of the snow day.
Of course there would be a mismatch between the met day and snow day if they were different. For example if you had a midnight met day, and 9am snow day, the SQL entry would be written at midnight, part way through the snow day, to resolve that I think you are back to customised routines.
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Weather Diary

Post by sfws »

Mark, thank you for the reply.
I did not realise that Steve had made provision for this feature by adding code.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm OK, the good news, CumulusMX still reads the value "SnowDepthHour" from the Cumulus.ini file, and uses it with the above web tag
That is a relief.


I will make some suggestions based on your other points, you decide subsequent action. I hope this covers everything you need from me, and my answers are not shot down by others.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm The snow "day" can have an independent rollover time from the station met day
The reason for this, I believe Steve said sometime, was so those with midnight rollover, did not have to go out at midnight to manually record details for the diary.
Those who follow professional meteorologists by using 9am/10am for normal weather recording would follow them by using 9am for snow (they would apply that year round hoping no snow in summer time so does not matter it does not swap to 10am)
mcrossley wrote: Sat 28 Mar 2020 4:25 pm I'd have to check when the custom SQL is run in relation to the rollover of the snow day.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm the SQL entry would be written at midnight, part way through the snow day, to resolve that I think you are back to customised routines.
These relate back to previous answer, I think it is reasonable to expect standard routines to work for matching times, and customised routines to be needed if times do not match.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm Of course there would be a mismatch between the met day and snow day if they were different.
In my opinion, the default should be that the snow day matches the met day, including the 9am to 10am switch if that is selected, although we hope that the chance of snow is low in period when summer time operates. I believe we should try to treat everything we report on same time basis. This would apply to any extra parameters like lightning, soil temperatures ... as well.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm The values for the web tags <#snowdepth>, <#snowlying>, <#snowfalling> are taken from the last diary entry for today? Today taking the SnowDepthHour into account.
Only <#snowdepth> is a web tag in Cumulus 1, and it does reset at the SnowDepthHour,
<#snowlying> is not available in Cumulus 1, but can be deduced by checking if first is zero or not.
Cumulus 1 also outputs the Snow Index this is basically the sum of all the snow depths in the selected period (month, year), Steve Loft said Philip Eden defined it as the sum of all significant (more than half land covered) snow depths at 9am, so I was a bit surprised the Cumulus 1 weather diary did not implement time-stamping the multiple entries during the day, and I never managed to prove whether Steve Loft's index calculation ignored all but last weather diary entry for each day.

There is nothing like <#snowfalling> provided in Cumulus 1 to let a web page know if snow is falling.The work around I and others have done is to actually read the weather diary locally and send the result to your web site. (The routine was actually developed by Miloš Jirík, it was written in Czech and is posted in this Cumulus support forum) . The suggestion uses a "$xml = simplexml_load_string($converted);" and "foreach ($xml->xpath('//ROW') as $item)" approach in PHP. Then you have to check the date in the row and see if it matches the day you are displaying on the web page before updating output.

I'm unsure what the other people who process snow records do, about multiple entries in a single day. It appears Miloš let each read overwrite any earlier read, so he did take his output from last diary entry for day. My Cumulus 1 script takes the worst case from all the rows for that day, i.e. I report the highest depth in any entry for day, I report snow falling if any record had snow falling ticked, etc.

My suggestion for MX is that it should provide the snow index, and treat like any other 'Monthly' tags.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm You can add multiple diary entries per day
True for Cumulus 1, my personal opinion is that one entry per day is better (to remove the worst case question), but I would store the time as the time given in the value "SnowDepthHour" from the Cumulus.ini file.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm Multiple entries per day should be doable, I'll take a look. Adding multiple entries is easy, but I'll have to think about how to handle editing them.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm My question is, how critical is it to have multiple entries vs. a single editable entry?
My personal opinion is don't bother, but if you do it would be way down in anybody's priorities.
mcrossley wrote: Sat 28 Mar 2020 4:25 pm Adding snow data via the builtin SQL would be possible, but you would have to use a custom SQL statement rather than the default.
There was a recent question about lightning SQL, we really desperately need some documentation on built-in SQL and custom SQL in MX.
Last edited by sfws on Fri 08 Jan 2021 9:06 am, edited 1 time in total.
freddie
Posts: 2434
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Weather Diary

Post by freddie »

sfws wrote: Sat 28 Mar 2020 5:31 pmThose who follow professional meteorologists by using 9am/10am for normal weather recording would follow them by using 9am for snow (they would apply that year round hoping no snow in summer time so does not matter it does not swap to 10am)
The definitions used by professional meteorologists (at least, in the UK) are these:
  • Snow day runs from midnight to midnight (UTC). If snow falls in this time then the date is recorded as a "day of snow".
  • A "day of snow" is futher split into two categories: (1) A day in which a mix of snow and liquid precipitation (rain and/or drizzle) occurs simultaneously (commonly called sleet); and (2) A day in which snow occurs but not simultaneously accompanied by liquid precipitation (rain and/or drizzle). If (1) and (2) both occur in the "day of snow" then it is categorised as (2) - i.e. the "worse" category.
  • Snow depth is measured at 0900 UTC. A "day with snow lying" is recorded if there is snow cover of at least 50% at 0900 UTC.
These are the offical definitions used.

Note that a "day of snow" is more likely in the non-winter month of April than it is in the winter month of November (in most locations).
Freddie
Image
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Weather Diary

Post by sfws »

freddie wrote: Sat 28 Mar 2020 6:51 pm Snow depth is measured at 0900 UTC. A "day with snow lying" is recorded if there is snow cover of at least 50% at 0900 UTC.
I got that right, the Philip Eden that Steve Loft quoted was working for an independent company at the time he invented snow index, but he took account of above.
freddie wrote: Sat 28 Mar 2020 6:51 pm Snow day runs from midnight to midnight (UTC)
That makes life complicated for implementing the weather diary, two different days 9am GMT (UTC) to 9 for most weather records, plus snow depth; midnight to midnight for snow falling

Your rain records separate 9am UTC to 9pm and 9pm to 9am, so presumably rain days are not midnight to midnight.

That basically stops Mark from being able to do any standard end-of-day action. I leave him to work out whether he could change his interface to implement the UK system. I recall lots of articles in this forum about the daily periods Australasia's BOM uses being different depending on parameter.

You are a mine of information, Freddie, meteorological and computational.
freddie
Posts: 2434
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Weather Diary

Post by freddie »

sfws wrote: Sat 28 Mar 2020 7:21 pm
freddie wrote: Sat 28 Mar 2020 6:51 pm Snow day runs from midnight to midnight (UTC)
That makes life complicated for implementing the weather diary, two different days 9am GMT (UTC) to 9 for most weather records, plus snow depth; midnight to midnight for snow falling
It's the same for days of hail and gale too. Fog is similar to snow depth, in that it is the existence of fog (visibility less than 1000m) at 0900 GMT that makes a "fog day". Snow depth and fog day are measured at an instant in time (i.e. 0900 UTC) whereas all the other elements are measured over a period of time.
In fact, for temperatures it is complicated too. The definition of maximum temperature is that it is attributed to the date on which your meteorological day begins - which sounds reasonable. For minimum temperature, however, it is attributed to the date on which your meteorological day ends. So Cumulus doesn't have it strictly correct. For example - my weather station (via Cumulus) reported at 0900 this morning that the maximum temperature was 13.1, and the minimum temperature was -1.2. Following the definitive rules, the maximum temperature is attributed to the 27th March, but the minimum is attributed to the 28th March. But in the Cumulus "dayfile" they will both be attributed to the 27th. As maximum temperature typically occurs mid afternoon and minimum temperature around sunrise, then you can see why this strange-sounding rule originated.
sfws wrote: Sat 28 Mar 2020 7:21 pmYour rain records separate 9am UTC to 9pm and 9pm to 9am, so presumably rain days are not midnight to midnight.
No, rain days are 0900 to 0900 (UTC). The date the rainfall total is attributed to is the date at the start of the meteorological day - this Cumulus does get correct.
sfws wrote: Sat 28 Mar 2020 7:21 pmYou are a mine of information, Freddie, meteorological and computational.
I have worked in both industries for nearly 35 years, so I hoped to have picked up a thing or two during that time :D
Freddie
Image
User avatar
PaulMy
Posts: 3777
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2 Plus 24-Hour FARS
Operating System: Windows8 and Windows10
Location: Komoka, ON Canada
Contact:

Re: Weather Diary

Post by PaulMy »

I have been using the Cumulus1 weather diary for snow fall and depth. Now also using the CumulusMX for the same. CumulusMX is a better for me as it allows decimal measurement. I do refer to the snow index in Cumulus1 on occasion, but developed my own from my manual records http://www.komokaweather.com/weather/snowfall-log.pdf

I also do a daily manual reading and reporting of rainfall, snowfall and snow depth for CoCoRaHS http://cloud.cocorahs.org/wys/2018-2019 ... -2019.html and they ask for readings to be at the same daily time as much as possible, between the hours of 7:00 and 9:00 am. I use this CoCoRaHS reporting for the weather diaries. CoCoRaHS objective is to have an ongoing measurement of precipitation which is used by various organizations including conservation authorities and others who use that for draught and flooding decisions. Automated readings are not accepted by CoCoRaHS.

Enjoy,
Paul
Last edited by PaulMy on Sun 29 Mar 2020 1:50 pm, edited 1 time in total.
Davis Vantage Pro2+
C1 www.komokaweather.com/komokaweather-ca
MX www.komokaweather.com/cumulusmx/index.htm /index.html /index.php
MX www.komokaweather.com/cumulusmxwll/index.htm /index.html /index.php
MX www. komokaweather.com/cumulusmx4/index.htm

Image
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Weather Diary

Post by sfws »

freddie wrote: Sat 28 Mar 2020 8:20 pm The definition of maximum temperature is that it is attributed to the date on which your meteorological day begins - which sounds reasonable. For minimum temperature, however, it is attributed to the date on which your meteorological day ends. So Cumulus doesn't have it strictly correct. For example - my weather station (via Cumulus) reported at 0900 this morning that the maximum temperature was 13.1, and the minimum temperature was -1.2. Following the definitive rules, the maximum temperature is attributed to the 27th March, but the minimum is attributed to the 28th March. But in the Cumulus "dayfile" they will both be attributed to the 27th. As maximum temperature typically occurs mid afternoon and minimum temperature around sunrise, then you can see why this strange-sounding rule originated.
I was aware of this conflict with Cumulus, and do understand the dawn and mid-afternoon aspect. There are some now very old posts where users and Steve argue about whether he got his design right, he claimed Cumulus 1 did have it strictly correct for UK Met O, but if people disagreed they should do what he did," write your own software". Prior to my use of Cumulus when I made manual weather measurements when I could, (and first school then work made 9am GMT hard to achieve), I did when comparing with official measurements make the flip with temperatures.

I have to add, my personal opinion was that when MX was being developed, it should have abandoned its need to be compatible with Cumulus 1 in these few respects and made some of these adjustments to better suit meteorological rules.To me they are more significant when comparing with official figures at month or year level, because quite often there are sharp changes around the period when one month ends.
PaulMy wrote: Sat 28 Mar 2020 9:22 pm CumulusMX is a better for me as it allows decimal measurement.
Paul, in my implementation, I entered integer figures into Cumulus 1 and divided by 10 for output, so I am sticking to same and not yet using MX decimal ability.

I am sorry to hear that you have organisations who make droughts and flooding, round here flooding was a big issue last month (the height of water in rivers and the distance away of flooded properties broke even 2000 flooding records), and (not that far away) many people who were flooded last month are now confined to their temporary homes with no repair work going on in their flooded premises.

I have moved from having a garden that was exceptionally dry (sand and gravel on a steep hill) to one that was waterlogged (clay near a brook) for my first few months! I have already added nearly 4 cubic metres of gravel to the front garden, and raised part of the back garden.
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Weather Diary

Post by mcrossley »

sfws wrote: Sat 28 Mar 2020 5:31 pm
mcrossley wrote: Sat 28 Mar 2020 4:25 pm The values for the web tags <#snowdepth>, <#snowlying>, <#snowfalling> are taken from the last diary entry for today? Today taking the SnowDepthHour into account.
Only the first is a web tag ...
The others were added in 3.1.1 - b3054
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Weather Diary

Post by sfws »

mcrossley wrote: Sun 29 Mar 2020 1:46 pm The others were added in 3.1.1 - b3054
I missed that, and neither you nor I edited https://cumuluswiki.org/a/BETA_webtags#Miscellaneous
NeilThomas
Posts: 266
Joined: Thu 11 Oct 2012 9:51 am
Weather Station: Davis Vantage Pro2
Operating System: Raspberry Pi 4
Location: Gloucester
Contact:

Re: Accessing the Weather Diary (CumulusMX)

Post by NeilThomas »

Hi.
I am trying to use an API call to retrieve weather diary data from the system. No matter what numbers I supply to the call, I only get one entry back and it is always
{"entry":"","snowFalling":0,"snowLying":0,"snowDepth":""}
even though I have weather entries. The call I am using is:
http://localhost:8998/api/data/diarydat ... gth=100000
When I use this on the rPi directly, the result is the same. Can anyone pint me in the correct direction to retrieve data from the diary using the API. I am aware that there are tags for 'snowFalling', 'snowLying' & 'snowDepth', but I want to be able to retrieve a full entry. I have downloaded and tried to run the php file' snow_diary.zip' but cannot get it to work.
Neil Thomas
website: weather.oaktreewebs.co.uk | Davis Vantage Pro II | CumulusMX, Raspberry Pi 4 | MX V4 build 4010
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Weather Diary (C1 and MX Differences) - Resolved already in MX

Post by mcrossley »

Shouldn't you be using something like...

Code: Select all

http://localhost:8998/api/data/diarydata?date=2020-12-23
NeilThomas
Posts: 266
Joined: Thu 11 Oct 2012 9:51 am
Weather Station: Davis Vantage Pro2
Operating System: Raspberry Pi 4
Location: Gloucester
Contact:

Re: Weather Diary (C1 and MX Differences) - Resolved already in MX

Post by NeilThomas »

Hi all

Thanks for the answers given which seem to have wandered off my initial request. Namely, how to access the diary entry that is not available as a tag. The use of the documented api calls doesn't return any values even for the snow depth etc. So can someone let me know how to use the API to access any of the data held in the diary database.

I have also sent a request to be able to assist with the Wiki. I have been running Cumulus for many years, the last three or four using a Raspberry Pi. I actually run two weather stations, my main system using a Davis Vantage Pro and both a Technoline / Oregon system both on rPis; using one for development / ideas and one as my main system. I am quite happy to help in whatever way possible.

Using the rPi I have completely re-developed the interface using up-to-date versions of the libraries used for it. I have written Python3 programs to enhance the system by making the Pi store data in customised mariadb on the Pi and then transfer the data (and the realtime, interval and end of day files) to my public site. I also now use the records and alarms tags to drive a rPi LED board to display LEDs for all the various features.
Neil Thomas
website: weather.oaktreewebs.co.uk | Davis Vantage Pro II | CumulusMX, Raspberry Pi 4 | MX V4 build 4010
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Weather Diary (C1 and MX Differences) - Resolved already in MX

Post by mcrossley »

I documented the API above - that is what the admin interface uses to get the data.

I just added a dummy entry for yesterday, then called http://localhost:8998/api/data/diarydat ... 2021-01-06
and got back...

Code: Select all

{"entry":"Bloomin' chilly this morning","snowFalling":0,"snowLying":1,"snowDepth":"1"}
?

If you want to contribute to the improvement of the admin interface I'd be more than happy!

I have approved your Wiki account request - sorry but the admins don't often venture into the new account request area!
NeilThomas
Posts: 266
Joined: Thu 11 Oct 2012 9:51 am
Weather Station: Davis Vantage Pro2
Operating System: Raspberry Pi 4
Location: Gloucester
Contact:

Re: Weather Diary (C1 and MX Differences) - Resolved already in MX

Post by NeilThomas »

Thanks Mark.

I actually worked that out for myself earlier this morning by looking at the js file for the diary editor.

Thanks for enabling editor rights to the wiki. I've read the disclaimer and will limit my editing according to my areas of expertise. I will at some stage update the api page for accessing the diary data.

Neil
Neil Thomas
website: weather.oaktreewebs.co.uk | Davis Vantage Pro II | CumulusMX, Raspberry Pi 4 | MX V4 build 4010
Post Reply