Additional Logging

[Big Daddy]

Please can I request that we have additional logging for some webtags e.g. UV, Sun Hours, Solar, Lightning so that we can have mothly, annual and alltime figures as we do for many of the others. I know there are already Today and Yesterday logs for these and its not a "put on the top of the list request", it would just be nice to keep these inline with other readings.

I assume this can be done via SQL and have just started learning SQL (I kow absolutely nothing). I have my database / table set up using the dayfile data as per Marks script so just need to work out how to get the data and export it to my website. Steep learning curve.

Thanks
Andy

[sfws]

I can't magic away your

Steep learning curve

but I can share a PHP function that includes the SQL to find highest/lowest of any weather parameter for any period

can have monthly, annual and all-time figures

, it also accepts that the highest/lowest might happen after the end of what is on the database.

SQL example.txt

I don't use Mark's script, my schema for example ensures that the column name for a time-stamp prefixes with a 'T' the column name for the value. From memory, Mark's script is based on a script by David A Jamieson that abbreviated Steve Loft's somewhat arbitrary naming in his early design. As far as I know, Mark's script does not cover lightning, so I am not quite sure what you will end up doing.

My schema which I worked out over a decade ago and has served me very well with Cumulus 1, has a number of columns that are not in the standard dayfile.txt, but derived from other log files. Consequently I use a different script for reading the various log files and updating the database tables I use, it was written for Cumulus 1, but is also working for MX. I originally had a slight problem with populating some columns derived from a file that is different in Cumulus 1 and MX, but resolved that this evening. In the future, I might discover how to use the MX SQL feature where you specify your own schema, but that will wait for a set of rainy days when I don't have lots of other tasks to do.


Anyway, the first parameter of my function is the column name for the value, in your case this could be the UV column, or the Solar column or the Lightning column. The next parameter is set to true to report the highest in period (obviously can set this to false and replace highest by lowest everywhere). The third parameter allows you to reduce the number of decimal places in the output from the number of decimal places stored on the database. My fourth parameter identifies whether the column is something like wettest day that does not require a time-stamp or something like UV that would peak at a particular time of day.
The next 2 parameters are the start and end dates for the period (I have a separate script that you tell whether you want monthly, yearly, or all-time, and it returns the start date for that period, by default the end date ($endPeriodDate) is what Cumulus web tag for today says, but I can also specify past periods).
The database will usually contain rows for days up to yesterday, so the next parameter is the highest value for today (in case that is higher (lower) than any day on database). The last parameter in the function I have shared is the highest (lowest) value for yesterday, just in case that day was not successfully updated onto database. (In my own version of the function I also input the highest (lowest) from the previous day as I save the yesterday web tags at the end of each day to give me prior day values).

The function includes some documentation on the global values, and what it does. It might not make much sense to you now, but when you know some PHP and SQL I hope it will make sense. Anyway, if you can't make use of this, maybe it will help someone else.

[Big Daddy]

Thanks sfws,

At the moemnt that is way over my head but as you say, hopefully in the future it will make a bit more sense. I already have some very basic php knowledge, and I mean very basic.

My website started off as one of Jacues DeRoche's weather by you templates and still uses the basic pronciples to obtain the data as php variables out of the Cumuus webtags rather than creating data as an sql database. Once I get a bit more familiar I will see what I can do but for the time being I am just "playing", starting with the very basics and working my way through.

Andy

[sfws]

I already have some very basic php knowledge, and I mean very basic.

My website started off as one of Jacques DeRoche's weather by you templates and still uses the basic principles to obtain the data as php variables out of the Cumulus web tags rather than creating data as an sql database. Once I get a bit more familiar I will see what I can do but for the time being I am just "playing", starting with the very basics and working my way through.

Andy

Yes, I have often looked at your web site.

One of the tables in my database is populated sometimes by php variables out of the Cumulus web tags, but other times by reading various log files.

I'm entirely self-taught on all the Cumulus features, I only started using Cumulus for "just playing" with web pages, and I tried Jacques DeRoche's weather templates at the start when they were only available in French (and you did not have to register to download them). I also implemented lots of examples from books, and tried to improve or change them before I moved onto anything more complex. Now my scripts are perhaps too complex, but I am proud of what I have achieved. I stuck to HTML and JavaScript for ages, it was Jacques who inspired me to move into php and Mark who inspired me to move into MySQL.

Stick with your basic php, and learn some basic SQL. Do something simple, try to record plants in your garden or stuff in your shed, i.e. something simple for first database. Or implement a database suggested in a simple SQL book or online guide. As you seem to be planning, only move onto Cumulus database when you feel comfortable with your "playing". I could give you some simple ideas, but I think it is better if you find what suits you.


There are a number of simple packages of apache, php, and a database that allow you to create a web server on a PC or other computer where you have full control and can more easily experiment.

[Big Daddy]

Hi and thanks, one step at a time, the same as I learnt html and a bit of php.

I have a good database on my 2nd Raspberry PI using MySQL and phpmyadmin. I uploaded the dayfile.txt and this gets updated daily from Cumulus MX. Its on a different server so that I can play as much as I like without any danger of messing up my CumulusMX installation.

Maybe in a while I will understand what I am doing.

Andy

[sfws]

I have a good database on my 2nd Raspberry PI using MySQL and phpmyadmin. I uploaded the dayfile.txt and this gets updated daily from Cumulus MX. Its on a different server so that I can play as much as I like without any danger of messing up my CumulusMX installation.

Maybe in a while I will understand what I am doing.

It is a month on, and I thought (reminded by another post I just replied to about use of databases) I would suggest some SQL for you to try in phpmyadmin:

Code: Select all

SELECT  'LogDate', 'HighUV', 'THighUV' FROM `dayfile`  WHERE  'LogDate' BETWEEN '2020-04-30' AND '2020-04-01' ORDER BY  'HighUV' DESC 'LogDate' ASC LIMIT 0,1 

That will tell you the date and time your UV measurement was highest last April.

Next you can probably guess for this year the highest UV can be found using

Code: Select all

SELECT  'LogDate', 'HighUV', 'THighUV' FROM `dayfile`  WHERE  'LogDate'  > '2019-12-31' ORDER BY  'HighUV' DESC  'LogDate' ASC LIMIT 0,1 

You don't need to understand SQL to type those queries in. But very simply "SELECT" is keyword to ask for values, the column names follow. The "FROM" clause identifies the database table. The "WHERE" clause is needed to tell the query which rows to retrieve from the table. The ORDER BY clause specifies which columns to sort on and whether ASC or DESC. We want the smallest date (LogDate ASC) and the largest UV (HighUV DESC). The Limit says which of the retrieved rows to keep, in this case starting count at first row retrieved (0) , return just one (1) row.

How you do this using PHP takes us back to the script I shared in the first post.

[Big Daddy]

Thanks, I will take a look.

Unfortunately I have been distracted recently with my new weather station so havent had time to go through your scripts or to look at SQL. I did buy an SQL for Dummies book which I am finding useful in explaining some really basic stuff which has given me a better understanding.

Andy

[sfws]

Please can I request that we have additional logging for ... e.g. UV, Sun Hours, Solar, Lightning
so that we can have monthly, annual and all-time figures as we do for many of the others.

Just looking at this 4 month old request, I note
* Lightning is somewhat different to other 3 examples
* For the others, implementation would involve adding more parameters to the .ini log files (as well as the code to process these), it would not change fields in the .txt files (nor change the database schema)
* that nobody else has chimed in to say they too want the additional functionality
* that following on from the major tasks of making Feels-like and Humidex fully implemented variables, this is I would guess an easier extension as (apart from lightning) today and yesterday already exist, so finding highest in month or year should be easier.

I guess lack of support for this request probably reflects that UV, Sun Hours, Solar sensors are used by a very small proportion of those using Cumulus MX (and indeed the majority of Cumulus users are still using Cumulus 1, although the ratio is getting more balanced), but also (as indicated in original request) that it is possible to write query scripts that approximately) derive these extra values from database records.

I don't have any of these sensors, but I can see a justification for UV, Sun Hours, Solar, being somewhere on the future "to do" list as an obvious extension of what MX currently offers.

As for lightning, nearly all posts in forum are by just 2 people, suggesting this should be even lower down the priority.
But it is also less clear to me what monthly, yearly, and all-time parameters you track (highest number of strikes in hour, day, month maybe; but how far away do you include?).
I would say this should be rejected unless someone comes up with a precise specification of what is a sensible implementation; and there is significant demand.

[mcrossley]

OK, my 2p on this.

- @sfws is correct in that not many people actually record UV. Probably because a properly calibrated UV sensor costs a lot more than most people are spending on a whole station.
- Most UV readings that re recorded are probably "suspect" - definitely so if they come from a Fine Offset/Ecowitt station at the moment, they are toy values.
- Likewise for lightning - the serious detectors are logging externally anyway. The detectors the likes of Ecowitt are making as not in the same league, just read about the issues of reliability of detection from them on the WX forum.
- The way Cumulus MX logs data is cumbersome, it is not conducive to adding more and more data, the log files are unwieldy as they are.

So unless there was a big demand for this, I'd not be inclined to implement it.

[Big Daddy]

Thanks for the responses, I have no problem with your answers and I know there are much more important things to develop and improve, especially as there is little demand.

My thinking was that for UV, lightning etc it would be good to have simlar records that we have for other measurements, e.g. highest monthly, alltime, max total sun hours etc and also monthly, alltime, yearly data for lightning strikes, number of strikes, nearest strike etc.

I will eventually try and get round to doing a bit more with SQL and maybe I can work something out this way.

Thanks for taking the time to consider it.

Andy

[rogerthn]

The data is there and look what I found
https://rogerthn.se/weather3/top10s.php