Webtag pages ready for review

[HansR]

On the question of sfws, I rewrote the webtag page and related pages for the past two weeks and I think they are ready for review.

What did I do?

1. I copied the Webtags page to Webtags_(preserving_history) without a redirect (so the page is orphaned, no other page points to it) and created a new page called Webtags which now receives all links to Webtags.
2. I decided to remove all history and focus on the current functionality. Cumulus 1 is gone, will fade out and won't come back.
3. While writing, I decided to rewrite the following pages as well (creating preserving_history pages on the way):
   • Recent History
   • Webtags/Parameters
   • Cumulusmx.db
   • I removed Webtag Applicability (i.e. it is moved to its preserved History version)
4. Partially edited a paragraph in Calculate Missing (derived spot values). This btw led me to believe a special page must be made where all equations of CMX are published (special prject).
5. I created a new file: Full list of Webtags

If you search for preserving history you will find in the page title list the actual archive.
The result of my work you will find under the links above.

I would highly appreciate comments on the pages mentioned above on topics as general usefulness, language, technical correctness and completeness. I won't ask completeness from the reviewers but it would be kind if you could take a page or a section and walk through it. I did not yet fulfill sfws' original reason for asking: there are webtags missing from version 3.12. The first step is to agree on a working page, then do the additions but I would appreciate the list already: I have no idea which Webtags are missing.

Testing webtags, naming and values returned (correctness), would be a bonus.

Please write your comments below not in one blurr of text but as value bullet pointed list so that I can rework the pages or comment on your remarks. Thank you.

NB: If I understood it well @sfws has worked on the wiki for roughly 10 years and has repeatedly expressed that he wished to do his gardening more intensively. That is more than highly appreciated. More than it may seem from some communications and this rework of the pages: in some two weeks I removed a lot of words that took him a bit more time to write together. Nevertheless, times are a changin' and it was time. Maintaining all history in the operational wiki is practically impossible. Thanks @sfws.

[mcrossley]

Thanks for this Hans, I have had a quick look and it looks great to me.

Perhaps the link to the full list could be more prominent, and accessible from the top level. Once you are familiar with tags, you I just want to access the full list quickly and not have to click through other pages to get there.

I'll have another study later and see if I have any other comments.

[HansR]

Pages as in first post have been finalized and where necessary the CumulusMX category has been added.

[saratogaWX]

Hi Hans,
I created a small PHP script to scrape the webtags definitions from https://www.cumuluswiki.org/a/Full_list_of_Webtags and compare/contrast to the WebTags.txt file generated by CUMX 2.22.3 (build 3214).

I've attached the preliminary results and hope you find it helpful. I do note that some of the 'missing' tags are defined as Time or Date versions, and my initial parsing didn't handle that.. I'll issue an update when that has been fixed.

Best regards,
Ken

[HansR]

Hi Ken,

Two minds one thought I had done similar about a week ago and mailed it to some but did not yet receive comment back. It's in C#, I'll publish it on github later today.

I did not search for duplicates and neither for what's in the wiki and is not generated by CMX. So thanks, valuable info and good idea.

The large list of tags available by CMX but not yet described in the wiki contains some errors though. E.g. #txbattery, #TdewpointTL and #TtempYH and more really do exist. So either there is some correction to make or you tested against another page (link is correct though).

Find attached CheckTaglist.zip, my results in TagsToDo.txt.
Link to the software follows later today
Software is here, is there somewhere I can run your php?

Regards,

[sfws]

You can choose to ignore input from me, or you can choose to learn from my experience, as it was not just uncertainty about the best way to list tags that stopped me from keeping my Wiki pages up to date with MX new tags.

Many MX releases ago, I did a comparison of WebTags.txt files (taking one produced by Cumulus 1.9.4 and one produced back then by MX) and Wiki web tags as on the original Wiki page. I also extracted all mentions of tag names from the MX release announcements (many new tags are never mentioned there). I tested tags in MX by using them in a special template file (a bit like the one that Water01 maintains, but this one deliberately tried different combinations of tag names and where relevant a variety of parameters, to identify what did and what did work). I also tested tags in some custom SQL, again to see what tag names would work, and what parameters could be added and what resulted. Some of the inconsistencies with parameters were described on what has become the preserved history Wiki page.

It was not done quickly by a script as Ken has used, instead for each of the different sources all the tag names were sorted into alphabetical order using a word processor (Libre Office Writer). The outputs I got back were added to the sorted tag name lists so a full comparison could be done.

This exercise was very tedious to do, and time consuming, but it did reveal a number of issues.

Like HansR has found in his exercise, the MX WebTags.txt file was not authoritative, as at the release I tested, some of the tag names in it did not yield any useful information, and some tags that MX did understand did not appear in that list. Of course some tags are specific to certain weather station types or certain sensors, so I could not test every combination.

I also found that tag names that will work in a template file, are not necessarily the same as those that will work as values for custom SQL. I did not go on to test MQTT, POST, GET, and any other ways that tags can be used. It was the time/date tag names, and their possible parameters, that seemed, at that release to be inconsistent, e.g. try getting dates from <#MonthRainRateHD> and <#MonthRainRateHT> in the different places where tags can be used. For those who use a meteorological day starting at 9am or 10am, there are further issues on the first day of a new month/year. For example take 1 June 2021 in custom rollover SQL; '<#metdateyesterday format=yyyy-MM-dd>' will return "2021-06-01", but '<#metdateyesterday format=yyyy-MM>' will return "2021-05", a different month!


It may be the latest MX release has altered some of the relevant coding and possibly there is more consistency. But nobody should expect it to be easy to get an authoritative list.

[HansR]

But nobody should expect it to be easy to get an authoritative list.

With some iterations, Marks input and my and Ken's software I think we will be there in the end.

Something like this is never finished though as long as there is development on CMX. And that is good, it shows it's alive and kicking. No problem. No one demands a 1 - 1 synchrone wiki vs software every minute of the day. Only what is reported and what is found to be in error will trigger change.

[saratogaWX]

Ok, a more through version of the parsing PHP script and the lists are much smaller and likely more accurate.

I've been developing the script on localhost, so it's not quite available online, but I will post it.

[HansR]

Ok, a more through version of the parsing PHP script and the lists are much smaller and likely more accurate.

I've been developing the script on localhost, so it's not quite available online, but I will post it.

OK. Thanks.

I'm curious how you parse, there must be knowledge in it because e.g. <#temp> exists twice in the page: once as the tag identifier and once in a description. However the software does not know and it should be a trigger to investigate. I detect it as a duplicate. Two different approaches. Nice

[saratogaWX]

I've cleaned it up and feel this is a reasonable V1.00 of the script:

https://saratoga-weather.org/CUMXutil/check-wiki.php

Source: https://saratoga-weather.org/CUMXutil/c ... p?sce=view

Feel free to implement on your own server. Just upload a WebTags.txt from the CumulusMX directory into the same directory on your site.
Run the script .. the first time, it will download and cache the Wiki page. To get a fresh copy of the Wiki page, just run check-wiki.php?reload

There's also a ?debug argument enabled which will produce a lot of text output and save a bunch of var_export() arrays to text files for additional debugging.

[HansR]

Thanks Ken.
Some interesting differences with my own approach (see github link above, I added the duplicate search this afternoon), if I may quote you:

# Whew.. all that work to get the Wiki tags ready for comparison.
# so.. do the compares

You kept going where I took some shortcuts
Nice work, thanks again.

[mcrossley]

I'm a bit puzzled by why the web tags file is not "authoritative"?

All it does is spit out all the web tags (held in a dictionary object) that CMX understands, the same dictionary is used to perform the lookups when parsing text containing the tags.

Not all those tags will produce meaningful/any data for all cases as they are configuration dependent. Davis tags will not work if you have an Ecowitt station, and air quality tags if you don't have a AQ sensor configured etc.

[HansR]

I'm a bit puzzled by why the web tags file is not "authoritative"?

All it does is spit out all the web tags (held in a dictionary object) that CMX understands, the same dictionary is used to perform the lookups when parsing text containing the tags.

Not all those tags will produce meaningful/any data for all cases as they are configuration dependent. Davis tags will not work if you have an Ecowitt station, and air quality tags if you don't have a AQ sensor configured etc.

For me the WebTags list produced by CMX is authoritative.

My TagsToDo output (see attachment) does not have the few tags Ken has when he compares Wiki to WebTags list. But this only would mean those tags can be removed from the wiki (or checked on correct spelling).

For the rest it produces duplicates in the Wiki (without interpretation so a tag in a description is counted as well, it needs a check) and tags in WebTags by CMX but not in the Wiki yet.

So yes, the WebTags bu CMX is taken as authoritative.

[saratogaWX]

Mark, I consider the CUMX WebTags.txt is the authoritative source for available tags.

We should strive to document all the available tags in the Wiki, and remove/correct tags in the Wiki that are not supplied by CUMX.

[saratogaWX]

I've been busy. I've added the missing tags from the current WebTags.txt to the Wiki (and documented where each tag came from in the comments with each update).

Now my https://saratoga-weather.org/CUMXutil/check-wiki.php checking script shows:

Comparing Wiki to WebTags list

.. 0 Wiki entries not in WebTags list.

Comparing WebTags list to Wiki

.. 0 WebTag entries not described in Wiki.

Note: the following WebTag.txt entries were likely not in the Wiki, but ignored since MX only generates 'n/a' values for them:
<#CpuName>, <#MemoryStatus>, <#DisplayMode>, <#DiskSize>, <#DiskFree>

We have now achieved 'webtag nirvana' with all available tags described in the Wiki