Welcome to the Cumulus Support forum.

Latest Cumulus MX release v3.1.2 build 3055 - 4 November 2019.
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014

See the Wiki for downloading either version.

AnnualDataSummary PHP

Discussion and support for 3rd-party (non-Sandaysoft) tools for Cumulus
gemini06720
Posts: 1700
Joined: Mon 10 Aug 2009 10:16 pm
Weather Station: No weather station
Operating System: No operating system
Location: World...

Re: AnnualDataSummary PHP

Post by gemini06720 » Fri 16 Nov 2012 10:24 am

I looked at the latest 'readDayfile.php' (found in the 'AnnualDataSummarySA.zip' posted on Wednesday).

You have been looking for a complex solution where the problems are minor.
Last edited by gemini06720 on Sun 09 Jun 2013 8:42 am, edited 1 time in total.

BCJKiwi
Posts: 883
Joined: Mon 09 Jul 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: AnnualDataSummary PHP

Post by BCJKiwi » Fri 16 Nov 2012 1:07 pm

Ray, Thanks for the feedback.

I am concerned to find there are issues with the script on your system.
1. the metdate is code suggested by others to deal with non-midnight rollover times on new Years Day but untested on PHP and subject to the method of reading in metdate. It is there for others as I use a midnight rollover (despite 9am being the 'official' NZ rollover time)
There are two options - use metdate directly from the CUtags file, or add it to the CU-Defs and use it from there.
The method in the code is the one that works on my system. If I write it the way you suggest I get no data in the calendar. I do not know what was used by others to determine if the Javascript version 'fix' worked - I asked but did not receive a reply.

2. The script as written passes validation on my systems (win XPsp3 ie8, win 7sp1 ie9 & chrome, and, win8 ie10).
I do not understand exactly how the following process works but what I do know (through much trial and error), is that it does not seem to matter much what is in

Code: Select all

$data[$id] = $buf_arr[$dayfilecol] .'/'. $buf_arr[$dayfilecol2];
as the .'/'. can be replaced with a single . or the original .'|'. in the present script and the result is the same except for what displays in the result. Have also just tested the script with a / in place of the | at line 117 and it works exactly the same and also validates. The one exception to this is that for winXP ie8, if the delimiter is not '|' at 117 then the / does not appear in the datacells - but still validates.

This whole process has been a very time consuming process and has involved a lot of study and trial and error as I have freely admitted I do not understand most of this but am trying to learn as I go.

The initial objective was to add additional data items, and I liked the idea of the / between min/max hi/lo values as without it, I find those tables hard to read.
What I found when I added the extra data items (& before I changed anything else) the page looked OK but the validation threw up over 360 errors. I then discovered the original code also showed those errors. The code now in place works, produces a layout that I find acceptable and validates. I have been unable to get any of sfws' latest suggestions to work. He has been most helpful but is unable to test his suggestions as he cannot run php scripts and my lack of knowledge means I am probalbly missing some slight variation that may be needed.

I do not understand how your results are so different from mine.
I have attached a very slightly different version of the code resulting from today's efforts (mostly cosmetic changes to make it easier for others to use the script). I would be interested to see if you find it any different (note the filenames in the attachment are the standard ones but those on my site are slightly different. However the files in the attachment are also on the site and can be acessed via http://silveracorn.co.nz/weather/datasummary.php rather than from the sidebar. See later posts for cleaner versions.
Last edited by BCJKiwi on Mon 19 Nov 2012 3:01 am, edited 3 times in total.

sfws
Posts: 466
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Final post in top ten topic
Operating System: Tempermental Win. 10 PC
Location: Moving

Re: AnnualDataSummary PHP

Post by sfws » Fri 16 Nov 2012 2:01 pm

Ray - the reason for using <#metdate> was simply that your
EDIT INSERT: (plural, to include contributions by Mark and 'beteljuice')
previous posting did not cope with 9am and 10am rollovers. For those
EDIT INSERT: the first dayfile.txt record of
a new year becomes available only at 9am or 10am on 2nd January (compare final line of your quote below).
gemini06720 wrote:Mark, how about adding a simple check for the first day of the first month of the year - if it is 01-01 then subtract one from the year, otherwise ignore:

Code: Select all

if (isset($_GET['year'])) { $tableYear = $_GET['year']; }
else
{
  if (date("n") == 1 and date("j") == 1)  { $tableYear = date("Y") - 1; }
  else { $tableYear = date("Y"); }
 }
I had to modify my script and that solve the lack of data for the first day of the year - the data for the first day of the year should (hopefully) be available at the end of the first day... ;)
Incidentally, when I was working on my own daily summary page last February, my tests included verifying what happened when invalid parameters
Daily Summary close up when invalid parameter selected Feb 2012.JPG
or year were supplied in the url. These pictures from today show both errors are handled well ('please select a parameter', or 'choose year' prompts seen).
Daily Summary close up when invalid year selected Feb 2012.JPG

EDIT: Since I uploaded this, I have changed the class for the "Choose Year" mesage so now it can be seen clearly against a background like the numerical values seen in my other posting.
Much of my data is brought in from EasyWeather prior to first installing Cumulus. Although I have apparent temperature as a selectable parameter, it is not available before current year, and the code handles that by displaying all cells in earlier years as null. On my design, null cells are transparent (CSS:

Code: Select all

td {empty-cells: hide;
and the background shows.
Cells with numbers have coloured background, as seen in an earlier post in this topic.
Also I display 'Loading' where the year selection box subsequently appears when script completes. Now I have almost a year's worth of Cumulus derived days as well as the over 2 years of EasyWeather, the reading process in JavaScript is becoming obviously slower when you first start, select a new parameter, or choose new year.
You do not have the required permissions to view the files attached to this post.
Last edited by sfws on Mon 19 Nov 2012 11:59 am, edited 2 times in total.

gemini06720
Posts: 1700
Joined: Mon 10 Aug 2009 10:16 pm
Weather Station: No weather station
Operating System: No operating system
Location: World...

Re: AnnualDataSummary PHP

Post by gemini06720 » Sat 17 Nov 2012 1:46 am

How do you test your scripts? Are you running your own local web server on your own computer?

I am running the XAMPP package (ie: Apache, MySQL, PHP, phpMyAdmin, Perl, FileZilla FTP Server) on my development (personal) computer. I have also added the Xdebug PHP Debugger and Profiler Tool (extension). I have also setup my PHP to display ALL errors - this is after all a development computer and I want to 'catch' and correct all errors before I upload the script to the online web server.
Last edited by gemini06720 on Sun 09 Jun 2013 8:42 am, edited 1 time in total.

sfws
Posts: 466
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Final post in top ten topic
Operating System: Tempermental Win. 10 PC
Location: Moving

Re: AnnualDataSummary PHP

Post by sfws » Sat 17 Nov 2012 5:26 am

gemini06720 wrote:What validation? I feel you are confusing things - the W3C Markup Validation Service only checks the markup validity of the HTML document produced by the PHP interpreter, not the code of the PHP script - the code of the PHP script can only be checked by the PHP interpreter!
I simply addressed the original posting's report of hundreds of errors in the Markup Validation Service by pointing out where the generated HTML was wrong. I left it to BCJKiwi to work out PHP changes needed.
Ray, I fully agree with everything you say in your posting except this bit:
gemini06720 wrote: Quote:
...the reason for using <#metdate> was simply that your previous posting did not cope with 9am and 10am rollovers.
Come on - the time the year number changes is at 00:00:01 local time, not at the 9am rollover time - 9am is the time the meteorological day starts, not the time the new day and new year start.
I have edited my earlier posting to clarify that, and supplied images twice the size of those posted before.

Also, Ray, an apology: In my earliest posting I quoted a number of websites using PHP for daily summaries, but I did not have yours on my list (of bookmarks created last February).
Last edited by sfws on Mon 19 Nov 2012 5:56 am, edited 2 times in total.

gemini06720
Posts: 1700
Joined: Mon 10 Aug 2009 10:16 pm
Weather Station: No weather station
Operating System: No operating system
Location: World...

Re: AnnualDataSummary PHP

Post by gemini06720 » Sat 17 Nov 2012 9:13 am

It would really really be nice (and really appreciated) if either a first name or even a 'believable' nickname was used in signatures ... it makes it so impersonal/unfriendly to reply to an unknown... 8-)

Most of my reply was addressed to 'BCJKiwi'. It was just after I posted my reply that I noticed I had in fact replied to two different messages and users... :oops:
...I simply addressed the original posting's report of hundreds of errors in the latter by pointing out where the generated HTML was wrong.
Indeed, I have the same problems with the older script I have on my online web server - when the page created by the script is sent to the W3C Markup Validation Service, it does report hundred of error - the culprits are all those 'id' found within all those '<th>' and '<td>' elements that have been deprecated with the last releases of HTML4. But, they were needed to keep the column/row JavaScript highlighter working.
...I don't have PHP testing environment set up.
You should really do something about that, even if it is just for dabbling with your 'not-online' web pages - download and install XAMPP - with that package installed, you can do all the testing directly on your computer, without the need to upload the modified templates in order to test them ... you know, during those upcoming long wintry evenings... :)
...I did hope that PHP experts would continue to chip in earlier to say some of the points you, Ray, raised in your posting.
Well, as many other users, I get saturated with the forum conversations and have to take some time off ... some time returning to the old threads, but most of the time, not...

I have been (almost completely) off the forums for a few months now, with a short visit every couple of days... I have been involved (with some other Cumulus users) in a design project or rather with the integration of weather information within clean and modern template sets. There is a lot of work involved with the integration and there is still a lot of work to be done. The project has 20 information pages and is growing a page or two a week. Originally, the project was to develop a new set of templates and make them available, but after much consideration, it was decided (at least on my part) to keep the templates for ourselves... :(

Hope to read more from you... Have a great weekend.

BCJKiwi
Posts: 883
Joined: Mon 09 Jul 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: AnnualDataSummary PHP

Post by BCJKiwi » Mon 19 Nov 2012 2:45 am

Ray, Thanks for your input.
Re testing - I had setup an XAMPP instal on a PC specially for the purpose of testing but then was diverted onto a LAMP install which I did not have configured correctly so the testbed was in limbo.
Have returned to XAMPP, completed the standard configuration and added folders and loaded the data. I found most of the issues you advised for which I thank you.
Have resolved them on the data summary page as well as a number of other minor issues on other pages.

It is evident that our hoster has the error level reporting turned well down as these errors are not reported by them and all the pages that had errors displayed as intended without indication of errors.
Also I find that in XAMPP these errors disappear and the page needs to be refreshed (sometimes more than once) to get them back. This makes it difficult to ensure that a change is in fact working as even when these errors are reported, in most instances, the intended result is displayed anyway. I would appreciate your recommendation regarding how to tweak the error level reporting in XAMPP - I am new to this so it takes some time to sort out this 'tuning'.
the metdate is code suggested by others to deal with non-midnight rollover times on new Years Day
"you have added the "$WX['$metdate']" variable to the code" ... "but did not test its proper operation!"
I do apreciate the metdate thing is problematic and that you do not see the need for it. The way it is coded in the sample pages posted in the earlier attachment is the only way the page displayed in the intended manner on our hosted site.
It is evident that I do not know the proper syntax to achieve the desired effect (of others) and that is why I specifically asked, when posting the attached files, that the code be checked by others which you have now kindly done. It was on our website because it displayed the correct result without apparent error when viewed there.
My understanding is that the data on a site at say 2 am on the first day of the year realtime is still in the meterological day of 31 December. So the idea is to not display the data as being on 1 January until after the new meterological day has become 1 Jan - i.e. after the rollover.
However, one would presume that the dayfile would not be posted to the webserver until after the rollover time anyway so it would seem this may fall into the category of "a solution looking for a problem". If this issue is of genuine concern then perhaps the better (more logical?) solution would be to extract the year from the last record in the dayfile rather than from the system. As we rollover at midnight, I have not really studied this in depth as it does not apply. I have reverted to the non-metdate code.
...use metdate directly from the CUtags file, or add it to the CU-Defs and use it from there...
Perhaps I should have been a bit more pedantic and expansive with my description.
Yes it would need to be added to the CUTags file as well as to the CU-Defs to be able to use it in CU-Defs, and, it would need to be available in the Cumulus version being run i.e. currently the Beta versions which it seems many are using and as noted elsewhere on this Forum. I had added notes to that effect in the script. However, no matter what I try I am unable to get metdate to work on the XAMPP test bed, so as noted above, for it to work the correct syntax would be required which I don't have and therefore I have removed it from the present code.
If I write it the way you suggest I get no data in the calendar.
If you write what?
$WX['metdate'] instead of $WX['$metdate'].
I do not understand exactly how the following process works...

Code: Select all

$data[$id] = $buf_arr[$dayfilecol] .'/'. $buf_arr[$dayfilecol2];
I do (and did) understand this was concatenating the two elements into the a single element - that is why it was done that way. With the help of XAMPP and your advice, the other parts of the code have been fixed and the page now appears to be error free. So this page is now entirely single columns and no colspan is required. The column/row highlighting also now works as expected which it never did with double columns.

See attached current versions of the scripts for review.
You do not have the required permissions to view the files attached to this post.
Last edited by BCJKiwi on Thu 22 Nov 2012 11:54 pm, edited 2 times in total.

BCJKiwi
Posts: 883
Joined: Mon 09 Jul 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: AnnualDataSummary PHP

Post by BCJKiwi » Mon 19 Nov 2012 5:17 am

Have now installed Xdebug and getting a little more specific detail on the few remaining errors in other parts of the site - NO Errors on the Annual Data Summary page!

The remaining errors relate to other mods I have made to ajax-dashboard & astronomy pages. Again these display as expected and show no errors on our hosted site.
So will work on those as I wish to ensure there are no issues at all.

Thanks

sfws
Posts: 466
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Final post in top ten topic
Operating System: Tempermental Win. 10 PC
Location: Moving

Re: AnnualDataSummary PHP

Post by sfws » Mon 19 Nov 2012 10:28 am

I found another PHP weather site with downloadable script http://weather.wildwoodnaturist.com/downloadsCVW.php for a colourful table with twin columns that validates as HTML http://validator.w3.org/check?uri=http% ... detail.php

gemini06720
Posts: 1700
Joined: Mon 10 Aug 2009 10:16 pm
Weather Station: No weather station
Operating System: No operating system
Location: World...

Re: AnnualDataSummary PHP

Post by gemini06720 » Mon 19 Nov 2012 12:49 pm

BCJKiwi, good move on installing XAMPP and Xdebug (I was going to suggest that plugin) - now you can create/edit/modify a script and see the results (almost) immediately - but you have to remember to clear the browser cache often as the browser will, by default, load the file that is in the cache rather than the one you have just modified...

If you have not done it already, I suggest that you set the PHP configuration file (php.ini) to display all errors.
Last edited by gemini06720 on Sun 09 Jun 2013 8:43 am, edited 1 time in total.

sfws
Posts: 466
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Final post in top ten topic
Operating System: Tempermental Win. 10 PC
Location: Moving

Re: AnnualDataSummary PHP

Post by sfws » Mon 19 Nov 2012 6:26 pm

gemini06720 wrote:confirm the time of the day when the 'dayfile.txt' is exactly created
It is part of rollover process, so after final processing of all logs with respect to the day just ended, but before initialising for day just starting.
From my observations, final measurements of day are collected a couple of seconds before rollover time, dayfile is created exactly on rollover time, and processed HTML files are ready by one second past, but that may vary (just by seconds) depending on what Cumulus is being asked to do in terms of uploading destinations. (Steve, this time I believe my brain is in gear and I am saving you having to answer, but you may have a better feel for the variation between implementations).
Uploading time of dayfile.txt will depend on what an individual set-up uses to do the upload - it may have programmed in a particular time some minutes later, Steve does not control that unless the Cumulus "Daily" field is used on the 'Internet Settings' screen. For that Cumulus Help says:
Daily The name (including path) of a command-line program to be run once a day, after the daily 'rollover' has been performed. The corresponding 'params' box can be used to specify any parameters to be passed to the command.
BCJKiwi wrote:perhaps the better (more logical?) solution would be to extract the year from the last record in the dayfile rather than from the system
Yes that gives correct year. I agree, I only brought it up in this topic because the other topic (referenced earlier) seemed to suggest it was a bug.
gemini06720 wrote:
Just had a thought - how about using the time the 'dayfile.txt' file was last modified, such as:

Code: Select all

$tableYear = date ('Y', filemtime('dayfile.txt'));
PHP is fast and I agree with both of you - leave it out or test last modification date (although
EDIT 3: (bolding)
latter will be calendar date, and for non-midnight rollover still need to subtract
EDIT 1: 9 or 10
hours
EDIT 2: from the original computer's system time
).
Last edited by sfws on Wed 21 Nov 2012 5:37 am, edited 3 times in total.

User avatar
steve
Cumulus Author
Posts: 26717
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: AnnualDataSummary PHP

Post by steve » Mon 19 Nov 2012 6:47 pm

To confirm the answer about when dayfile.txt is updated, Cumulus uses a timer which triggers once a minute, on the minute. It checks each time whether the configured rollover hour has been reached (i.e. a new meteorological day is just starting), and if so, does the rollover, part of which involves writing an entry to dayfile.txt. The exact timing will depend on what the system is doing at the time. This assumes that Cumulus is actually running at the time. If it wasn't, and it's using logger data, then it does the rollover when it encounters an entry from the logger which takes it into the next meteorological day.

Note that I don't follow threads such as this one particularly closely, so may not always see questions in them which are directed at me.
Steve

BCJKiwi
Posts: 883
Joined: Mon 09 Jul 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: AnnualDataSummary PHP

Post by BCJKiwi » Mon 19 Nov 2012 8:59 pm

If I understand Steve's reply fully, the dayfile will only be created AFTER the actual rollover time.

The file timestamp should be retained after uploading - i.e. it will remain the same as it was on the the Cumulus PC, not assume a new timestamp from the Hosting system.

Therefore Ray's proposal should work as the file is dated by the local system Cumulus is running in and Cumulus is getting its time from that system. This would be more efficient than reading the file.

However, I presume it would still need the '-1 day' so how would that be coded?
Duhh, we are only using the year so no need to -1 day!

Code: Select all

$tableYear = date ('Y', filemtime($dayfile));
as the var $dayfile is set up at the beginning of the script to take account of any path requirements - in our case;

Code: Select all

$dayfile='./data/dayfile.txt';   // set path to dayfile.txt to match site requirements.
Re the Caching issue - have just set the browser on the test bed to never 'check for newer versions of stored pages: Every Time I visit the webpage' under tools/internet options/temporary internet files and History settings.

Tags.txt ot CUtags.txt ?
On my system Cumulus sends CUtagsT.txt to the website as CUtags.php.
Last edited by BCJKiwi on Thu 22 Nov 2012 11:47 pm, edited 3 times in total.

gemini06720
Posts: 1700
Joined: Mon 10 Aug 2009 10:16 pm
Weather Station: No weather station
Operating System: No operating system
Location: World...

Re: AnnualDataSummary PHP

Post by gemini06720 » Tue 20 Nov 2012 2:19 am

sfws (since neither of you has heeded to my request of using a first name... :( ), I do not like using JavaScript - why force an operation on the users' computer when that operation can be performed at the server level.

There are many uses for JavaScript, but, indeed, there is one important point to respect (in my views), JavaScript should only be used when the same/similar operation cannot be performed on the server (with one of the server specific programming languages), such as producing special display effects...

BCJKiwi, with the 'PHP/AJAX Website Template Set', the main file that contains the Cumulus webtags is 'tags.txt' (have a look at it - the latest 'official' release is 1.08, dated January 2012 - my 'tags.txt' has (almost) all of Cumulus webtags (606), has version 1.08h, dated August 2012, is up-to-date).

In order to obtain the required variables needed by the web page templates, some pre-processing needs to be done.
  • -the 'tags.txt' is the master template (if I could give it such a name);
    -the 'gen-CUtags.php' script is used to process the 'tags.txt' template;
    -the result of that operation is a new generated template named 'CUtags.txt';
    -the above operations are 'normally' never performed by the users;
    -then, Cumulus processes 'CUtags.txt' and generates 'CUtags.php';
    -the 'CUtags.php' can be used by all web page templates (using the format "$WX['variable']");
    -the 'CUtags.php' will also be read by 'CU-defs.php' script to create/modify variable names.

BCJKiwi
Posts: 883
Joined: Mon 09 Jul 2012 8:40 pm
Weather Station: Davis VP2 Cabled with Solar
Operating System: Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: AnnualDataSummary PHP

Post by BCJKiwi » Tue 20 Nov 2012 3:41 am

Have never run Gen-CUtags - well maybe just the once - don't really recall now - just edit the file in the Cumulus PC as needed.

As shown elsewhere on this forum http://sandaysoft.com/forum/viewtopic.php?f=20&t=8123, I have posted the full set of tags for 1.9.3 1047 (except the two large point data arrays.

Ray, you know who I am as we have exchanged emails. Those are my real initials plus an indication of where in the world I am. I trust you know what a Kiwi is and where it comes from ;).
Last edited by BCJKiwi on Tue 20 Nov 2012 9:54 pm, edited 1 time in total.

Post Reply