PHP Script - RealTimeLog PHP Script

[TNETWeather]

.
PHP Script - RealTimeLog

This is a first shot at a script to collect the realtime.txt data that is uploaded to a website so that you can use historical data.

Basically what this script does is each time it is run, it looks at the contents of the current realtime.txt file and saves it to a logfile called realtime.log.

It slightly changes the date field so that is uses and ISO date format which is sortable. This is useful for keeping the data in order for more advanced functions.

The data that is stored is what is contained within the realtime.txt file.

This should work on any PHP5 based Unix web server. I have not tested it on a Windows based PHP5 server and I really don't know much about IIS with PHP installed.

Ideally, this script needs to run once a minute. This can best be achieved by using cron (Crontab) on the web server. Most Unix based Web Servers and ISP's support this type of function. If not, there are a few options:

1) use something like Pseudo-Cron. This works okay if your website has a lot of traffic. If it doesn't, then you will easily miss the 1 minute increments.

2) Execute the script remotely using your workstation. Unfortunately, this is not a function within Cumulus, but you could schedule the calling of the script via the net. I don't have specifics on how to do this... but if someone needs it, I'm sure it can be found quickly enough. Of course this means your computer needs to be on (Cumulus as well)....

Another issue is that Cumulus needs to be uploading the realtime.txt file. It should be uploading it at least once a minute. Any faster than that won't matter much.

Configuration

Some is needed....

Code: Select all

$SITE['home']           = "YOU MUST SET THIS";
$SITE['datafile']       = "realtime.log";
$SITE['source']         = "realtime.txt";

The Home Variable needs to be set to your home directory on the web server. You can get this information from the Php Server Check Script.

The Datafile variable should be okay... Most of the other scripts that will use this data will be setup to use the log as that where it is.

Same for the source variable. If this script is loaded on your server in the same location as where the realtime.txt file is uploaded to, you won't need to change it.

You MUST create the data file

In order for this script to work, the datafile MUST existing and it MUST have 0666 (rw-rw-rw) permissions. This would be

Code: Select all

chmod 0666 realtime.log

in most Unix environments and cpannel like interfaces should be able to do this.

You might need to chmod the script realtimelog.php as well to something like chmod 0755.

Setting up the Cron Entry

As mentioned above, the best way to execute this would be via cron.

A typical crontab entry for this script would be:

Code: Select all

* * * * * /path_to_php_cli/php /path_to_your_webroot/realtimelog.php > /dev/null 2>&1

You must know the location of the PHP CLI command. You might need to ask where that is located...

Source Code:
TNET Cumulus Project Page

What the logfile will look like:

Almost like the realtime.txt file except that it will have a changed date field and it will contain an ever growing number of entries which will be the historical information of your site.

Sample:

Code: Select all

2008-10-11 09:15:16 12.2 94 11.3 7.7 7.8 231 0.0 14.6 1019.1 SW 3 kts
2008-10-11 09:16:16 12.2 94 11.3 8.2 6.1 232 0.0 14.6 1019.1 SW 3 kts
2008-10-11 09:17:16 12.2 94 11.3 9.6 12.2 244 0.0 14.6 1019.2 WSW 3 kts
2008-10-11 09:18:16 12.2 94 11.3 11.3 5.2 218 0.0 14.6 1019.2 SW 4 kts
2008-10-11 09:19:16 12.2 94 11.3 10.1 6.1 248 0.0 14.6 1019.1 WSW 3 kts
2008-10-11 09:20:16 12.2 94 11.3 9.0 10.4 225 0.0 14.6 1019.1 SW 3 kts
2008-10-11 09:21:16 12.2 94 11.3 9.8 6.1 240 0.0 14.6 1019.2 WSW 3 kts
2008-10-11 09:22:16 12.2 94 11.3 10.9 11.3 235 0.0 14.6 1019.3 SW 4 kts
2008-10-11 09:23:16 12.3 94 11.4 12.0 13.9 242 0.0 14.6 1019.3 WSW 4 kts
2008-10-11 09:24:16 12.3 93 11.2 12.6 9.6 245 0.0 14.6 1019.3 WSW 4 kts
2008-10-11 09:25:16 12.3 93 11.2 12.7 13.0 243 0.0 14.6 1019.3 WSW 4 kts
2008-10-11 09:26:16 12.3 93 11.2 12.1 9.6 230 0.0 14.6 1019.2 SW 4 kts
2008-10-11 09:27:16 12.3 93 11.2 11.5 9.6 244 0.0 14.6 1019.2 WSW 4 kts
2008-10-11 09:28:16 12.3 93 11.2 11.8 7.8 244 0.0 14.6 1019.2 WSW 4 kts
2008-10-11 09:29:16 12.3 93 11.2 11.9 8.7 240 0.0 14.6 1019.2 WSW 4 kts
2008-10-11 09:30:16 12.3 93 11.2 10.5 8.7 235 0.0 14.6 1019.2 SW 3 kts
2008-10-11 09:31:16 12.3 93 11.2 9.4 8.7 235 0.0 14.6 1019.2 SW 3 kts
2008-10-11 09:32:16 12.3 93 11.2 9.3 7.0 236 0.0 14.6 1019.3 SW 3 kts
2008-10-11 09:33:16 12.3 93 11.2 9.1 8.7 240 0.0 14.6 1019.3 WSW 3 kts
2008-10-11 09:34:16 12.3 93 11.2 10.2 9.6 236 0.0 14.6 1019.3 SW 3 kts
2008-10-11 09:35:16 12.2 93 11.1 12.4 6.1 243 0.0 14.6 1019.3 WSW 4 kts
2008-10-11 09:36:16 12.2 93 11.1 12.9 8.7 244 0.0 14.6 1019.3 WSW 4 kts

Debug...

This script if called with debug will output debug information which might help if you are having problems.

Call the script like

Code: Select all

http://YOURWEBSERVER/realtimelog.php?debug

and it will output debug information as it runs like:

Code: Select all

DEBUG MODE ON
Current directory is: /opt/coolstack/apache2/vhosts/cumulus
Changing directory to: /www/vhosts/cumulus
Current directory is: /opt/coolstack/apache2/vhosts/cumulus
Attempting to load: realtime.txt
We loaded: 1 - Should be 1
Saving Data as: 2008-10-11 01:43:19 65.1 37 38.2 0.0 0.0 258 0.00 0.00 1001.8 WSW 0 mph
COMPLETE

[pinto]

Maybe an aditional tip:

I had to chmod 755 realtimelog.php

Jozef

[TNETWeather]

.
Version 2.0

This version simply changes how the data is collected and saved. The original script looked for the specific fields and wrote them out.

This script loads the number of fields found and writes them out. If the number of fields increased due to a change in the format of the realtime.txt file, it will continue to work.

[arthurhh]

As a follow up this script works fine on a WAMP (Windows PHP Mysql Apache) machine with PHP5 for windows installed. I had to use a seperate cron program to call it every 1 minute. No problems with overloading my system as Virtual cron seems to have low impact on system resources

[gwheelo]

Needless to say - I am way in over my head and sinking. I am attempting to install and use the RealTimeLog script and know nothing about php or CRON.

My web host requires that they set up any CRON activity and I have asked them to install the RealTimeLogScript. They have responded that they have and a new file has appeared in my "cumulus" directry called "cron.php." If I access that page I receive the following error: Parse error: syntax error, unexpected T_CONCAT_EQUAL in /users/web/gwheelo/web/cumulus/cron.php on line 153.

Not knowing what I do to enable logging or dealing with the error, I am humbly turning to smarter minds for help. A copy of the CRON.php file is attached.

Respectfully,

George Wheelock

I am receiving a message that "The extension php is not allowed so the script is copied below.

Code: Select all

<?php
############################################################################
# A Project of TNET Services, Inc.
############################################################################
#
#   Project:    Cumulus Project
#   Package:    Cumulus Weather Logging
#   Module:     realtimelog.php
#   Version:    2.0 - October 17th, 2008
#   Purpose:    This script should be called from cron or some type of
#               scheduling system once a minute to extract out the
#               contents of the current realtime.txt file and store
#               it in a logfile.
#   Authors:    Kevin W. Reed <programs@tnet.com>
#               TNET Services, Inc.
#   Copyright:  (c) 1992-2008 Copyright TNET Services, Inc.
#
#   License:   
#
#   This program is free software; you can redistribute it and/or
#   modify it under the terms of the GNU General Public License
#   as published by the Free Software Foundation; either version 2
#   of the License, or (at your option) any later version.
#  
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#  
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307,
#   USA.
#
############################################################################
#   HISTORY
############################################################################
#
#   1.0     Released October 10th, 2008
#           Initial Release
#
#   2.0     Released October 17th, 2008
#           This version works with N lengthed output from the realtime.txt
#           file so that if it is expanded, it simply starts logging the
#           expanded variables.
#
############################################################################
#   This document uses Tab 4 Settings
############################################################################
$DATA   = array();
$SITE   = array();
check_sourceview();
############################################################################
# CONFIGURATION INFORMATION
############################################################################
# NOTES
#
# The home should point to your webservers home directory.
#
# The datafile MUST exist and MUST be chmod 0666 to work as the webserver
# Needs to have permission to write to this file.
#
# The source should point to the realtime.txt file that you are uploading
# to your server.  You should be uploading this at least once a minute
#
# This script should be called via cron or similar scheduler. It must be
# called using PHP CLI... so that it can collect the data properly.
#
# A sample cron entry would look like:
#
# * * * * * php /path_to_your_webroot/realtimelog.php > /dev/null 2>&1
#
# You may need to chmod the realtimelog.php script to 0755.
#
############################################################################
$SITE['version']        = "2.0";
$SITE['home']           = "/users/web/gwheelo/web/cumulus";
$SITE['datafile']       = "realtime.log";
$SITE['source']         = "realtime.txt";
#---------------------------------------------------------------------------
$SITE['debug']          = 0;
############################################################################
check_debug();

// Change directory to home so that we know where we are

if ($SITE['debug']) {
    echo "Current directory is: " . getcwd() . " <br/>\n";
    echo "Changing directory to: " . $SITE['home'] . " <br/>\n";
}

if ($SITE['home'] != "/users/web/gwheelo/web/cumulus") {
    chdir($SITE['home']);
} else {
    echo "You have not set the home directory<br/>";
    exit;
}

if ($SITE['debug']) {
    echo "Current directory is: " . getcwd() . " <br/>\n";
}
// Load the current contents of the source file (normally realtime.txt)

if ($SITE['debug']) {
    echo "Attempting to load: " . $SITE['source'] . " <br/>\n";
}
$rawdata = file($SITE['source']);
if ($SITE['debug']) {
    echo "We loaded: " . count($rawdata) . " - Should be 1<br/>\n";
}

// Extract the data into an array

$DATA = preg_split('/ +/', $rawdata[0]);

if ( $SITE['debug'] ) {
    echo "Found " . count($DATA) . " Fields<br/>\n";
}

// Fix date so that it is in ISO format so that it is sortable
   
$DATA[0] = "20" . substr($DATA[0],6,2) . "-" . substr($DATA[0],3,2) .
    "-" . substr($DATA[0],0,2);

// Open up the datafile and write out to the end of the file the new
// Information

// Output data with spaces between each field
// It doesn't matter how many fields there are

$fp = fopen( $SITE['datafile'] , 'a' );
if ($fp) {
   
    // Collect all the fields into the variable $info
    $info = "";     # Empty info field
    $first = 1;     # Flag for first variable
    foreach($DATA as $key) {
        // Trim extra spaces off the end (Removes \n's as well)
        $key = trim($key);
       
        // If there is anything left...
        if (strlen($key) > 0 ) {
           
            // Was this the first variable
            if ($first) {
                // yep..
                $first=0;
            } else {
                // Nope... add a space
                $info .= " ";
            }
            // Add the field to the end of the info variable
            $info ..= $key;
        }
    }
   
    // If debug, display what we ended up with
   
    if ($SITE['debug']) {
        echo "Saving Data as: " . $info . "<br/>\n";
    }   
   
    // Write the data
   
    fwrite($fp, $info . "\n" ) ;
    fclose($fp);
} else {
   
    // Opps... we had an error opening the file
   
    if ($SITE['debug']) {
        echo "UNABLE TO SAVE DATA - CHECK PERMISSIONS <br/>\n";
    }   
    echo "Unable to open file<br/>";
}

if ($SITE['debug']) {
    echo "COMPLETE<br/>\n";
}  
// We are done
exit;

function check_debug () {
    global $SITE;
   
    if ( isset($_GET['debug']) ) {
        $SITE['debug'] = 1;
        echo "DEBUG MODE ON<br/>\n";
    }
}

function check_sourceview () {
    global $SITE;
   
    if ( isset($_GET['view']) && $_GET['view'] == 'sce' ) {
        $filenameReal = __FILE__;
        $download_size = filesize($filenameReal);
        header('Pragma: public');
        header('Cache-Control: private');
        header('Cache-Control: no-cache, must-revalidate');
        header("Content-type: text/plain");
        header("Accept-Ranges: bytes");
        header("Content-Length: $download_size");
        header('Connection: close');
        readfile($filenameReal);
        exit;
    }
}
############################################################################
#   END OF MODULE
############################################################################

[TNETWeather]

.

The error is pointing at line 153 which should contain...

Code: Select all

        if (strlen($key) > 0 ) {
            
            // Was this the first variable
            if ($first) {
                // yep.. 
                $first=0;
            } else {
                // Nope... add a space
                $info .= " ";         # <=== only 1 decimal point.
            }
            // Add the field to the end of the info variable
            $info .= $key;
        }

Your copy of the script contains for that line instead:

Code: Select all

$info ..= $key;     # has two decimal points should be only 1

Also at the very top of the script your path is wrong.

You have

Code: Select all

$SITE['home'] = "/users/web/gwheelo/web/cumulus";

When it should be

Code: Select all

$SITE['home'] = "/users/web/gwheelo/cumulus";

[gwheelo]

Brilliant! - logging away!

Where the double decimal came from is a mystery - the script was a cut and paste to my hosting service. I have gone back to your original script and that was correct - so clearly an error on my end.

The "home" directory, however, was correct in my script. See:

Does PHP work?
Yes, it does. Your server has PHP Version 4.4.3
Current GD status:
• GD support: YES
• GD Version: bundled (2.0.28 compatible)
• FreeType Support: yes
• FreeType Linkage: with freetype
• T1Lib Support: no
• GIF Read Support: yes
• GIF Create Support: yes
• JPG Support: yes
• PNG Support: yes
• WBMP Support: yes
• XBM Support: yes
• JIS-mapped Japanese Font Support: no
Home Directory: /users/web/gwheelo/web/cumulus
If you don't see "Yes, it does." in large font above, then PHP is not enabled.
For most Graphic scripts to run, you must see GD support with YES
For scripts that require TrueType support, it must say YES for the line FreeType Support

Now - what happens when realtime.log grows and grows - one day it will be larger than required! How is it controlled?

Next step - on to the graphs.

Thanks for all your help!

GW

[TNETWeather]

.
Ever growing log files...

Been thinking about this for a while.... and kept running into different issues as I thought it through.

The easy sounding solution was to have the realtimelog.php script write to a dated file for each month. When a new month starts, it simply creates a new month log and starts to log into that.

However, that makes for a complicated trick at the begining of the month when you attempt to show the past number of hours for a data file that doesn't contain that much data. It's doable, but complicated.

So, I have come up with a different multi-part solution instead that allows for detailed data save by the month and a logfile that remains reasonable for simple scripts to use.

Add an Automatic Trim

Really only needs to be done once a day, but could be done more often. At a specified time, the realtimelog.php script will truncate its data by rewriting the output to the log with only the last 96 hours worth of data. This would keep the log small enough for detailed graphs without getting out of hand.

It sounds like a lot of work but it is actually quite fast and easily done in seconds.

Changing output to quarter hour

Right now, the realtimelog.php script outputs once a minute to the realtime.log file. The graphs only use entries in it at most on the quarter hour though. I planned on adding an option to the script that would reduce the output of that log to the quarter hour (saving only on the 00 15 30 and 45 minute increments) to the realtime.log file. This would reduce the growth of the log from 1440 lines per day to 96.

However, these changes would lose some valuable data... so I plan on adding a monthly log with full detail...

Add a monthly log output.

This would be saved to just like the current realtime.log file except it would only stores that months data. This would automatically be created each new month when the next data arrived.

Code: Select all

realtime-2008-10.txt
realtime-2008-11.txt
etc...

This would provide historical data in simple logfile format of the data that would be maintained. It could even be compressed at the end of the month for even more reduction but for most that would not be necessary.

From the monthly data, you could come up with Hi/Low info from the server side etc...

[gwheelo]

Sounds like a great solution, perhaps even more flexible to use than one long file. I take it that we will be able to modify the current php script once you have finalized the code.

Thanks again,

GW

[andro700]

I am having a problem changing my permissions on my realtime.log. Here is what I am getting

550 CHMOD 666 realtime.log: Operation not permitted

Here is my debug page.

DEBUG MODE ON
Current directory is: /virtual/users/e14497-14796/web/cumulus
Changing directory to: /virtual/users/e14497-14796/web/cumulus
Current directory is: /virtual/users/e14497-14796/web/cumulus
Attempting to load: realtime.txt
We loaded: 1 - Should be 1
Found 43 Fields
Saving Data as: 2008-10-29 21:44:38 45.2 64 33.5 0.0 0.0 167 0.00 0.00 29.86 SSE 0 mph F in in 102.2 -0.01 0.98 1.14 0.00 79.5 28 45.2 -3.1 68.3 15:50 30.4 03:02 13.0 12:54 20.0 11:52 30.20 00:00 29.86 20:00 1.8.3 482 2.0 45.2 5.4
COMPLETE

Anybody have any ideas on how to fix this.

Chuck

[andro700]

I was wondering if anybody has a follow up to the reason I am unable to set the permissions to 666 on the realtime.log.

Thanks,
Chuck

[TNETWeather]

Your web host might be restricting it or the tool you are using doesn't have permission to make that change.

Sometimes you have to use 777 instead. Even though it is not a executable or directory.

[bpsmicro]

I've been playing with this a bit, so that i could play with the JpGraph stuff. But I'm having a heck of a time finding a suitable mechanism to run .php scripts every minute on a Win2003 server that doesn't involve running what's essentially a DOS program (PHP) in an endless loop and praying that it doesn't negatively effect the server (which is a domain server).

Just for my own education what's missing in the existing realtime.txt file that the JpGraph stuff can't use (that requires essentially one text file to be parsed into a second text file every minute)?
Is there a prayer this whole "realtime.log" thing could be added in the actual Cumulus program at some point, thus eliminating the need for this script entirely?

Brad.

[steve]

Just for my own education what's missing in the existing realtime.txt file that the JpGraph stuff can't use (that requires essentially one text file to be parsed into a second text file every minute)?

The realtime.txt file is just a snapshot of conditions at that point in time. The graphs need a set of points to plot. So the script appends a snapshot periodically to a file to give that set of points.

Is there a prayer this whole "realtime.log" thing could be added in the actual Cumulus program at some point, thus eliminating the need for this script entirely?.

I'm not really up to speed on any of this stuff, but as I understand it, having helped Paul set his up, it's just a case of executing a PHP script once a minute, so it's rather like updating Weather Underground. That being the case, I can't see any reason why Cumulus couldn't execute the script once a minute. I could put a general 'hit this page every n minutes' type thing in, where you specify the URL and the interval.

Steve

[bpsmicro]

That would be nice, since you already have a timer running. But I wonder if it's not just easier to just generate/append the entire file yourself and eliminate the PHP script need entirely. Just tossing that out as a "perhaps easier" option (probably not, but I mention it anyway).

My end goal is to look at coming up with a "last 24-hour" outdoor temp chart similar to the current Trend chart, but adding in lines for wind chill and humidex (yes, I know technically wind chill & humidex aren't "temperatures", but they're used as such for the masses in Canada). I know very little about PHP, but it looks like something I might have fun playing with in my copious spare time.