Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Cumulus RealTimeLog Rotate Script (PHP)

Other discussion about creating web sites for Cumulus that doesn't have a specific subforum

Moderator: daj

Post Reply
TNETWeather

Cumulus RealTimeLog Rotate Script (PHP)

Post by TNETWeather »

Cumulus RealTimeLog Rotate Script
This is designed to work with the realtime.log files that are generated by this project: http://cumulus.tnetweather.com/projects/realtimelog. That project creates realtime.log files from the realtime.txt files uploaded to the server by Cumulus, but it doesn't ever trim the file.

This script does that. I took my existing script that rotates my realtime.log files and pulled it out of my system. All of my scripts use common config scripts to get names, places, files, folders, timezones, configs, colors and even common functions... So the result is not as tidy as the original script, but I really can't distribute the script with my configs in it.
What the script does.
o Can be run from CLI or Web (CLI = Command Line Interface). Personally, I use CLI in a crontab.
o If run by WEB, it requires a code to be provided. This prevents someone from messing with your data by calling your script via the web.
o It reads in the current realtime.log file once
o It then parses the data to find out what months data is in the file
o Next it creates a backup file of the original data into an archive directory.
o Next it writes out Archives of each of the non-current months data to the archive directory.
o Last it over writes the existing realtime.log with just the current months data.

You should like I do, only run this once a month since it will only generate the archive files if the current month is newer than data in the logfile. Personally, I run it from cron on the 4th day of the month after midnight. This ensures there is 48 hours worth of data that I use for charts, trends and things.
Permissions
If you are running this via CLI, you most likely don't need to deal with this. As your cron will most likely run as you and the files you have are owned by you.

If however, you use this via the web, the web server needs to be able to write to the files.

Specifically... Whatever you use for the archive directory most likely should have 0777 (rwxrwxrw) permissions (Note this is a directory, NOT a file). This will allow the webserver to write to new files in the directory.

The realtime.log needs to most likely need to be 0666 (rw-rw-rw-) permissions.

Settings:
The following settings are used...

Your Timezone - Ref: http://php.net/manual/en/timezones.php

Code: Select all

$SITE['timezone']		= 	'America/Phoenix';
Location of your realtime.log file. If the script is in the same dir as the log this should be okay.

Code: Select all

$SITE['realtime_log'] 	= 	'./realtime.log';
Location of your archive data. If you don't have one, you will need to create it (See permissions above)

Code: Select all

$SITE['archive_dir']	= 	'./archives/';
The Secret Code... If you use the script via the WEB, you need to set this to a set of numbers. Letters don't work.

Code: Select all

$SITE['codeval']		= 	intval('SET_ME_PLEASE');
SAVE YOUR SELF A HEADACHE!
If you are trying this for the first time, PLEASE make sure you backup your realtime.log file yourself somewhere so that if it messes it up, you can recover from it. The script does make a backup, but if that file fails you may end up without any of your historical data in the realtime.log.

Backups are a smart thing to do.... Be smart!

DOWNLOAD
You can download this from my project page:

o http://cumulus.tnetweather.com/projects ... elogrotate
W2SWR
Posts: 18
Joined: Thu 08 Mar 2012 2:31 am
Weather Station: WH-1090
Operating System: vista
Location: Little River, SC.

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by W2SWR »

I am having a problem getting it to work with cron. I can manually trip the php via my browser but no dice using cron. here is my cronjob. The php file is installed on my website and I am using the cron on my host. am I missing something?

Code: Select all

0 23 4 * * php /home/mysite/public_html/path/to/rotate_realtime-0.1.php -code="1234"
Image
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by steve »

Have you actually got "/home/mysite/public_html/path/to/rotate_realtime-0.1.php" in there, or have you obfuscated it for posting here? It needs to be the actual path to the script.
Steve
W2SWR
Posts: 18
Joined: Thu 08 Mar 2012 2:31 am
Weather Station: WH-1090
Operating System: vista
Location: Little River, SC.

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by W2SWR »

Yes it is obfuscated and the actual path is good. I am just not sure of the use of "-q", "-code=", and if am missing anything.
Image
TNETWeather

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by TNETWeather »

W2SWR wrote:I am having a problem getting it to work with cron. I can manually trip the php via my browser but no dice using cron. here is my cronjob. The php file is installed on my website and I am using the cron on my host. am I missing something?

Code: Select all

0 23 4 * * php /home/mysite/public_html/path/to/rotate_realtime-0.1.php -code="1234"
Note that Cron does not necessarily have the same environment you have.

You should make sure that you fully qualify the path to the proper php executable (should be the CLI version)...
In my case it is /usr/local/bin/php which most likely is NOT where yours is located.

When you find where it is, you can test to see what version is is with:

Code: Select all

PHP 5.4.3 (cli) (built: May 12 2012 13:29:27) 
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies
    with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator
Note the (cli)...

Some servers have more than the CLI version. I can't recall what it is called, but it will not work the same.
W2SWR
Posts: 18
Joined: Thu 08 Mar 2012 2:31 am
Weather Station: WH-1090
Operating System: vista
Location: Little River, SC.

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by W2SWR »

I am a real newby at cron with CLI. can you give a full example of how you would set up a job to run this file assuming its usr/bin/php. I will find out from my host its location.

thanks
mike
Image
W2SWR
Posts: 18
Joined: Thu 08 Mar 2012 2:31 am
Weather Station: WH-1090
Operating System: vista
Location: Little River, SC.

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by W2SWR »

I got it to work using
0 23 4 * * lynx -dump 'http://mysite.com/path/to/rotate_realti ... ?code=1234' > /dev/null 2>&1
But is this the best way ?
Image
Mapantz
Posts: 1774
Joined: Sat 17 Dec 2011 11:55 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 11 x64
Location: Dorset - UK
Contact:

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by Mapantz »

Hi
I wonder if somebody kind enough could help me?

I'm using the realtimecumulus graph on my front page, and I'm using TNET's script to copy data minute-by-minute to a file called realtime.log (done by a cron job)
I then downloaded the rotate script to enable the trim feature, so that realtime.log didn't get too big. The problem is, the realtimecumulus graph stops working when the realtime.log gets somewhere around 1600KB, and the rotate script doesn't trim anything that small.

I was hoping someone who knows what they are doing, could possibly tinker with the script, so that it can trim my realtime.log down to 24 hours of data when the script is run? It could even be trimmed down to keep the last 3 hours of data to be fair, as that is all I show on the graph.

Thanks in advance.
rotate_realtime.txt
You do not have the required permissions to view the files attached to this post.
Image
User avatar
N0BGS
Posts: 205
Joined: Sat 10 Nov 2012 2:26 am
Weather Station: Davis Vantage Pro 2
Operating System: Win10vm,VMWare ESXi 7.0
Location: Hermon, Maine USA
Contact:

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by N0BGS »

Kind of late responding...but I run this script in a cron job on my Linux server to trim realtime.log

Code: Select all

#!/bin/bash
cat /var/www/html/realtime.log  > /var/www/html/realtime.tmp
sleep 5
rm -f /var/www/html/realtime.log
tail -qn 675 /var/www/html/realtime.tmp > /var/www/html/realtime.log
rm -f /var/www/html/realtime.tmp
Don't know if you could adapt this to your situation or not.
Blitzortung Station 1809
Mapantz
Posts: 1774
Joined: Sat 17 Dec 2011 11:55 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 11 x64
Location: Dorset - UK
Contact:

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by Mapantz »

N0BGS wrote:Kind of late responding...but I run this script in a cron job on my Linux server to trim realtime.log

Code: Select all

#!/bin/bash
cat /var/www/html/realtime.log  > /var/www/html/realtime.tmp
sleep 5
rm -f /var/www/html/realtime.log
tail -qn 675 /var/www/html/realtime.tmp > /var/www/html/realtime.log
rm -f /var/www/html/realtime.tmp
Don't know if you could adapt this to your situation or not.
Hi - Thanks for the reply. I got it working in the end, but now i've ran in to another problem. Since the turn of the month, the realtime.log is not being trimmed properly anymore. It is gowing too big because it has April 30th's minute by minute data and May 2nd's - No May 1st. It also caused it to be corrupted (which I fixed by deleting spaces) The rotate script fails at the turn of the month.. no idea how to fix that, it's beyond my knowledge unfortunately. :oops:

How does yours work? Where do you put your code to for the cronjob to run it?
Image
User avatar
N0BGS
Posts: 205
Joined: Sat 10 Nov 2012 2:26 am
Weather Station: Davis Vantage Pro 2
Operating System: Win10vm,VMWare ESXi 7.0
Location: Hermon, Maine USA
Contact:

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by N0BGS »

Looking at this line by line:

#!/bin/bash
cat /var/www/html/realtime.log > /var/www/html/realtime.tmp < this line copies realtime.log to realtime.tmp
sleep 5 < wait 5 seconds before proceeding
rm -f /var/www/html/realtime.log < delete realtime.log
tail -qn 675 /var/www/html/realtime.tmp > /var/www/html/realtime.log < read 675 lines of realtime.tmp (about 12 hours worth. You can change that number) and copy the trimmed file back to realtime.log
rm -f /var/www/html/realtime.tmp <delete the realtime.tmp file

The file, in my case it's named trimrealtimelog, has to be executable so use the Linux command:

Code: Select all

chmod 744 trimrealtimelog
to make it so.

I run this every 15 minutes in a crontab like this:

Code: Select all

0,15,30,45 * * * * /bin/bash /var/www/trimrealtimelog >/dev/null &2>1
Make sense?
Blitzortung Station 1809
water01
Posts: 3215
Joined: Sat 13 Aug 2011 9:33 am
Weather Station: Ecowitt HP2551
Operating System: Windows 10 64bit
Location: Burnham-on-Sea
Contact:

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by water01 »

It's a good bet that the "tail" command has different parameters in W10, as it did when I switched it from windows XP to W7. I would check that first as that is probably why the file is getting so large it isn't being "tailed".

Edit. Just checked and the tail command is not in the standard W10 install so you will have to get an executable from the internet.
David
Image
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Cumulus RealTimeLog Rotate Script (PHP)

Post by mcrossley »

The tail I tried in Windows 7 was seriously broken - can't remember the details now, but I found it unusable when I was using the realtime log file. Using a DB is the way to go! ;)
Post Reply