Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.9.1 (build 3096) - 31 October 2020 (please see announcement regarding releases since 3.5.0)
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of the NOAA report and Snow Index drop-down menus to 2030)

Use this Wiki link to Download the Software (Cumulus MX / Cumulus 1 and other related items).

Parse dayfile.txt PHP no database

Other discussion about creating web sites for Cumulus that doesn't have a specific subforum

Moderator: daj

TNETWeather

Parse dayfile.txt PHP no database

Post by TNETWeather »

PHP Dayfile Parsing without Database


Released Script:

This is now a released script.

http://cumulus.tnetweather.com/projects.phpdayfile

Code: Select all

v0.11 2010-02-12 Updated windrun units
v0.10 2010-02-10 Initial Beta Release
...

This morning while waiting for access to a server at work, I took a quick look at the dayfile.txt

Seemed like it would be easy to parse and display on the fly without a database, JavaScript etc.. just using PHP.

So I...

o Setup an FTP process on SysBackSE to upload the file once a day at 12 min after Midnight. This actually took most of the time since I've not setup a new job in this for sometime. Seems like they have changed the interface to make it less intuitive over time.
o Wrote a crude script to read the dayfile.txt and store it into an array
o Setup a simple loop looking for the day and year I want to display
o Output the data in a table with CSS tags so that it can be formatted however it needs to be.

Goals were:

o Fast, easy to setup.
o No database needed
o No JavaScript needed
o Try to keep data 700px or less in width so that it fits better on my type of pages.

The dayfile.txt uploaded today was about 48kb in size so parsing it is very fast. Other than doing a date conversion (ISO Date format) so that I could reorder the records if needed, nothing else was really done with the data other than suck it into an Array using file and preg_split.

To keep the width smaller, I didn't bother outputting the times of things like time of max temp etc.. Didn't really have much desire to see that anyway. It is of course there if needed.

I didn't bother doing any sorts on columns other than keep the days in order.

Rough output of the script is:

http://cumulus.tnetweather.com/test/dayfile.php

Used inside a website it looks cleaner...

http://cumulus.tnetweather.com/trends/monthly

Eventually over the years, the file will get larger as it never breaks up into year files. But since it is uploaded only once a day, it would be easy if necessary to just split it up into separate files when parsed. Right now, it is just reading the file, not writing anything.

Needs some tweaking to know what formats are being used, but that info could be obtained from the realtime.txt file itself. Also would need a way to get the file to the server. Cumulus doesn't have such a once a day feature.

Could uses Daz's tool if you don't already have a way to upload.

I just added it to one of my SysBackSE which I already have for other misc weather related stuff including workstation backups.

Oh... options.

If you call the raw script with no options, it takes today's year and month as the data set to use. Using m=XX changes the month, and y=XXXX changes the year.

So if you wanted to see Nov 2009

http://cumulus.tnetweather.com/test/day ... =11&y=2009

As par usual, spent more time documenting that actually using the script.

TNETWeather

Re: Parse dayfile.txt PHP no database

Post by TNETWeather »

If you are looking at this with FF and have WebDev as an addon, you can click on edit CSS and plug in the following for a quick idea of formatting using the CSS tags it outputs.

Code: Select all

#daytable { width: 750px; border-collapse: collapse;}
#daytable th { color: #ffffff; font-size: 75%; background-color: #6B6B6B; vertical-align: bottom; border: 1px solid black; line-height: 120%; text-align: center;}
#daytable td { font-size: 90%; vertical-align: top; width: 9%; text-align: center; border: 1px solid black; padding: 2px;}
#daytable .day { width: 4%; background-color: #6B6B6B; color: #ffffff;}

User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Parse dayfile.txt PHP no database

Post by daj »

Are you going to tell us boys and girls how to access the source code? ;)
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

TNETWeather

Re: Parse dayfile.txt PHP no database

Post by TNETWeather »

Was still doing some clean up...

http://cumulus.tnetweather.com/test/day ... p?view=sce

BETA...

Will be moved to the scripts page a bit later. Want to see a comma delimited version of the day file so I can adjust for that, and also check for units from the realtime.txt file.

Had to take out some hard coded stuff... like where the dayfile.txt is located and make it a setting.

User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Parse dayfile.txt PHP no database

Post by daj »

Sorry Kevin didn't realize it was a teaser for what was to come. Here's my CSV if that helps with testing

http://dl.dropbox.com/u/33488/dayfile.txt
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: Parse dayfile.txt PHP no database

Post by nitrx »

You also would test https://dl-web.dropbox.com/get/sharedda ... w=76fde431 with the semicolum and the dah in the date :idea:
Last edited by nitrx on Wed 10 Feb 2010 10:23 pm, edited 1 time in total.

User avatar
GraemeT
Posts: 308
Joined: Wed 21 Oct 2009 11:19 am
Weather Station: La Crosse WS-2355 & WS-2306
Operating System: Windoze 7
Location: Bayswater, Australia
Contact:

Re: Parse dayfile.txt PHP no database

Post by GraemeT »

Yes, I like Kevin's php solution, too.

I'll have a play with that.
For a start it means I don't have to scan through a 3800-page pdf to work out what to do next in MySQL!
Cheers,
Graeme.

Image

User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Parse dayfile.txt PHP no database

Post by daj »

LOL -- I guess it all depends what you want to do with the data
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: Parse dayfile.txt PHP no database

Post by nitrx »

In mysql you can make queries like 'what was tyhe coolest day in 2009 :D

User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Parse dayfile.txt PHP no database

Post by daj »

Yes. My yearly review queries the SQL for its results. Simple data at the moment but more analysis will follow..... http://www.grantownweather.co.uk/year

You could do the same by iterating through the dayfile, but there is far more flexibility with SQL.

Kevin's script is great for displaying the data in a nice tabular format on the site
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

TNETWeather

Re: Parse dayfile.txt PHP no database

Post by TNETWeather »

Strange that you both have extra delimiters at the end of each line. ...

So far, the only change I needed to make was:

Code: Select all

	$values = preg_split('/,/',$key);
to

Code: Select all

    	if (strpos($key, ";") !==FALSE ){
    		$values = preg_split('/;/',$key);
    	} else {
    		$values = preg_split('/,/',$key);
    	}
Not looking at the realtime.txt file to know what the units would be.

Here is Nitrx's data which is comma decimal.

http://cumulus.tnetweather.com/test/dayfile1.php
I'll have a play with that.
For a start it means I don't have to scan through a 3800-page pdf to work out what to do next in MySQL!
Really depends on what your goals are. Some people prefer to have the data in SQL so they can do some cool stuff with it. This won't replace that. but if all you wanted was a simple list it works.

User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Parse dayfile.txt PHP no database

Post by daj »

TNETWeather wrote:Strange that you both have extra delimiters at the end of each line. ...
Mine were added as part of something else I was working on. Steve recently added an extra two fields to the end of the Dayfile and I now have blank extras on every line.

For my PHP to SQL I had to consider the line having the extra data or not per line
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: Parse dayfile.txt PHP no database

Post by nitrx »

Why is only february displayed ? Youre faster as I can ftp

User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Parse dayfile.txt PHP no database

Post by daj »

TNETWeather wrote: Here is Nitrx's data which is comma decimal.

http://cumulus.tnetweather.com/test/dayfile1.php
Is the "Max Rain Rate mm/hr" correct -- looks more like a time to me
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: Parse dayfile.txt PHP no database

Post by nitrx »

I don't get data (the dayfile.txt is in the same dir) http://www.apeldoorn.tk/weer/dayfile.php well I see after it later my new dayyfile runs in half an hour it's 23:45 over here :roll:

Post Reply