Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

a PHP import script for DayFile and Monthly log files

Discussion and support for 3rd-party (non-Sandaysoft) tools for Cumulus
Post Reply
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

a PHP import script for DayFile and Monthly log files

Post by daj »

I have written a PHP script to read the contents of the dayfile.txt or a Monthly log file and import these into a MySQL database on your webserver.

Handy if you want to get the Cumulus data onto the web to query it and write your own reports/graphs directly from SQL

Full details on the wiki http://wiki.sandaysoft.com/a/ImportCumulusFile

As an example, I upload my dayfile.txt just after midnight each day and then process it using this script. The result is my Historic Data page ... http://www.grantownweather.co.uk/historic/historic.php... which reader the data live from the SQL database and presents it in a table for viewing.

All my monthly log files are also in SQL so I am working on analysis tools for those too
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by nitrx »

Goodmorning David
I get errors at first I got an error could not conect to database I changed the databasehost from nummeric to 'localhost' but then I get the error 'failed to open file' , I'm sure its in the directory I've specified. :|

When I succed how do I present de data on a webpage ? I even dont understand the securuty key when the url to proceed the script is on the web.

Looking forward to use this future anyway
User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by nitrx »

Oep fixed I forgot a trailing point . for the import url I used /dayfile.txt instead ./dayfile.txt so the data are imported
my other questions are still open :D

Edit only one record is imported and the datafield is 0000-00-00 I think its again about the delimiters

LogDate HighWindGust HWindGBear THWindG MinTemp TMinTemp MaxTemp TMaxTemp MinPress TMinPress MaxPress TMaxPress MaxRainRate TMaxRR TotRainFall AvgTemp TotWindRun HighAvgWSpeed THAvgWSpeed
Wijzigen Verwijderen 0000-00-00 6 N 2;12: 4 2;19: 2 5;1 8 6;4 1 0 0 0
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by daj »

Ron

I will make a modification to the code to allow you to easily change the delimiter. You could edit the script yourself but let me make it a little easier....two minutes

On your other questions....

How to present the data...that is out with the scope of the script. All we are doing with the script is providing a mechanism to get the data into a SQL database. There are hundreds of tools that can extract data and present it on a website. For example, JpGraph can pull it's data from SQL; on my site I extract the data into a table to view onscreen;.... I will probably release my table page to the Community too

Not understanding the security key....Not sure what else I can say. We need to provide a way to stop anyone running your import script. So if I knew you were using the script I could probably guess the parameters and could potentially 'screw' up the data by importing the wrong thing. To overcome this I added a simple checking process. You add a phrase/word to the PHP script (the $key=" ") line. Then when YOU call the script you include that key....www.myserver.com/ImportCimulusFile.php. ... ingspecial

If SOMEONE ELSE (say me) attempts to run the script I can not as I do not know you special key

So I already know almost enough to run the script myself....

Code: Select all

http://www.apeldoorn.tk/weer/ImportCumulusFile.php?file=./dayfile.txt&table=MyTable&type=dayfile
But I do not know your key (which you should change in the script!) so I can not run it


Hope that helps
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by nitrx »

Thanks for your explanation David, I'm a complete noob in php and sql programming so it would be great when you can add an import script for my delimeters.

About presenting on the web I only need some examples I'm not intendendig using the same layout but it has no rush.
Thanks in advance
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by daj »

Version 1.1 now released to include the option to change delimiter

Ron, you need to download the update and change all the settings for username, etc again. In addition

Code: Select all

$delimter=",";
should be changed to

Code: Select all

$delimter=";";
to accommodate your semi-colon

let me know how it goes. Make sure version 1.1 is downloaded and you browser does not try to pull a cached version down!
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by nitrx »

Almost good but there is only one record imported the date is 0000-00-00 other values look fine now for your infomation this is how my dayfile.txt is on the server http://www.apeldoorn.tk/weer/dayfile.txt I'm going to lunch ;)
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by daj »

I blame Steve! ;)

The file formats are noted as being comma separated and the date is always in the format dd/mm/yy

You are semi-colon separated and your date is dash delimited.

let me tweak again, and add another delimiter for date
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by daj »

Now updated to include a date delimiter too
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by steve »

daj wrote:I blame Steve! ;)

The file formats are noted as being comma separated and the date is always in the format dd/mm/yy
Comma-separated means different things in different countries, sadly. The files are still called "CSV" files, even though the "C" is not strictly a comma. I guess I should say they are "list-separator separated".

The date format is really just an indicator of the order of the numbers and the number of digits in each part. I guess I should say the format is "dd<some character>mm<some character>yy".

To be honest I had absolutely no idea when I wrote the code that when I told Delphi to use a comma to separate the items that it would be clever and use the system's list separator instead (just as well, really, given that the numbers themselves have commas in them; I didn't know that would happen either). I had no idea either that when I told Delphi to use "dd/mm/yy" as the date format that it would actually not use a slash but would use the system date separator instead.
Steve
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by daj »

Thanks Steve, no worries -- I just needed to blame someone. LOL I might put a note on the Wiki page for the file formats to note that delimiters may change in your country.

It's a learning curve for me (and others too) so we can build that into our thinking in future scripts

The world would be so much easier if we all did the same thing, but probably much more boring
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by nitrx »

Well it works great job (almost) all data are imported but the values don't have decimals ergo -10,5 (temperature) in my case is stored as -10

I think we can't blame Steve we should blame all the diffrent conventions of notations the Mile , inches and feet an Fahrenheits should be forbidden thats why some missiles crash when diffrent nations do the programming of the computers I think :bash:
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by daj »

Anyone want to help here?

I have no idea why 10,5 would drop it's decimal in MySQL.
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
steve
Cumulus Author
Posts: 26702
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by steve »

I would guess that MySQL is expecting a decimal point rather than a comma?
Steve
User avatar
nitrx
Posts: 1297
Joined: Sun 13 Dec 2009 1:21 pm
Weather Station: WH1080
Operating System: Windows 10
Location: Apeldoorn The Netherlands
Contact:

Re: a PHP import script for DayFile and Monthly log files

Post by nitrx »

daj wrote:Anyone want to help here?

I have no idea why 10,5 would drop it's decimal in MySQL.
It doesn't say it's decimal but float (really don't know what it is ..)
Post Reply