Database v text files

[Sadgit]

Hello

I've used Cumulus since, well a very long time with my older stations etc. I have been away from Cumulus for sometime, as it just worked. With a new weather station arriving soonish, my appetite has been whetted again. I've spent all day since 6:30am fannying about trying to teach myself html/css to go back to basics and create my own site. Thanks to some guidance from mapantz.

My brain now hurts and I am wondering, should I start to use the SQL Database and of so, does that do away with the text files? (they are great to backup) Thoughts please?

Cheers
Mark

[HansR]

For what it's worth.

To be honest, I think you just posed a question with which we can continue into eternity replying to

You can argue for and against (as you probably already did). I argue against, because you do not really require complex queries. Meteorology is just about time series, straightforward and simple with some statistics added. And if you want something more complex (e.g. you want an average of every first hour of every third day of all months of May you ever measured) it will require some thought in SQL as well. And whatever you do, in the end the data needs to be in some memory structure, some variable, to handle it. My guess is, from text file, that is much more easy to accomplish.

I made my choice some time ago for text files and can tell you, it is not the data selection you need to think about. SQL or text based, just as easy or difficult. Actually, if it were to go to SQL I would probably dump the database into txt files and start from there
It is the presentation which makes the difference.

I like simplicity, so I choose for as little tools as possible, as little transfers as possible and as much automated as possible.

And btw,

I've spent all day since 6:30am fannying about trying to teach myself html/css to go back to basics and create my own site

I don't think HTML/CSS is enough to create a data driven site. Transparency for the user is important, keep complexity behind a wall. But if you make your own site, there can be no wall.

[water01]

I don't think HTML/CSS is enough to create a data driven site

?????

But that is what the Standard Cumulus Website with Cumulus MX providing data via webtags is!!

[HansR]

But that is what the Standard Cumulus Website with Cumulus MX providing data via webtags is!!

So there is quite a world behind HTML/CSS to drive that standard site.

[freddie]

should I start to use the SQL Database and of so, does that do away with the text files? (they are great to backup) Thoughts please?

Hi Mark,

To answer the second question that you posed: using a database does not do away with the text files. Cumulus generates them regardless of whether you use a database.

[Mapantz]

The first reason I started using a database was because I wanted fancy looking graphs that mcrossley produced. Then, a data summary script appeared that used SQL. The first thing I noticed was that it loaded the data much quicker than it did by reading from the dayfile.txt.

Although I have broadband, my upload speed is terrible. So the less I upload, the better. I get CMX to insert all of my data to SQL now (apart from the NOAA-type display)

My records pages read from SQL as do every single graphs page.

I don't even upload any standard files anymore, i've got everything I need in 3 small custom files.

I just like the efficiency of it all.

[sfws]

trying to teach myself html/css to go back to basics and create my own site.

That is a valid way to vary the standard web pages, which as water01 says use templates filled with web tags to put values into the HTML. So changing the content, means changing HTML, changing the look means editing the CSS.

I am wondering, should I start to use the SQL Database and of so, does that do away with the text files? (they are great to backup)

1. As Freddie says the database is an alternative way to access past data, the text files are still used and updated.
2. You can back up SQL databases too. There are various ways that I won't specify for now.
3. If your brain hurts from learning HTML and CSS, it is not the right time to learn SQL, but when you do learn SQL you will find it makes querying large database tables much easier than doing the equivalent search in text files. So you will be able to do very much more than you can do just using web tags, again I won't give specific examples here.
4. There are of course other script languages you could learn, but my view is that they are more difficult than HTML, CSS, SQL. These include PHP (a hypertext preprocessor that can generate HTML and it can run SQL against a variety of databases; for example I use it to interrogate the MX weather diary [SQLite] as well as analyse all days [mySQL] in any month for stuff I cannot get from monthly web tags) which is parsed in a web server, a bit like Cumulus parses a template file with web tags and then generates a HTML web page. There is JavaScript which runs in the browser where a web page is viewed and can vary the HTML content. Then there are other languages like C# (in which MX is written) that can generate so very much more as Hans says, but I suspect is even harder to learn, I might try to learn it in future, but I find the others are enough for now.

As Mapantz says, you can leave the text files as local files MX processes, and on your web server have a set of web pages that depend on SQL enquiries for their content. You still need the HTML and CSS knowledge for designing those web pages. Despite Hans' claims on benefits of simple systems running on client's machine only when needed; I believe there are security advantages in doing all processing on web server and letting the end user see no more than standard HTML web pages, hiding the database so it cannot be hacked.

[Sadgit]

thanks all.. I will enable SQL as I can see the benefits from viewing Mapantz graphs and even more so if the text files are still generated. PS where do you get these scripts please? I particularly love the wind ones..

I'm won't be learning SQL, I used to be a DBA erm nearly 30 years ago, saying that also a software engineer (ada 95, now that was a pain!!:D) , but I long left that behind. I've just never looked at web design and even though I have used CU for over a decade I have always used someone elses templates aka now Ken's and the default CU one but I have decided not thing really suits my wants/needs, hence time to play. I got so far using html, but now I need to use php to make it work and now a bit stuck

If anyone can point me to a working php version of say yesterday/today page that would be really appreciated. I tried to pull Ken's apart but that is far more complicated that I need/want right now

Cheers
Mark

[HansR]

but that is far more complicated that I need/want right now

That, actually, is the whole problem of all php-based templates / 'stand-alone' procedures.
If you really want to grow into a technique, start from scratch either with the SQL database or the text based files.
Then do some exercises before you definitively decide on your technique to use. Look at the number of scripts, transport actions, ease of programming (how do you feel with a language) etc.... Oh, and take a look where I got in several months time, the actual website took me two weeks, the reports I made earlier

[sfws]

If anyone can point me to a working php version of say yesterday/today page that would be really appreciated

BCJKIwi has a today/yesterday page written in php see viewtopic.php?f=18&t=17845&start=30#p138283. I think that might fit in with your CU usage.

A very long time ago, see viewtopic.php?f=14&t=8631&start=30#p71770 I developed both JavaScript and PHP scripts that showed two days - yesterday and the same day a year earlier. I know I, and some other people (you might see from that topic who else used it), modified it to show today as well.
For showing today, yesterday and a year ago, there is a non-PHP version at http://christelow.dyndns.org/Website/files/today.htm and a PHP version at https://warehamwx.co.uk/tod_yes.php. The owners of those sites might help you.

My own version has so much extra in it, it would muddle you if I were to share the whole file, but this might give you a taster:

Code: Select all

<tr>
				<th class="label">Total Rainfall</th>
				<td class='site_data'><?php echo "$rfall $rainunit"; ?></td>
				<td></td>				
				<td class='site_data_RH'><?=$rfallY . ' ' .  $rainunit;?></td>
				<td></td>				
</tr>

That is using PHP variables that are set as follows in another file that is processed by Cumulus:

Code: Select all

$rfall = <#rfall>;
$rainunit = '<#rainunit>';
$rfallY = <#rfallY>;

It also illustrates two ways of displaying PHP variable in HTML, using 'echo' and using the '=' shorthand for echo.
The blank table cells allow you to put time-stamps there in rows for other derivatives.

[Sadgit]

thanks.. I'm not the best at this kind of thing as you can see.. I am good at stealing but not creating

http://www.marton-weather.co.uk/test/index.htm

just need to try and php the htm file

[water01]

You might like a MX version of Cumulus webtags for MX. I converted this one some time ago viewtopic.php?f=39&t=13252&p=103860&hil ... xt#p103830 and I still use it for my PHP website so it still works. I seem to remember someone updated it but I searched the MX forums and couldn't find it.

You need to add it as an extra file upload and process it and then upload it as cumuluswebtags.php as per these instructions in the wiki https://cumuluswiki.org/a/Php_webtags .

Don't forget "In your customised pages, use the instruction require cumuluswebtags.php" (i.e. at the top in between <head> and </head> put <?php require "cumuluswebtags.php";?>
and ignore "IMPORTANT This file is only for use with Cumulus v1.9.4, it will not work with Cumulus MX. " as this file has been modified to work with MX.

[sfws]

I seem to remember someone updated it, but I searched the MX forums and couldn't find it.

David, the latest version is by BCJKiwi and you probably saw it in his CumulusMX and Cumulus1 UI style Multilingual Websites topic, for simplicity I have attached it here. It is missing a lot of web tags that have been added to MX as it is designed to work with both Cumulus 1 and MX.

Of course there have been many new web tags added to MX since 2015 you last edited your version, so although your version still works it restricts users to which web tags they can access. Compare the tags in your version with the latest list on the web tag wiki page.
That actually includes a fairly recent list of all the tags available in MX, but there are more since even that list. The wiki page on web tags at the top describes how to get Cumulus to output all available tags in whatever version you use so you can check all that are available.

The real reason I think people have given up issuing new versions of that file for general use is not just the difficulty of keeping the web tag content up to date, but also the difficulty of predicting what input and output modifiers someone else might want. This is something Ken True pointed out when he made such a file for his Saratoga Templates, his file only contained what his templates needed.

My own version uses lots of output formatters, and where applicable a range of input modifiers, so I can choose PHP variables that have times in different formats, and so I can use the recent tags for a range of different past times. I also use several different files updated at different intervals, for example the yesterday tags don't change during the day, so can be set to today tags but uploaded at end of day. I upload yesterday tags at end of day to give me the day before. It is something I'm still refining since I moved to MX and had access to EOD uploads. But keeping my files updated for all the new tags that Mark has introduced is another challenge I won't get on top of while I'm busy in the garden. (I'm only indoors now because I have had to put the mower battery on charge).

[Sadgit]

thanks both.. I'll have a play but looks like I won't have enough time to do my own template. It is more complicated than I 1st thought

[Sadgit]

well well well... this is so easy!! I think my brain has finally worked it out. I have a index.php, calling a header.php, menu.php and using the parsed php web tags.. Now for the hard part, making it look half decent and deciding on what I want on there.. I don't have an artistic bone in my body !!