Page 2 of 3

Re: Future data formats

Posted: Sat 05 Jan 2013 10:37 pm
by peterh
While I admire your desire to cater for the desires of your users, I think that, from a code maintenance point of view, you'd be better off ditching the flat-file-format storage thingy.
I mean, srsly. We are storing structured data here... and when you're storing structured data in a flat-file-system, you are essentially emulating a database anyway.

With Entity Framework (available to all of us, except for those who run on Win2k, which is dead anyway), you can support MySql (free), SQL Server, SQL Express (free, I think) and Obstacle, oops, sorry, I meant to say Oracle, officah... with all these engines using the same code. You'd have to write some code to facilitate the generation of storage models and mapping models for each database engine, but I have that code right here on the shelf, at least for SQL Server and Obstacle. I think that MySql would be capable of using the same storage models and mapping files as SQL Server... but if they don't, that's easy to solve.

I cannot see how anyone would have to worry about running MySql. If you only use it for Cumulus, the system load would be extremely small.

The main rationale for dumping the flat file system storage model is that it makes a huge difference in terms of software maintenance. Plus: using a database for storage is not going to make the software less stable, or slower. Less code to maintain means less moving parts, and thus more stability.

Last but not least:
If you give us something that is as good as Cumulus, and you are providing it as donationware, I think that justifies the fact that you, as a software architect, decide how you architect it.

Re: Future data formats

Posted: Sat 05 Jan 2013 10:53 pm
by Karv
What peterh said!

MySQL does for flat files what the petrol engine did for the horse and cart.

Re: Future data formats

Posted: Sun 06 Jan 2013 10:50 am
by steve
The last time I looked, the MySQL client was very expensive - in fact it's quite difficult/impossible to get an actual price from Oracle without setting up an account. Is there an alternative that isn't from Oracle that's free for closed source applications?

Note that if I do ever produce a 'new' version of Cumulus, I intend it to be cross-platform. And I will have to re-use a lot of my existing code, because I simply can't start from scratch again, so it's almost certain that the data storage will be as in my previous post. And the SQL option will almost certainly be SQLite. With possibly an option to upload to a remote MySQL server, if a free client is available.

Re: Future data formats

Posted: Sun 06 Jan 2013 10:56 am
by peterh
I beg thee pardon?

From the MySql website:
"MySQL Community Edition is the freely downloadable version of the world's most popular open source database."

http://www.mysql.com/products/community/

Re: Future data formats

Posted: Sun 06 Jan 2013 11:02 am
by steve
And the sentence which follows the one you quoted?

Re: Future data formats

Posted: Sun 06 Jan 2013 11:04 am
by peterh
Looking at MySql versus SQLite, it looks like SQLite might be a better option if you're going to package a db engine.

The only issue with SQLite is that it doesn't support "database creation" for an Entity Framewodk CodeFirst approach, but that seems like a minor issue.

Personally, I would prefer an approach that allows me to either install SQLite, or use an existing SQL or MySql database engine. With EF, that's a piece of cake. Plus, EF makes coding against a database a lot easier.

Re: Future data formats

Posted: Sun 06 Jan 2013 11:06 am
by peterh
steve wrote:And the sentence which follows the one you quoted?
You mean this one: "It is available under the GPL license and is supported by a huge and active community of open source developers."?

That, imho, is even better news. What do you think is wrong with it"?

Re: Future data formats

Posted: Sun 06 Jan 2013 11:08 am
by peterh
Some info from StackOverflow:
Stackoverflow wrote:SQLite:

easier to setup
great for temporary (testing databases)
great for rapid development
great for embedding in an application
doesn't have user management
doesn't have many performance features
doesn't scale well.

MySQL:
far more difficult/complex to set up
better options for performance tuning
can scale well if tuned properly
can manage users, permissions, etc.
http://stackoverflow.com/questions/3630/sqlite-vs-mysql

Re: Future data formats

Posted: Sun 06 Jan 2013 11:10 am
by steve
Cumulus isn't GPL, and it's unlikely that a future version would be either - sorry.

SQLite is a great database for this purpose. It's what Cumulus 2 uses/used.

Re: Future data formats

Posted: Sun 06 Jan 2013 11:32 am
by peterh
Aargh. I keep confusing GPL and LGPL... :-(

In that case, SQLite is the better option. But we had already learnt that this is the case anyway.

Re: Future data formats

Posted: Sun 06 Jan 2013 10:33 pm
by BCJKiwi
I'm no expert in this so perhaps I should not be posting, however,
Since the development of Cumulus would not in fact be altering in any way the actual mySQL code, and mySQL would not actually need to be distributed with Cumulus then I don't see the problem.

Steve, you could build the Cumulus system, distribute Cumulus and the end user could download and instal mySQL themselves. Then the GPL license would be between the end user and Cumulus would not have modified nor redistributed mySQL.

I have been thinking about this whole issue and am not entirely sure that Cumulus itself needs to have a SQL file store although I do think it would be better if it did.

The real benefit of a proper database comes when data extraction/reporting/manipulation is done. For the weather applications this is typically via a website.
So if the website ideal database schema is planned for, then the local files could mirror that structure but not necessarily be SQL. A schema in which different elements are in different files provides better long term flexibility and ease of ongoing maintenance.

Re: Future data formats

Posted: Sun 06 Jan 2013 10:38 pm
by peterh
This would be true... but unfortunately for the majority of users, a database would need to be bundled in a default install. That, or a default install would use the flat-file system storage mechanism.
I can see my wife downloading and installing mySql (she manages a 150+ server park), but not my son-in-law.

Re: Future data formats

Posted: Mon 07 Jan 2013 12:41 am
by BCJKiwi
peterh wrote:This would be true... but unfortunately for the majority of users, a database would need to be bundled in a default install. That, or a default install would use the flat-file system storage mechanism.
I can see my wife downloading and installing mySql (she manages a 150+ server park), but not my son-in-law.
Does the GPL actually 'forbid' distribution if mySQL itself is unmodified? Surely the schema and table(s) in the Cumulus database would not be considered part of the source covered by the GPL.

Re: Future data formats

Posted: Mon 07 Jan 2013 6:13 am
by peterh
I not the local license specialist, but if I remember correctly this time, the GPL says that if you distribute MySQL with your software, your software should be GPL as well.
LGPL doesn't have this restriction.

Re: Future data formats

Posted: Mon 07 Jan 2013 8:44 am
by steve
In order to update a MySQL database, Cumulus would have to be linked with a MySQL client library, e.g. libmysql.dll as provided by Oracle. This DLL is dual-licensed, so any code using it has to be GPL (or FOSS) or have a commercial licence.