Converting monthly log files to an sqlite database
Posted: Mon 06 Feb 2012 4:50 pm
Hi, for a project I'm working on I developed a simple Python module, that given the directory path to where your monthly log files are kept will write an sqlite3 database from the logs in the same directory.
It requires Python to be installed ( I'm running 2.7.2 ) and you can change the 'D:\data' to whatever your monthly log file path is, it creates a database file called 'weather_database' with a 'weather_log' table and parses Cumulus' date fields into a utc unix timestamp field called time_seconds. You can rerun the script after more data has been added to the log files and it will write the new records to the same database.
You can download the module at the link below.
https://bitbucket.org/jamesmorrison/cum ... er_data.py
Cheers
It requires Python to be installed ( I'm running 2.7.2 ) and you can change the 'D:\data' to whatever your monthly log file path is, it creates a database file called 'weather_database' with a 'weather_log' table and parses Cumulus' date fields into a utc unix timestamp field called time_seconds. You can rerun the script after more data has been added to the log files and it will write the new records to the same database.
You can download the module at the link below.
https://bitbucket.org/jamesmorrison/cum ... er_data.py
Cheers