Review large amounts of data

[Warrandyte]

Folks, I need to access the last 2 years worth of data at hourly intervals.

Anyone done this before?

Whats the best way to approach it - given excels mlimitations, and the lack of files available.

is there a way to amend the logging in cumulus to support this analysis?

[steve]

is there a way to amend the logging in cumulus to support this analysis?

No, sorry.

[mcrossley]

I'd import the monthly log files into a SQL database, then pretty much any analysis you want is available to you. There is a "Cumulus File Import" PHP script in the Wiki to create the tables and import the data into MySQL - it could be adapted to pretty much any SQL database though.

[yv1hx]

Folks, I need to access the last 2 years worth of data at hourly intervals.

Anyone done this before?

Whats the best way to approach it - given excels mlimitations, and the lack of files available.

is there a way to amend the logging in cumulus to support this analysis?

Warrandyte:

I learn the same lesson than yours by the hard way: My data files spans about 5 years, most of them with a 1 minute resolution, so it was almost impossible to process in a efficient manner with MS Excel.

So, in the recent past I have been working with PHP scripts for extracting & processing my data and plotting with gnuplot using my Raspberry Pi linux box.

Note that i'm not a professional programmer, I choose PHP & gnuplot because was easy to learn and have plenty of documentation and examples over the net.

[mcrossley]

What limits are you hitting in Excel? Two years is roughly 17,000 hours, and Excel will handle up to 1 million rows.