Folks, I need to access the last 2 years worth of data at hourly intervals.
Anyone done this before?
Whats the best way to approach it - given excels mlimitations, and the lack of files available.
is there a way to amend the logging in cumulus to support this analysis?
Welcome to the Cumulus Support forum.
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4019) - 03 April 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4019) - 03 April 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Review large amounts of data
-
- Posts: 41
- Joined: Sat 06 Apr 2013 11:46 pm
- Weather Station: Fine Offset with Solar
- Operating System: Windows XP, 7 8 and 10
- Location: TIrrannaville NSW Australia
- steve
- Cumulus Author
- Posts: 26701
- Joined: Mon 02 Jun 2008 6:49 pm
- Weather Station: None
- Operating System: None
- Location: Vienne, France
- Contact:
Re: Review large amounts of data
No, sorry.Warrandyte wrote:is there a way to amend the logging in cumulus to support this analysis?
Steve
- mcrossley
- Posts: 12756
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Review large amounts of data
I'd import the monthly log files into a SQL database, then pretty much any analysis you want is available to you. There is a "Cumulus File Import" PHP script in the Wiki to create the tables and import the data into MySQL - it could be adapted to pretty much any SQL database though.
- yv1hx
- Posts: 223
- Joined: Mon 05 Apr 2010 10:40 pm
- Weather Station: No station yet ...
- Operating System: Win XP Professional
- Location: Some point in the Earth
Re: Review large amounts of data
Warrandyte:Warrandyte wrote:Folks, I need to access the last 2 years worth of data at hourly intervals.
Anyone done this before?
Whats the best way to approach it - given excels mlimitations, and the lack of files available.
is there a way to amend the logging in cumulus to support this analysis?
I learn the same lesson than yours by the hard way: My data files spans about 5 years, most of them with a 1 minute resolution, so it was almost impossible to process in a efficient manner with MS Excel.
So, in the recent past I have been working with PHP scripts for extracting & processing my data and plotting with gnuplot using my Raspberry Pi linux box.
Note that i'm not a professional programmer, I choose PHP & gnuplot because was easy to learn and have plenty of documentation and examples over the net.
Marco
- mcrossley
- Posts: 12756
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Review large amounts of data
What limits are you hitting in Excel? Two years is roughly 17,000 hours, and Excel will handle up to 1 million rows.