Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4018) - 28 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Importing historic data

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
User avatar
whitling2k
Posts: 31
Joined: Thu 22 Jul 2021 8:03 pm
Weather Station: Ecowitt GW1103
Operating System: Raspberry PiOS
Contact:

Importing historic data

Post by whitling2k »

I have just started using CumulusMX to to archive and display my weather obs. In the past I have used Easyweather exports, imported to Excel with some custom software I wrote to clean, and aggregate to daily, weekly and monthly stats.

I have about 7 years worth of 15 minute data I would like to import into CumulusMX, so it's all available from one place.

Can I ask:
1. Is it wise to run MX with 7+ years and increasing's worth of data?
2. Is it technically possible to import all this old data?
3. If so - is it a good idea to do so?

I think the process is broadly (with a non-operational copy of the CumulusMX folder....)
1. split my historic data into csv files for each month, following the mmmyylog.txt naming convention. Is there a list of month names anywhere?
2. copy them into the \data\ folder
3. Somehow update the max/min text files?
This is where my knowledge gets a bit ropey.

Does the createmissingdata create the max/mins in the max/min files?

I guess I can create the max/mins when processing the data into monthly csv files - where abouts are max/mins data stored?
User avatar
PaulMy
Posts: 3777
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2 Plus 24-Hour FARS
Operating System: Windows8 and Windows10
Location: Komoka, ON Canada
Contact:

Re: Importing historic data

Post by PaulMy »

Hi,
I will give it a try, but some more technical minded users may be able to add, or perhaps correct me.
Can I ask:
1. Is it wise to run MX with 7+ years and increasing's worth of data?
Yes, that is one of the purposes. I have data in Cumulus starting in October 2008, and other users even before then http://www.komokaweather.com/cumulusmx/ ... hp?lang=en
2. Is it technically possible to import all this old data?
Yes, there are some Cumulus utilities to do that with certain types of other weather program files https://cumuluswiki.org/a/Software#Other_utilities
3. If so - is it a good idea to do so?
Yes, historical data is interesting and also helpful in comparing and looking at current weather conditions http://www.komokaweather.com/cumulusmx/noaareports.php
I think the process is broadly (with a non-operational copy of the CumulusMX folder....)
1. split my historic data into csv files for each month, following the mmmyylog.txt naming convention. Is there a list of month names anywhere?
In English: monthly log files start with Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec and while I am not sure, believe also similar for other languages. The file must be txt without any formatting

2. copy them into the \data\ folder
After they are converted in the exact Cumulus text format and with .txt extension
3. Somehow update the max/min text files?
There are several .ini data files for max/min - alltime, month, monthlyalltime, today, year, yesterday
and the records can be edited through the dashboard Edit options
This is where my knowledge gets a bit ropey.

Does the createmissingdata create the max/mins in the max/min files?
CreateMissing currently is only to add missing days in dayfile.txt, and can do so only if there is data in the MmmYYlog.txt log file for the missing day(s).
I guess I can create the max/mins when processing the data into monthly csv files - where abouts are max/mins data stored?
All the data files such as monthly log, dayfile, the max/min .ini files are in \data folder.

You haven't asked about MySQL DB and that is for someone else to help if needed.

Enjoy,
Paul
Davis Vantage Pro2+
C1 www.komokaweather.com/komokaweather-ca
MX www.komokaweather.com/cumulusmx/index.htm /index.html /index.php
MX www.komokaweather.com/cumulusmxwll/index.htm /index.html /index.php
MX www. komokaweather.com/cumulusmx4/index.htm

Image
User avatar
whitling2k
Posts: 31
Joined: Thu 22 Jul 2021 8:03 pm
Weather Station: Ecowitt GW1103
Operating System: Raspberry PiOS
Contact:

Re: Importing historic data

Post by whitling2k »

Nice one, Thanks Paul.

I have much reading and experimenting to be getting on with! I have a second Pi, so I think I will copy a snapshot of my MX folder onto that and play/break as I work out what's what.

Thanks also for the pointers to the readme pages. I missed some of those when researching.

I particularly like your annual data summary page - is that part of a third party template, or can I just not see it because I don't have enough data yet?
User avatar
PaulMy
Posts: 3777
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2 Plus 24-Hour FARS
Operating System: Windows8 and Windows10
Location: Komoka, ON Canada
Contact:

Re: Importing historic data

Post by PaulMy »

You're welcome... the Wiki is excellent with lots of information but can be overwhelming if just beginning or just new to Cumulus.

The Annual Data is a third-party script started by forum member and expanded by others over time, and evolved in many variations. You can begin to use it even with just one day's dayfile data. The beteljuice is the author for the script included with BCJKiwi's third party template MXUI Website viewtopic.php?f=18&t=17845
It is one of my favourites and allows me to easily pull of stats for a Month's Review and historical comparison on Facebook page.

Enjoy,
Paul
Davis Vantage Pro2+
C1 www.komokaweather.com/komokaweather-ca
MX www.komokaweather.com/cumulusmx/index.htm /index.html /index.php
MX www.komokaweather.com/cumulusmxwll/index.htm /index.html /index.php
MX www. komokaweather.com/cumulusmx4/index.htm

Image
User avatar
whitling2k
Posts: 31
Joined: Thu 22 Jul 2021 8:03 pm
Weather Station: Ecowitt GW1103
Operating System: Raspberry PiOS
Contact:

Re: Importing historic data

Post by whitling2k »

I'm processing out my historic data files and have a query I can't see in the documentation - can anyone advise?

For the dayfile.txt and monthly log files, are missing rows best left in the data as blank values, or left out all together?

I have some large gaps (days-months) and short gaps (single rows to hours), so there will be gaps in both the day files and monthly logs.


i.e. (simplified of course)

Code: Select all

date,temp,humidity,etc.....
1/1 00:15, 15, 67, ...
1/1 00:30, 16, 47, ...
1/1 00:45, 15, 67, ...
1/1 01:00, , , ...         [no values here...]
1/1 01:15, , , ...           |
1/1 01:30, , , ...         [...to here]
1/1 01:45, 15, 67, ...
1/1 02:00, 16, 47, ...

or

Code: Select all

date,temp,humidity,etc.....
1/1 00:15, 15, 67, ...
1/1 00:30, 16, 47, ...
1/1 00:45, 15, 67, ...  [3 rows missing]
1/1 01:45, 15, 67, ...
1/1 02:00, 16, 47, ...

Phil23
Posts: 884
Joined: Sat 16 Jul 2016 11:59 pm
Weather Station: Davis VP2+ & GW1000 (Standalone)
Operating System: Win10 Pro / rPi Buster
Location: Australia

Re: Importing historic data

Post by Phil23 »

whitling2k wrote: Tue 03 Aug 2021 3:44 pm
I have some large gaps (days-months) and short gaps (single rows to hours), so there will be gaps in both the day files and monthly logs.
I had that issue with my old Fine Offset data years back.

I basically just invented some data based on reports from the BOM site that's 4km away by air.
Temp & humidity were pretty simple as it wasn't hard to create an equation to relate their data to mine.
:Now: :Today/Yesterday:

Image

Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
User avatar
whitling2k
Posts: 31
Joined: Thu 22 Jul 2021 8:03 pm
Weather Station: Ecowitt GW1103
Operating System: Raspberry PiOS
Contact:

Re: Importing historic data

Post by whitling2k »

OK - does that mean MX can't handle blanks at all?

I can easily interpolate across some of the shorter gaps, and like you say - I can find a local PWS and calibrate that against my obs to fill in those blanks.

As long as it's necessary, I don't want to waste my time if it can handle blanks.

Out of interest - how does MX respond to - for example - sensor failures? Does it all come crashing down if it looses the link to the sensors?
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Importing historic data

Post by mcrossley »

Cumulus cannot handle completely blank entries. Just leave those lines out.

As Paul said, the CreateMissing utility will create your dayfile for you from the monthly files once you get them sorted.

If a sensor goes offline it depends on the setting of "Sensor contact lost". If that is enabled, then Cumulus will stop logging if any of the primary sensors (T, H, R, W) are missing. If not enabled then logging will continue but missing values will flat-line at their last value.
User avatar
whitling2k
Posts: 31
Joined: Thu 22 Jul 2021 8:03 pm
Weather Station: Ecowitt GW1103
Operating System: Raspberry PiOS
Contact:

Re: Importing historic data

Post by whitling2k »

Wow, OK. Just an update in way of saying thanks to those responding so far. I do listen to and follow advice!

I have finally managed to construct all my monthly log files. Which has been more challenging than it could have been due to getting COVID, being busy with life and work, and also starting logging way back in 2014 without a plan or end-point in-mind. But I got there and have just made my first full test import into CumulusMX. It has been an interesting process, and my data is better for it. Numerous erroneous spikes flattened, 1000s of tiny gaps interpolated across, and the storage format and units all made consistent, not to mention a better understanding of how MX works.

Number of records processed: 2538
Added : 2443
Updated: 0
No Data: 86 - please check the log file for the errors
Were OK: 9

Very happy with that! The 86 missing rows were expected - I knew about them. I also I coded a piece of vba software to write the 83 log files from Excel for me. It's a bit hacky, but I can share the key parts if anyone else is using EXCEL to clean and prepare historic data for import.

This is still in my test environment, so the next step is to merge the constructed dayfile lines etc and logfiles into my production environment, and update all the max/mins - without losing any data in the process, which should be straight forwards if my Pi Zero can handle the strain of all those i/o.

Then have a go at installing the templates Paul mentioned.

Finally devise a tidy way to backup my logfiles and dayfile to the cloud. I currently snapshot the entire MX folder to my NAS once a week just in case. I am thinking about a more precise rsync to just backup the important data and config.
All Data Chart.jpg
You do not have the required permissions to view the files attached to this post.
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Importing historic data

Post by mcrossley »

whitling2k wrote: Thu 21 Oct 2021 7:55 pm
This is still in my test environment, so the next step is to merge the constructed dayfile lines etc and logfiles into my production environment, and update all the max/mins - without losing any data in the process, which should be straight forwards if my Pi Zero can handle the strain of all those i/o.
Good progress, you could always make a local install of CMX on your PC to do the data merge, then upload the resulting files to the pi once you are happy?
User avatar
whitling2k
Posts: 31
Joined: Thu 22 Jul 2021 8:03 pm
Weather Station: Ecowitt GW1103
Operating System: Raspberry PiOS
Contact:

Re: Importing historic data

Post by whitling2k »

I have a spare Pi, so was able to execute and test it all on there, then copy over the appropriate files to the 'live' Pi.

It all seems to have worked well apart from a rainfall spike where I made my current sensor's lifetime rainfall accumulation match with my historic, it interpreted it as a dump of 4005mm rain in 15 minutes! Easy enough to smooth out of the day and log files as it was otherwise a completely dry day.

Which did raise a question about rainfall counter - should it reset annually, or continuously count from the very first T=0

MX seems to refer to the sensor's inbuilt counts for some rainfall readings and instantaneous read-outs with calculations for others.

All is well now though, just waiting for the spike to drop out of the recent graphs.
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Importing historic data

Post by mcrossley »

The rain counter is provided by the station. For Davis kit CMX uses the annual rainfall total, so that does reset every year at the start of your rainfall season.
FO/Ecowitt stations just have a running counter that resets when the sensor is power cycled.

The rain counter is for internal use only, it isn't exposed to the users directly.
Post Reply