New with cumulusMX and question how to add historie data

[Ruby the frenchy]

Hi all
first of all I want to say hello to the forum

I own a Frogger Wh2650 since begin of last year
This Weather station send the data to ecowitt and my personal (running on a synology) Webserver
This personal website create for each day a separat csv. the PHP I use is from Raffaello Di Martino http://www.kwos.it
The CSV contain all the data send from the wh2650
How can I import this data into cumulusmx
I will attach a sample csv, maybe somebody can help me

btw. cumulusmx runs with the latest version on a Pi and works like a charme

sample.csv

[Gyvate]

First of all you could use the CMX backfill option.
For the past year your data will still be available in the Ecowitt cloud and CMX can download them.
There are however a few restrictions:
Ecowitt keeps your data posted by your WH2650 into their cloud for one year only.
The past 2 month will be in 5 minute intervals (i.e. one record every 5 minutes), Older than 2 months up to 1/2 year it will be 30 minute intervals and 1/2 year until 1 year it will be 6 hour intervals. CMX can of course only download what is available.

In order to use that in general, you will have to create an API key and an APP key in your ecowitt.net profile, and you will need the MAC address of your WH2650 (either from the device page in your ecowitt account or from the device list in WS View (Plus).
Then you have to activate the backfill option in CMX using these keys/MAC (Settings --> Station settings --> Ecowitt Data Access API)

What I suggest is that you download everything from scratch and later on fill gaps (or replace records) from the data you have collected so far.
For this you have to shut down/stop CMX and make a copy of the content of your .\CumulusMX\data directory (a) for backup and b) for filling gaps if necessary). Then you delete the whole directory content except for the file today.ini
This file you have to edit - in the first and second row/line you replace the actual date minus one year
(this work you can do with a copy of today.ini while CMX is still running - I would first edit the today.ini copy, then stop CMX, then copy the data directory, then delete the content of the data directory and copy/save the edited today.ini inside it. Then restart CMX (Ecowitt Data Access API settings set and saved before stopping CMX !!!).

#Last updated: 01.05.2022 20:59:00
[General]
Date=01.05.2022
Timestamp=2022-05-01T20:59:00
CurrentYear=2022
CurrentMonth=5
CurrentDay=1

becomes (if it were today that you started)

#Last updated: 01.05.2022 20:59:00
[General]
Date=30.04.2021
Timestamp=2021-04-30T00:00:00
CurrentYear=2021
CurrentMonth=4
CurrentDay=30

all other entries in today.ini you put to 0.00, 00:00 (time), and 2021-04-30 00:00 (sometimes 2021-04-30T00:00) date/time etc.
everything needs to be zeroed respectively the date set to 30.04.2021
Then you save this file into the .\CumulusMX\data directory (if you are running CMX on Linux, the path reads ../CumulusMX/data)

Then restart CMX and let it work.
It should download all available data from you from the Ecowitt cloud and create all necessary log files inside the data directory.
Once finished (which might take some time) just let it run.
If you have minute based data in your log files, you can replace all the MmmYYlog.txt files (Mmm=Jan, Feb, etc..., YY = 21, 22) including April 2022 by those which you have saved before. On the May files you can work with the editor inside the CMX dashboard (Menu item "Data logs", "monthly logs".
This would be the first step.

If you want to fill the half hour gap or even 6 hour gaps with your data saved on your personal web site (Meteotemplate ?),
then you can build up the monthly CMX log files older than two months from there.

The file monthlyfileheader.txt in the .\CumulusMX directory describes the structure of the CMX monthly log file, the file dayfileheader.txt the structure of dayfile.txt.

You can manually (or by some program) create a CSV file with the proper structure (convert your web site export into the CMX format) and copy the content inside the MmmYYlog.txt files
These files are already CSV files (the ending .txt does not say that the content is not CSV). Make sure that you are using the proper date and time format and the proper delimiters. (date is usually either DD/MM/YY or DD.MM.YY and the delimiters are "," or ";" depending on your locale. The locale also defines the decimal point, if it's a "." or a ",".
Just have a look into the existing CMX log files.
As CMX usually doesn't touch the older files any more, you can easily work on them while CMX is running (I would still make a safety backup copy before starting to work on them ).

[Ruby the frenchy]

Hi
sorry for my late responce
thanks for all the information.
I've tried this yesterday but noting happens.
I will setup a new Pi with CumulusMX and will see if it then works

BR Uwe

[Gyvate]

please activate the debug option (Program Settings --> Logging options ---> tag Debug logging
if "nothing" happens inspite of the Ecowitt Data Access API" option filled (and saved !) and the today.ini file set on today - 1 year as described earlier,
post the logging file from ..\CumulusMX\MXdiags (format YYYYMMDD-hhmmss - e.g. 20220504-100634.txt) The file name is the startup time of CMX (or the wrap over day/time if the debug file becomes bigger than 20 MB).

[Ruby the frenchy]

It works
What I have done different to the last time.
I stopped cumulusmx with "sudo systemctl stop cumulusmx"
Last time I let the Pi reboot

And the debug tells me that cumulusmx fetch all the data from ecowitt.

Great and thanks for your patience

BR Uwe

Update:
I see that the rain data are not in. But I have read the latest release info. I will repeat the import