CumulusMX Process Date [SOLVED]

[Cambium]

How do I get CumulusMX to process from a specific date?

I'm trying to have it pick up some Ecowitt data from Dec 24 to Jan 4 without me inputting the data into folders.

I deleted the Jan2024 log file, the dayfile lines after Dec 23rd, some other files, and the today.ini folder, but its still processing from last time I opened it. ?

Thank You

[Mapantz]

cumulus.ini

[Gyvate]

you have to edit and save today.ini after (!) shutting down CMX
like below
#Last updated: 23.12.2023 23:59:00
[General]
Date=24.12.2023
Timestamp=2023-12-23T23:59:00
CurrentYear=2024
CurrentMonth=1
CurrentDay=30

then restart CMX - it will now load whatever 5-minute data is available from 24-Dec-2023 00:00 until the restart and save them into Dec23log.txt and Jan24log.txt and will then continue reading from your console. It will also update dayfile.txt (which you may have to check for double entries and remove them after the download is completed)

[water01]

cumulus.ini

Just in case Mapantz's reply is not clear change the Start Date in cumulus.ini to the date you want it start. Make sure you use the same format of date.

Can either be done by editing Cumulus.ini with CumulusMX closed or by editing the Records Began Date in Station Settings > General Settings > Advanced Options.

If you Do it via the Dashboard you have to restart CumulusMX.

[Gyvate]

I don't get the point here with Cumulus.ini
the OP wants to download data from the Ecowitt cloud from date/time to date/time - the backfill option of CMX determines the start for downloading from the time stamp in today.ini and doesn't care about Cumulus.ini
I never did anything to Cumulus.ini when doing backfilling when CMX already had continued after a break without filling the gap.
Just edit today.ini.
Then do some housekeeping with what was already there and with dayfile.txt (if applicable, i.e. if midnight was crossed)

e.g. CMX didn't record any records from 23-Dec-2023 23:59 and with the next restart continued at the 4-Jan-2024 without backfilling for whatever reason.
0. shut down CMX
1. set today.ini as described before
2. copy the content of Jan24log.txt until today to a separate txt file and delete it from Jan24log.txt (if there is an ExtraLog202312.txt/ExtraLog202401.txt do accordingly)
3. start CMX with the today.ini settings for 23-Dec-2023 23:59:00
4. once ready edit Jan24log.txt between two archivings at the full minute: copy what was saved before and replace the 5-minute records CMX downloaded from 04-Jan-2024 on by the data already archived earlier and saved. Make sure you save the file before the minute is over.
Make some minor corrections if necessary - make sure the time stamp sequence is correct and no double entries are there ...
Then check if dayfile.txt has double entries. If so, decide which ones you want to keep, delete the others (this you can do with a proper editor like Notepad, Notepad++ or from within CMX as in the example only 26 entries may be concerned). Or with Nano if you are Linux based.

Doing all the single minute based records of the monthly log files from within CMX would be a task which would take a very long time .... that feature is only helpful when only a few records need to be added, deleted or modified (if the archiving is every minute we are talking about 1,440 records per day - 4th thru 30th of January: 26 days times 1,440 = 37,440 - good luck ...

[Cambium]

you have to edit and save today.ini after (!) shutting down CMX
like below
#Last updated: 23.12.2023 23:59:00
[General]
Date=24.12.2023
Timestamp=2023-12-23T23:59:00
CurrentYear=2024
CurrentMonth=1
CurrentDay=30

then restart CMX - it will now load whatever 5-minute data is available from 24-Dec-2023 00:00 until the restart and save them into Dec23log.txt and Jan24log.txt and will then continue reading from your console. It will also update dayfile.txt (which you may have to check for double entries and remove them after the download is completed)

Excellent! Thank You. I decided to delete all files and just drag the backup I had from July 2023. Now I'm waiting for everything since then. Haha. Thanks for the ini tip!! Thats what I needed to know. Good reminder for next time.

[Mapantz]

I don't get the point here with Cumulus.ini

Because, that's what I did when I had some issues.

I had data missing due to swapping over the logging device. I stopped CMX, changed the date and time in the cumulus.ini, started CMX, and it downloaded the missing data.

[Gyvate]

Excellent! Thank You. I decided to delete all files and just drag the backup I had from July 2023. Now I'm waiting for everything since then. Haha. Thanks for the ini tip!! Thats what I needed to know. Good reminder for next time.

the only thing is - the retention time at ecowitt.net
5-minute data is only kept there for 90 days, farther back it will be a 30 minute interval for up to 360 days back...
see https://www.wxforum.net/index.php?topic=40730.0 chapter 14

[Cambium]

Excellent! Thank You. I decided to delete all files and just drag the backup I had from July 2023. Now I'm waiting for everything since then. Haha. Thanks for the ini tip!! Thats what I needed to know. Good reminder for next time.

the only thing is - the retention time at ecowitt.net
5-minute data is only kept there for 90 days, farther back it will be a 30 minute interval for up to 360 days back...
see https://www.wxforum.net/index.php?topic=40730.0 chapter 14

Ohhhhh great catch and reminder! Crap. Debating whether to just use my newer backup and swap out the July,Aug,Sept,Oct,Nov monthly logs OR just do it over again using your suggestion with editing the ini

[Cambium]

Well that backfired. Everything had double the data.

I'm not sure now how to use the July-Nov Monthly log folders I had with the 5 minute updates to pick up on records and dayfile.

Let me try the Create Missing. Problem is it starts from 2011 each time

[Gyvate]

@Cambium
while running, CMX only updates only the current month logs and the .ini files
=> you can be at ease editing the older MMMYYlog.txt files with an editor, make sure they are in the proper sequence, no double entries etc.

dayfile.txt will only be updated at midnight (respectively at the end of the meteorological day if it is different from midnight for you).

So together with backup copies (which you hopefully have) you should be able to rebuild your data base while CMX collects current data

before starting to use CreateMissing, make sure you have read the WiKi about it:
https://www.cumuluswiki.org/a/Calculate_Missing_Values
and
https://www.cumuluswiki.org/a/Amending_dayfile
before you are messing up more than necessary

[Cambium]

@Cambium
while running, CMX only updates only the current month logs and the .ini files
=> you can be at ease editing the older MMMYYlog.txt files with an editor, make sure they are in the proper sequence, no double entries etc.

dayfile.txt will only be updated at midnight (respectively at the end of the meteorological day if it is different from midnight for y

So together with backup copies (which you hopefully have) you should be able to rebuild your data base while CMX collects current data

Interesting. So are you aaying that I can change October 2023 log file numbers and when CMX runs the next day it will update the day file for the days in that month?

[Gyvate]

yes and no -
you can change it while it is running - but if the day file needs changes or even an entry, you have to do that manually, CMX won't go back.
It only creates entries for the past day at midnight or (same situation) when backfilling from ecowitt.net and crossing midnight
the header description of the day file (and other files) is in the ...\CumulusMX directory,
but for a few entries you better change it from within CMX (Data Logs --> Dayfile) as you have a proper mask which tells you what is what

[Cambium]

yes and no -
you can change it while it is running - but if the day file needs changes or even an entry, you have to do that manually, CMX won't go back.
It only creates entries for the past day at midnight or (same situation) when backfilling from ecowitt.net and crossing midnight
the header description of the day file (and other files) is in the ...\CumulusMX directory,
but for a few entries you better change it from within CMX (Data Logs --> Dayfile) as you have a proper mask which tells you what is what

Thanks again but it was 2am when I had an AH HA moment. I just pasted the dayfile backup I had for July 2023-November 2023. It has all the 5 minute reads from the logs I have for those months..

All fixed. Thanks all!