Welcome to the Cumulus Support forum.
Latest Cumulus MX V4 release 4.2.1 (build 4043) - 19 October 2024
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V4 release 4.2.1 (build 4043) - 19 October 2024
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
CumulusMX Process Date [SOLVED]
Moderator: mcrossley
-
- Posts: 274
- Joined: Wed 03 Aug 2011 11:19 pm
- Weather Station: ProWeatherTP3000
- Operating System: Windows10
- Location: Coastal CT, USA
- Contact:
CumulusMX Process Date [SOLVED]
How do I get CumulusMX to process from a specific date?
I'm trying to have it pick up some Ecowitt data from Dec 24 to Jan 4 without me inputting the data into folders.
I deleted the Jan2024 log file, the dayfile lines after Dec 23rd, some other files, and the today.ini folder, but its still processing from last time I opened it. ?
Thank You
I'm trying to have it pick up some Ecowitt data from Dec 24 to Jan 4 without me inputting the data into folders.
I deleted the Jan2024 log file, the dayfile lines after Dec 23rd, some other files, and the today.ini folder, but its still processing from last time I opened it. ?
Thank You
Last edited by Cambium on Wed 31 Jan 2024 12:14 pm, edited 2 times in total.
- Gyvate
- Posts: 298
- Joined: Wed 16 Dec 2020 2:14 pm
- Weather Station: GW1x00/WH2650WiFi/HP2553/GW2000
- Operating System: Win 11 (PC/RPi), Raspbian 11,WSL
- Location: Saarbrücken, Germany
- Contact:
Re: CumulusMX Process Date
you have to edit and save today.ini after (!) shutting down CMX
like below
#Last updated: 23.12.2023 23:59:00
[General]
Date=24.12.2023
Timestamp=2023-12-23T23:59:00
CurrentYear=2024
CurrentMonth=1
CurrentDay=30
then restart CMX - it will now load whatever 5-minute data is available from 24-Dec-2023 00:00 until the restart and save them into Dec23log.txt and Jan24log.txt and will then continue reading from your console. It will also update dayfile.txt (which you may have to check for double entries and remove them after the download is completed)
like below
#Last updated: 23.12.2023 23:59:00
[General]
Date=24.12.2023
Timestamp=2023-12-23T23:59:00
CurrentYear=2024
CurrentMonth=1
CurrentDay=30
then restart CMX - it will now load whatever 5-minute data is available from 24-Dec-2023 00:00 until the restart and save them into Dec23log.txt and Jan24log.txt and will then continue reading from your console. It will also update dayfile.txt (which you may have to check for double entries and remove them after the download is completed)
Weather Landing Page: http://meshka.eu
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
-
- Posts: 3487
- Joined: Sat 13 Aug 2011 9:33 am
- Weather Station: Ecowitt HP2551
- Operating System: Windows 10 64bit
- Location: Burnham-on-Sea
- Contact:
Re: CumulusMX Process Date
Just in case Mapantz's reply is not clear change the Start Date in cumulus.ini to the date you want it start. Make sure you use the same format of date.
Can either be done by editing Cumulus.ini with CumulusMX closed or by editing the Records Began Date in Station Settings > General Settings > Advanced Options.
If you Do it via the Dashboard you have to restart CumulusMX.
- Gyvate
- Posts: 298
- Joined: Wed 16 Dec 2020 2:14 pm
- Weather Station: GW1x00/WH2650WiFi/HP2553/GW2000
- Operating System: Win 11 (PC/RPi), Raspbian 11,WSL
- Location: Saarbrücken, Germany
- Contact:
Re: CumulusMX Process Date
I don't get the point here with Cumulus.ini
the OP wants to download data from the Ecowitt cloud from date/time to date/time - the backfill option of CMX determines the start for downloading from the time stamp in today.ini and doesn't care about Cumulus.ini
I never did anything to Cumulus.ini when doing backfilling when CMX already had continued after a break without filling the gap.
Just edit today.ini.
Then do some housekeeping with what was already there and with dayfile.txt (if applicable, i.e. if midnight was crossed)
e.g. CMX didn't record any records from 23-Dec-2023 23:59 and with the next restart continued at the 4-Jan-2024 without backfilling for whatever reason.
0. shut down CMX
1. set today.ini as described before
2. copy the content of Jan24log.txt until today to a separate txt file and delete it from Jan24log.txt (if there is an ExtraLog202312.txt/ExtraLog202401.txt do accordingly)
3. start CMX with the today.ini settings for 23-Dec-2023 23:59:00
4. once ready edit Jan24log.txt between two archivings at the full minute: copy what was saved before and replace the 5-minute records CMX downloaded from 04-Jan-2024 on by the data already archived earlier and saved. Make sure you save the file before the minute is over.
Make some minor corrections if necessary - make sure the time stamp sequence is correct and no double entries are there ...
Then check if dayfile.txt has double entries. If so, decide which ones you want to keep, delete the others (this you can do with a proper editor like Notepad, Notepad++ or from within CMX as in the example only 26 entries may be concerned). Or with Nano if you are Linux based.
Doing all the single minute based records of the monthly log files from within CMX would be a task which would take a very long time .... that feature is only helpful when only a few records need to be added, deleted or modified (if the archiving is every minute we are talking about 1,440 records per day - 4th thru 30th of January: 26 days times 1,440 = 37,440 - good luck ...
the OP wants to download data from the Ecowitt cloud from date/time to date/time - the backfill option of CMX determines the start for downloading from the time stamp in today.ini and doesn't care about Cumulus.ini
I never did anything to Cumulus.ini when doing backfilling when CMX already had continued after a break without filling the gap.
Just edit today.ini.
Then do some housekeeping with what was already there and with dayfile.txt (if applicable, i.e. if midnight was crossed)
e.g. CMX didn't record any records from 23-Dec-2023 23:59 and with the next restart continued at the 4-Jan-2024 without backfilling for whatever reason.
0. shut down CMX
1. set today.ini as described before
2. copy the content of Jan24log.txt until today to a separate txt file and delete it from Jan24log.txt (if there is an ExtraLog202312.txt/ExtraLog202401.txt do accordingly)
3. start CMX with the today.ini settings for 23-Dec-2023 23:59:00
4. once ready edit Jan24log.txt between two archivings at the full minute: copy what was saved before and replace the 5-minute records CMX downloaded from 04-Jan-2024 on by the data already archived earlier and saved. Make sure you save the file before the minute is over.
Make some minor corrections if necessary - make sure the time stamp sequence is correct and no double entries are there ...
Then check if dayfile.txt has double entries. If so, decide which ones you want to keep, delete the others (this you can do with a proper editor like Notepad, Notepad++ or from within CMX as in the example only 26 entries may be concerned). Or with Nano if you are Linux based.
Doing all the single minute based records of the monthly log files from within CMX would be a task which would take a very long time .... that feature is only helpful when only a few records need to be added, deleted or modified (if the archiving is every minute we are talking about 1,440 records per day - 4th thru 30th of January: 26 days times 1,440 = 37,440 - good luck ...
Weather Landing Page: http://meshka.eu
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
-
- Posts: 274
- Joined: Wed 03 Aug 2011 11:19 pm
- Weather Station: ProWeatherTP3000
- Operating System: Windows10
- Location: Coastal CT, USA
- Contact:
Re: CumulusMX Process Date
Excellent! Thank You. I decided to delete all files and just drag the backup I had from July 2023. Now I'm waiting for everything since then. Haha. Thanks for the ini tip!! Thats what I needed to know. Good reminder for next time.Gyvate wrote: ↑Tue 30 Jan 2024 6:14 pm you have to edit and save today.ini after (!) shutting down CMX
like below
#Last updated: 23.12.2023 23:59:00
[General]
Date=24.12.2023
Timestamp=2023-12-23T23:59:00
CurrentYear=2024
CurrentMonth=1
CurrentDay=30
then restart CMX - it will now load whatever 5-minute data is available from 24-Dec-2023 00:00 until the restart and save them into Dec23log.txt and Jan24log.txt and will then continue reading from your console. It will also update dayfile.txt (which you may have to check for double entries and remove them after the download is completed)
-
- Posts: 1890
- Joined: Sat 17 Dec 2011 11:55 am
- Weather Station: Davis Vantage Pro2
- Operating System: Windows 11 x64
- Location: Dorset - UK
- Contact:
Re: CumulusMX Process Date
Because, that's what I did when I had some issues.
I had data missing due to swapping over the logging device. I stopped CMX, changed the date and time in the cumulus.ini, started CMX, and it downloaded the missing data.
- Gyvate
- Posts: 298
- Joined: Wed 16 Dec 2020 2:14 pm
- Weather Station: GW1x00/WH2650WiFi/HP2553/GW2000
- Operating System: Win 11 (PC/RPi), Raspbian 11,WSL
- Location: Saarbrücken, Germany
- Contact:
Re: CumulusMX Process Date
the only thing is - the retention time at ecowitt.net
5-minute data is only kept there for 90 days, farther back it will be a 30 minute interval for up to 360 days back...
see https://www.wxforum.net/index.php?topic=40730.0 chapter 14
Weather Landing Page: http://meshka.eu
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
-
- Posts: 274
- Joined: Wed 03 Aug 2011 11:19 pm
- Weather Station: ProWeatherTP3000
- Operating System: Windows10
- Location: Coastal CT, USA
- Contact:
Re: CumulusMX Process Date
Ohhhhh great catch and reminder! Crap. Debating whether to just use my newer backup and swap out the July,Aug,Sept,Oct,Nov monthly logs OR just do it over again using your suggestion with editing the iniGyvate wrote: ↑Tue 30 Jan 2024 8:30 pmthe only thing is - the retention time at ecowitt.net
5-minute data is only kept there for 90 days, farther back it will be a 30 minute interval for up to 360 days back...
see https://www.wxforum.net/index.php?topic=40730.0 chapter 14
-
- Posts: 274
- Joined: Wed 03 Aug 2011 11:19 pm
- Weather Station: ProWeatherTP3000
- Operating System: Windows10
- Location: Coastal CT, USA
- Contact:
Re: CumulusMX Process Date
Well that backfired. Everything had double the data.
I'm not sure now how to use the July-Nov Monthly log folders I had with the 5 minute updates to pick up on records and dayfile.
Let me try the Create Missing. Problem is it starts from 2011 each time
I'm not sure now how to use the July-Nov Monthly log folders I had with the 5 minute updates to pick up on records and dayfile.
Let me try the Create Missing. Problem is it starts from 2011 each time
- Gyvate
- Posts: 298
- Joined: Wed 16 Dec 2020 2:14 pm
- Weather Station: GW1x00/WH2650WiFi/HP2553/GW2000
- Operating System: Win 11 (PC/RPi), Raspbian 11,WSL
- Location: Saarbrücken, Germany
- Contact:
Re: CumulusMX Process Date
@Cambium
while running, CMX only updates only the current month logs and the .ini files
=> you can be at ease editing the older MMMYYlog.txt files with an editor, make sure they are in the proper sequence, no double entries etc.
dayfile.txt will only be updated at midnight (respectively at the end of the meteorological day if it is different from midnight for you).
So together with backup copies (which you hopefully have) you should be able to rebuild your data base while CMX collects current data
before starting to use CreateMissing, make sure you have read the WiKi about it:
https://www.cumuluswiki.org/a/Calculate_Missing_Values
and
https://www.cumuluswiki.org/a/Amending_dayfile
before you are messing up more than necessary
while running, CMX only updates only the current month logs and the .ini files
=> you can be at ease editing the older MMMYYlog.txt files with an editor, make sure they are in the proper sequence, no double entries etc.
dayfile.txt will only be updated at midnight (respectively at the end of the meteorological day if it is different from midnight for you).
So together with backup copies (which you hopefully have) you should be able to rebuild your data base while CMX collects current data
before starting to use CreateMissing, make sure you have read the WiKi about it:
https://www.cumuluswiki.org/a/Calculate_Missing_Values
and
https://www.cumuluswiki.org/a/Amending_dayfile
before you are messing up more than necessary
Weather Landing Page: http://meshka.eu
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
-
- Posts: 274
- Joined: Wed 03 Aug 2011 11:19 pm
- Weather Station: ProWeatherTP3000
- Operating System: Windows10
- Location: Coastal CT, USA
- Contact:
Re: CumulusMX Process Date
Interesting. So are you aaying that I can change October 2023 log file numbers and when CMX runs the next day it will update the day file for the days in that month?Gyvate wrote: ↑Wed 31 Jan 2024 7:06 am @Cambium
while running, CMX only updates only the current month logs and the .ini files
=> you can be at ease editing the older MMMYYlog.txt files with an editor, make sure they are in the proper sequence, no double entries etc.
dayfile.txt will only be updated at midnight (respectively at the end of the meteorological day if it is different from midnight for y
So together with backup copies (which you hopefully have) you should be able to rebuild your data base while CMX collects current data
- Gyvate
- Posts: 298
- Joined: Wed 16 Dec 2020 2:14 pm
- Weather Station: GW1x00/WH2650WiFi/HP2553/GW2000
- Operating System: Win 11 (PC/RPi), Raspbian 11,WSL
- Location: Saarbrücken, Germany
- Contact:
Re: CumulusMX Process Date
yes and no -
you can change it while it is running - but if the day file needs changes or even an entry, you have to do that manually, CMX won't go back.
It only creates entries for the past day at midnight or (same situation) when backfilling from ecowitt.net and crossing midnight
the header description of the day file (and other files) is in the ...\CumulusMX directory,
but for a few entries you better change it from within CMX (Data Logs --> Dayfile) as you have a proper mask which tells you what is what
you can change it while it is running - but if the day file needs changes or even an entry, you have to do that manually, CMX won't go back.
It only creates entries for the past day at midnight or (same situation) when backfilling from ecowitt.net and crossing midnight
the header description of the day file (and other files) is in the ...\CumulusMX directory,
but for a few entries you better change it from within CMX (Data Logs --> Dayfile) as you have a proper mask which tells you what is what
Weather Landing Page: http://meshka.eu
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
CumulusMX http://meshka.eu/CumulusMX
CUtils http://meshka.eu/CUtils
Ecowitt WiKi: http://meshka.eu/Ecowitt/dokuwiki
-
- Posts: 274
- Joined: Wed 03 Aug 2011 11:19 pm
- Weather Station: ProWeatherTP3000
- Operating System: Windows10
- Location: Coastal CT, USA
- Contact:
Re: CumulusMX Process Date
Thanks again but it was 2am when I had an AH HA moment. I just pasted the dayfile backup I had for July 2023-November 2023. It has all the 5 minute reads from the logs I have for those months..Gyvate wrote: ↑Wed 31 Jan 2024 7:45 am yes and no -
you can change it while it is running - but if the day file needs changes or even an entry, you have to do that manually, CMX won't go back.
It only creates entries for the past day at midnight or (same situation) when backfilling from ecowitt.net and crossing midnight
the header description of the day file (and other files) is in the ...\CumulusMX directory,
but for a few entries you better change it from within CMX (Data Logs --> Dayfile) as you have a proper mask which tells you what is what
All fixed. Thanks all!