Welcome to the Cumulus Support forum.
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4018) - 28 March 2024
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4018) - 28 March 2024
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Ecowitt Api Historic Data
Moderator: mcrossley
-
- Posts: 3215
- Joined: Sat 13 Aug 2011 9:33 am
- Weather Station: Ecowitt HP2551
- Operating System: Windows 10 64bit
- Location: Burnham-on-Sea
- Contact:
Re: Ecowitt Api Historic Data
Does the missing data actually exist on the Ecowitt.net site. You can find out by changing the graphing data to monthly. If it doesn't exist it cannot be downloaded.
- mcrossley
- Posts: 12694
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Ecowitt Api Historic Data
MX is reporting an exception when processing your historic data - strangely the exception message is blank. I'll investigate using your data now I have it....Box wrote: ↑Mon 14 Mar 2022 8:40 pm Well I did that and from the attached mxdiags it seems to have downloaded loads of stuff but there still seems to be a gap in the Mar22log.txt file and I cannot see the data in the data viewer
The gap is between 0130 on 8 Mar and 1930 on the 10th
Is there some sort of housekeeping actvity that hasn't happened yet?
- mcrossley
- Posts: 12694
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Ecowitt Api Historic Data
The cause of your issue is your CO2 sensor, in the historic data Ecowitt are sending the 24 hour average values, but the current readings are missing - MX was expecting them and is erroring at this point.
I have made the code much more robust in the handling of the historic data, so it will now only error for a single sensor and the rest should come through. It also checks for situations like your C02 where part of the expected data is missing and will just process what it can find.
If you are willing to test a debug build for me, please send a direct message and I will get one zipped up for you.
I have made the code much more robust in the handling of the historic data, so it will now only error for a single sensor and the rest should come through. It also checks for situations like your C02 where part of the expected data is missing and will just process what it can find.
If you are willing to test a debug build for me, please send a direct message and I will get one zipped up for you.
-
- Posts: 40
- Joined: Sun 15 Nov 2020 8:57 am
- Weather Station: FineOffset Gw1002
- Operating System: Win10
Re: Ecowitt Api Historic Data
DM sentmcrossley wrote: ↑Tue 15 Mar 2022 10:48 am The cause of your issue is your CO2 sensor, in the historic data Ecowitt are sending the 24 hour average values, but the current readings are missing - MX was expecting them and is erroring at this point.
I have made the code much more robust in the handling of the historic data, so it will now only error for a single sensor and the rest should come through. It also checks for situations like your C02 where part of the expected data is missing and will just process what it can find.
If you are willing to test a debug build for me, please send a direct message and I will get one zipped up for you.
Thanks for the investigations, I thought it was me!
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: Ecowitt Api Historic Data
Strangely my Ecowitt MX install stopped logging at 9:00am Yesterday.
At 9:05am a Cron Job shut it down & backed it up then Started it again.
Didn't notice till around 4:00pm & at that point the Console showed no data.
Could ping the GW1000 & a restart of MX made no difference.
Didn't check WS-View, but the HP-2551 was Ok & so was the Data on Ecowitt.net.
Essentially I had a data gap from 9:00am to 4:19pm when it restarted after power cycling the GW1000.
Later in the Evening I tried shutting it down again (7:34pm) & restoring a backup from 12:05am,
(3 daily Cron backups still configured for testing),
Restarted & don't see any data being downloaded & now have the data gap from midnight to 7:35pm.
Have I missed something or done something wrong?
Handful of Diags files attached, all events should be covered within the scope of my settings.
Cheers
Phil.
Edit:- Was actually before 4:00pm I noticed, as Diags files shows a restart of MX at 3:54pm.
Have had a quick browse, but not sure what sticks out apart from a firmware version error when attempting to connect to the GW.
Possibly explains the gap in data, but not why I can't retrieve with the API.
At 9:05am a Cron Job shut it down & backed it up then Started it again.
Didn't notice till around 4:00pm & at that point the Console showed no data.
Could ping the GW1000 & a restart of MX made no difference.
Didn't check WS-View, but the HP-2551 was Ok & so was the Data on Ecowitt.net.
Essentially I had a data gap from 9:00am to 4:19pm when it restarted after power cycling the GW1000.
Later in the Evening I tried shutting it down again (7:34pm) & restoring a backup from 12:05am,
(3 daily Cron backups still configured for testing),
Restarted & don't see any data being downloaded & now have the data gap from midnight to 7:35pm.
Have I missed something or done something wrong?
Handful of Diags files attached, all events should be covered within the scope of my settings.
Cheers
Phil.
Edit:- Was actually before 4:00pm I noticed, as Diags files shows a restart of MX at 3:54pm.
Have had a quick browse, but not sure what sticks out apart from a firmware version error when attempting to connect to the GW.
Possibly explains the gap in data, but not why I can't retrieve with the API.
You do not have the required permissions to view the files attached to this post.
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
-
- Posts: 40
- Joined: Sun 15 Nov 2020 8:57 am
- Weather Station: FineOffset Gw1002
- Operating System: Win10
Re: Ecowitt Api Historic Data
Ok
Build 3172 loaded and ran fine
I found a data backup from before the gap and ran it
The CMX screen told me it had processed historry data between 22 Feb and today
I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period
MX diags and mar22log.txt attached
Build 3172 loaded and ran fine
I found a data backup from before the gap and ran it
The CMX screen told me it had processed historry data between 22 Feb and today
I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period
MX diags and mar22log.txt attached
You do not have the required permissions to view the files attached to this post.
-
- Posts: 40
- Joined: Sun 15 Nov 2020 8:57 am
- Weather Station: FineOffset Gw1002
- Operating System: Win10
Re: Ecowitt Api Historic Data
cmx screenshot
You do not have the required permissions to view the files attached to this post.
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: Ecowitt Api Historic Data
Still haven't looked that far, but the 9:05am restart is where the problem started.
Will try again from the backups again later in the day when I have time to check everything.
Backups are available from 9, 17 & midnight for a few days back.
Edit:- Might run it in a Terminal Session after the restore, so I can see the responses as I start it.
Looking at one of the logs, I may not have restored correctly as after the restore it started with this....
Code: Select all
2022-03-15 09:05:15.594 GW1000 Connect attempt 1
2022-03-15 09:05:16.622 GW1000 reconnected
2022-03-15 09:05:16.622 Connected OK
2022-03-15 09:05:16.623 Reading firmware version
2022-03-15 09:05:17.646 DoCommand(CMD_READ_FIRMWARE_VERSION): No response received
2022-03-15 09:05:17.646 GW1000 firmware version: ???
2022-03-15 09:05:17.648 Reading GW1000 system info
2022-03-15 09:05:18.405 DoCommand(CMD_READ_SSSS): Invalid response
2022-03-15 09:05:18.406 command resp=80, checksum=OK
2022-03-15 09:05:18.756 This Cumulus MX instance is running the latest version
2022-03-15 12:03:25.285 WS Connect From : 192.168.59.24
2022-03-15 15:54:03.477 WS Client Disconnected
2022-03-15 15:54:04.725 WS Connect From : 192.168.59.24
Backups are available from 9, 17 & midnight for a few days back.
Edit:- Might run it in a Terminal Session after the restore, so I can see the responses as I start it.
Looking at one of the logs, I may not have restored correctly as after the restore it started with this....
Code: Select all
2022-03-15 19:34:25.662 Opening station
2022-03-15 19:34:25.688 Last update time from today.ini: 15/03/2022 7:30:00 PM
2022-03-15 19:34:25.690 Read today file: Date = 15/03/2022, LastUpdateTime = 15/03/2022 7:30:00 PM, Month = 3
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: Ecowitt Api Historic Data
Just Restored my 9:05am backup from 15/03 & restarted MX, but still seeing the data gap.
It extends from 9:00am yesterday up to the restart time.
Restarted in a Terminal session & output is MX Diags file is a copy I grabbed while it's still running in the Terminal Session.
Cheers.
Edit:- Did also Try running an instance of FTP Now to see it that Re-generated the Graphs.
It extends from 9:00am yesterday up to the restart time.
Restarted in a Terminal session & output is MX Diags file is a copy I grabbed while it's still running in the Terminal Session.
Cheers.
Edit:- Did also Try running an instance of FTP Now to see it that Re-generated the Graphs.
You do not have the required permissions to view the files attached to this post.
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: Ecowitt Api Historic Data
Not sure what I should expect to see in this area of the log.
Code: Select all
2022-03-16 11:46:17.401 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:17.426 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:17.426 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-15%2009:01:00&end_date=2022-03-16%2009:01:00&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:19.775 This Cumulus MX instance is running the latest version
2022-03-16 11:46:21.752 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:22.419 API.GetHistoricData: Exception:
2022-03-16 11:46:22.419 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:22.423 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:22.424 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-16%2009:02:00&end_date=2022-03-16%2011:46:22&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:23.158 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:23.182 API.GetHistoricData: Exception:
2022-03-16 11:46:23.182 Lock: Station releasing the lock
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
- mcrossley
- Posts: 12694
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Ecowitt Api Historic Data
But the log files show a run that started on 15 Mar @ 20:23Box wrote: ↑Tue 15 Mar 2022 9:18 pm Ok
Build 3172 loaded and ran fine
I found a data backup from before the gap and ran it
The CMX screen told me it had processed historry data between 22 Feb and today
I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period
MX diags and mar22log.txt attached
The last data at that point was 10 Mar @ 07:50
The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.
That debug build appears to be working now for you.
- mcrossley
- Posts: 12694
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Ecowitt Api Historic Data
You have the same error as @box was getting "API.GetHistoricData: Exception:"Phil23 wrote: ↑Wed 16 Mar 2022 1:37 am Not sure what I should expect to see in this area of the log.
Code: Select all
2022-03-16 11:46:17.401 GetHistoricData: Starting Historic Data Process 2022-03-16 11:46:17.426 API.GetHistoricData: Get Ecowitt Historic Data 2022-03-16 11:46:17.426 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-15%2009:01:00&end_date=2022-03-16%2009:01:00&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min 2022-03-16 11:46:19.775 This Cumulus MX instance is running the latest version 2022-03-16 11:46:21.752 API.GetHistoricData: Ecowitt API Historic Response code: 200 2022-03-16 11:46:22.419 API.GetHistoricData: Exception: 2022-03-16 11:46:22.419 GetHistoricData: Starting Historic Data Process 2022-03-16 11:46:22.423 API.GetHistoricData: Get Ecowitt Historic Data 2022-03-16 11:46:22.424 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-16%2009:02:00&end_date=2022-03-16%2011:46:22&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min 2022-03-16 11:46:23.158 API.GetHistoricData: Ecowitt API Historic Response code: 200 2022-03-16 11:46:23.182 API.GetHistoricData: Exception: 2022-03-16 11:46:23.182 Lock: Station releasing the lock
I have a new build to fix this.
-
- Posts: 40
- Joined: Sun 15 Nov 2020 8:57 am
- Weather Station: FineOffset Gw1002
- Operating System: Win10
Re: Ecowitt Api Historic Data
Maybe I am not uncerstanding how this works , what data backup would I need to get the gap filled? I thought I had used a data backup from 22 Feb, so surely it would have filled in the gap on 8-10 Mar?mcrossley wrote: ↑Wed 16 Mar 2022 2:47 pmBut the log files show a run that started on 15 Mar @ 20:23Box wrote: ↑Tue 15 Mar 2022 9:18 pm Ok
Build 3172 loaded and ran fine
I found a data backup from before the gap and ran it
The CMX screen told me it had processed historry data between 22 Feb and today
I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period
MX diags and mar22log.txt attached
The last data at that point was 10 Mar @ 07:50
The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.
That debug build appears to be working now for you.
-
- Posts: 40
- Joined: Sun 15 Nov 2020 8:57 am
- Weather Station: FineOffset Gw1002
- Operating System: Win10
Re: Ecowitt Api Historic Data
Should I doctor the Mar22log.txt file so that the last recorded data is before the current gap on 8 Mar and ensure it is all filled in (including data that is already there)?Box wrote: ↑Wed 16 Mar 2022 3:01 pmMaybe I am not uncerstanding how this works , what data backup would I need to get the gap filled? I thought I had used a data backup from 22 Feb, so surely it would have filled in the gap on 8-10 Mar?mcrossley wrote: ↑Wed 16 Mar 2022 2:47 pmBut the log files show a run that started on 15 Mar @ 20:23Box wrote: ↑Tue 15 Mar 2022 9:18 pm Ok
Build 3172 loaded and ran fine
I found a data backup from before the gap and ran it
The CMX screen told me it had processed historry data between 22 Feb and today
I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period
MX diags and mar22log.txt attached
The last data at that point was 10 Mar @ 07:50
The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.
That debug build appears to be working now for you.
- mcrossley
- Posts: 12694
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: Ecowitt Api Historic Data
You would need to go back to a backup from before "the gap". The logs you posted were from after.
The screen shot showed going back to 22 Feb which was before, but there were no logs from that run.
The screen shot showed going back to 22 Feb which was before, but there were no logs from that run.