Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4018) - 28 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Ecowitt Api Historic Data

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

water01
Posts: 3215
Joined: Sat 13 Aug 2011 9:33 am
Weather Station: Ecowitt HP2551
Operating System: Windows 10 64bit
Location: Burnham-on-Sea
Contact:

Re: Ecowitt Api Historic Data

Post by water01 »

Does the missing data actually exist on the Ecowitt.net site. You can find out by changing the graphing data to monthly. If it doesn't exist it cannot be downloaded.
David
Image
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Ecowitt Api Historic Data

Post by mcrossley »

Box wrote: Mon 14 Mar 2022 8:40 pm Well I did that and from the attached mxdiags it seems to have downloaded loads of stuff but there still seems to be a gap in the Mar22log.txt file and I cannot see the data in the data viewer

The gap is between 0130 on 8 Mar and 1930 on the 10th

Is there some sort of housekeeping actvity that hasn't happened yet?
MX is reporting an exception when processing your historic data - strangely the exception message is blank. I'll investigate using your data now I have it....
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Ecowitt Api Historic Data

Post by mcrossley »

The cause of your issue is your CO2 sensor, in the historic data Ecowitt are sending the 24 hour average values, but the current readings are missing - MX was expecting them and is erroring at this point.

I have made the code much more robust in the handling of the historic data, so it will now only error for a single sensor and the rest should come through. It also checks for situations like your C02 where part of the expected data is missing and will just process what it can find.

If you are willing to test a debug build for me, please send a direct message and I will get one zipped up for you.
Box
Posts: 40
Joined: Sun 15 Nov 2020 8:57 am
Weather Station: FineOffset Gw1002
Operating System: Win10

Re: Ecowitt Api Historic Data

Post by Box »

mcrossley wrote: Tue 15 Mar 2022 10:48 am The cause of your issue is your CO2 sensor, in the historic data Ecowitt are sending the 24 hour average values, but the current readings are missing - MX was expecting them and is erroring at this point.

I have made the code much more robust in the handling of the historic data, so it will now only error for a single sensor and the rest should come through. It also checks for situations like your C02 where part of the expected data is missing and will just process what it can find.

If you are willing to test a debug build for me, please send a direct message and I will get one zipped up for you.
DM sent

Thanks for the investigations, I thought it was me!
Phil23
Posts: 884
Joined: Sat 16 Jul 2016 11:59 pm
Weather Station: Davis VP2+ & GW1000 (Standalone)
Operating System: Win10 Pro / rPi Buster
Location: Australia

Re: Ecowitt Api Historic Data

Post by Phil23 »

Strangely my Ecowitt MX install stopped logging at 9:00am Yesterday.

At 9:05am a Cron Job shut it down & backed it up then Started it again.
Didn't notice till around 4:00pm & at that point the Console showed no data.
Could ping the GW1000 & a restart of MX made no difference.

Didn't check WS-View, but the HP-2551 was Ok & so was the Data on Ecowitt.net.

Essentially I had a data gap from 9:00am to 4:19pm when it restarted after power cycling the GW1000.

Later in the Evening I tried shutting it down again (7:34pm) & restoring a backup from 12:05am,
(3 daily Cron backups still configured for testing),
Restarted & don't see any data being downloaded & now have the data gap from midnight to 7:35pm.

Have I missed something or done something wrong?
Handful of Diags files attached, all events should be covered within the scope of my settings.

Cheers

Phil.

Edit:- Was actually before 4:00pm I noticed, as Diags files shows a restart of MX at 3:54pm.
Have had a quick browse, but not sure what sticks out apart from a firmware version error when attempting to connect to the GW.
Possibly explains the gap in data, but not why I can't retrieve with the API.
You do not have the required permissions to view the files attached to this post.
:Now: :Today/Yesterday:

Image

Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Box
Posts: 40
Joined: Sun 15 Nov 2020 8:57 am
Weather Station: FineOffset Gw1002
Operating System: Win10

Re: Ecowitt Api Historic Data

Post by Box »

Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached
You do not have the required permissions to view the files attached to this post.
Box
Posts: 40
Joined: Sun 15 Nov 2020 8:57 am
Weather Station: FineOffset Gw1002
Operating System: Win10

Re: Ecowitt Api Historic Data

Post by Box »

cmx screenshot
You do not have the required permissions to view the files attached to this post.
Phil23
Posts: 884
Joined: Sat 16 Jul 2016 11:59 pm
Weather Station: Davis VP2+ & GW1000 (Standalone)
Operating System: Win10 Pro / rPi Buster
Location: Australia

Re: Ecowitt Api Historic Data

Post by Phil23 »

Still haven't looked that far, but the 9:05am restart is where the problem started.

Code: Select all

2022-03-15 09:05:15.594 GW1000 Connect attempt 1
2022-03-15 09:05:16.622 GW1000 reconnected
2022-03-15 09:05:16.622 Connected OK
2022-03-15 09:05:16.623 Reading firmware version
2022-03-15 09:05:17.646 DoCommand(CMD_READ_FIRMWARE_VERSION): No response received
2022-03-15 09:05:17.646 GW1000 firmware version: ???
2022-03-15 09:05:17.648 Reading GW1000 system info
2022-03-15 09:05:18.405 DoCommand(CMD_READ_SSSS): Invalid response
2022-03-15 09:05:18.406 command resp=80, checksum=OK
2022-03-15 09:05:18.756 This Cumulus MX instance is running the latest version
2022-03-15 12:03:25.285 WS Connect From : 192.168.59.24
2022-03-15 15:54:03.477 WS Client Disconnected
2022-03-15 15:54:04.725 WS Connect From : 192.168.59.24
Will try again from the backups again later in the day when I have time to check everything.
Backups are available from 9, 17 & midnight for a few days back.

Edit:- Might run it in a Terminal Session after the restore, so I can see the responses as I start it.
Looking at one of the logs, I may not have restored correctly as after the restore it started with this....

Code: Select all

2022-03-15 19:34:25.662 Opening station
2022-03-15 19:34:25.688 Last update time from today.ini: 15/03/2022 7:30:00 PM
2022-03-15 19:34:25.690 Read today file: Date = 15/03/2022, LastUpdateTime = 15/03/2022 7:30:00 PM, Month = 3
:Now: :Today/Yesterday:

Image

Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Phil23
Posts: 884
Joined: Sat 16 Jul 2016 11:59 pm
Weather Station: Davis VP2+ & GW1000 (Standalone)
Operating System: Win10 Pro / rPi Buster
Location: Australia

Re: Ecowitt Api Historic Data

Post by Phil23 »

Just Restored my 9:05am backup from 15/03 & restarted MX, but still seeing the data gap.
It extends from 9:00am yesterday up to the restart time.

Restarted in a Terminal session & output is
Screenshot 2022-03-16 120838.jpg
MX Diags file is a copy I grabbed while it's still running in the Terminal Session.

Cheers.

Edit:- Did also Try running an instance of FTP Now to see it that Re-generated the Graphs.
You do not have the required permissions to view the files attached to this post.
:Now: :Today/Yesterday:

Image

Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Phil23
Posts: 884
Joined: Sat 16 Jul 2016 11:59 pm
Weather Station: Davis VP2+ & GW1000 (Standalone)
Operating System: Win10 Pro / rPi Buster
Location: Australia

Re: Ecowitt Api Historic Data

Post by Phil23 »

Not sure what I should expect to see in this area of the log.

Code: Select all

2022-03-16 11:46:17.401 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:17.426 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:17.426 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-15%2009:01:00&end_date=2022-03-16%2009:01:00&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:19.775 This Cumulus MX instance is running the latest version
2022-03-16 11:46:21.752 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:22.419 API.GetHistoricData: Exception:
2022-03-16 11:46:22.419 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:22.423 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:22.424 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-16%2009:02:00&end_date=2022-03-16%2011:46:22&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:23.158 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:23.182 API.GetHistoricData: Exception:
2022-03-16 11:46:23.182 Lock: Station releasing the lock
:Now: :Today/Yesterday:

Image

Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Ecowitt Api Historic Data

Post by mcrossley »

Box wrote: Tue 15 Mar 2022 9:18 pm Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached
But the log files show a run that started on 15 Mar @ 20:23
The last data at that point was 10 Mar @ 07:50

The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.

That debug build appears to be working now for you.
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Ecowitt Api Historic Data

Post by mcrossley »

Phil23 wrote: Wed 16 Mar 2022 1:37 am Not sure what I should expect to see in this area of the log.

Code: Select all

2022-03-16 11:46:17.401 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:17.426 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:17.426 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-15%2009:01:00&end_date=2022-03-16%2009:01:00&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:19.775 This Cumulus MX instance is running the latest version
2022-03-16 11:46:21.752 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:22.419 API.GetHistoricData: Exception:
2022-03-16 11:46:22.419 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:22.423 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:22.424 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-16%2009:02:00&end_date=2022-03-16%2011:46:22&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:23.158 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:23.182 API.GetHistoricData: Exception:
2022-03-16 11:46:23.182 Lock: Station releasing the lock
You have the same error as @box was getting "API.GetHistoricData: Exception:"
I have a new build to fix this.
Box
Posts: 40
Joined: Sun 15 Nov 2020 8:57 am
Weather Station: FineOffset Gw1002
Operating System: Win10

Re: Ecowitt Api Historic Data

Post by Box »

mcrossley wrote: Wed 16 Mar 2022 2:47 pm
Box wrote: Tue 15 Mar 2022 9:18 pm Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached
But the log files show a run that started on 15 Mar @ 20:23
The last data at that point was 10 Mar @ 07:50

The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.

That debug build appears to be working now for you.
Maybe I am not uncerstanding how this works , what data backup would I need to get the gap filled? I thought I had used a data backup from 22 Feb, so surely it would have filled in the gap on 8-10 Mar?
Box
Posts: 40
Joined: Sun 15 Nov 2020 8:57 am
Weather Station: FineOffset Gw1002
Operating System: Win10

Re: Ecowitt Api Historic Data

Post by Box »

Box wrote: Wed 16 Mar 2022 3:01 pm
mcrossley wrote: Wed 16 Mar 2022 2:47 pm
Box wrote: Tue 15 Mar 2022 9:18 pm Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached
But the log files show a run that started on 15 Mar @ 20:23
The last data at that point was 10 Mar @ 07:50

The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.

That debug build appears to be working now for you.
Maybe I am not uncerstanding how this works , what data backup would I need to get the gap filled? I thought I had used a data backup from 22 Feb, so surely it would have filled in the gap on 8-10 Mar?
Should I doctor the Mar22log.txt file so that the last recorded data is before the current gap on 8 Mar and ensure it is all filled in (including data that is already there)?
User avatar
mcrossley
Posts: 12694
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Ecowitt Api Historic Data

Post by mcrossley »

You would need to go back to a backup from before "the gap". The logs you posted were from after.
The screen shot showed going back to 22 Feb which was before, but there were no logs from that run.
Post Reply