Cumulus Support

Does the missing data actually exist on the Ecowitt.net site. You can find out by changing the graphing data to monthly. If it doesn't exist it cannot be downloaded.

Box wrote: ↑Mon 14 Mar 2022 8:40 pm Well I did that and from the attached mxdiags it seems to have downloaded loads of stuff but there still seems to be a gap in the Mar22log.txt file and I cannot see the data in the data viewer

The gap is between 0130 on 8 Mar and 1930 on the 10th

Is there some sort of housekeeping actvity that hasn't happened yet?

MX is reporting an exception when processing your historic data - strangely the exception message is blank. I'll investigate using your data now I have it....

The cause of your issue is your CO2 sensor, in the historic data Ecowitt are sending the 24 hour average values, but the current readings are missing - MX was expecting them and is erroring at this point.

I have made the code much more robust in the handling of the historic data, so it will now only error for a single sensor and the rest should come through. It also checks for situations like your C02 where part of the expected data is missing and will just process what it can find.

If you are willing to test a debug build for me, please send a direct message and I will get one zipped up for you.

mcrossley wrote: ↑Tue 15 Mar 2022 10:48 am The cause of your issue is your CO2 sensor, in the historic data Ecowitt are sending the 24 hour average values, but the current readings are missing - MX was expecting them and is erroring at this point.

I have made the code much more robust in the handling of the historic data, so it will now only error for a single sensor and the rest should come through. It also checks for situations like your C02 where part of the expected data is missing and will just process what it can find.

If you are willing to test a debug build for me, please send a direct message and I will get one zipped up for you.

DM sent

Thanks for the investigations, I thought it was me!

Strangely my Ecowitt MX install stopped logging at 9:00am Yesterday.

At 9:05am a Cron Job shut it down & backed it up then Started it again.
Didn't notice till around 4:00pm & at that point the Console showed no data.
Could ping the GW1000 & a restart of MX made no difference.

Didn't check WS-View, but the HP-2551 was Ok & so was the Data on Ecowitt.net.

Essentially I had a data gap from 9:00am to 4:19pm when it restarted after power cycling the GW1000.

Later in the Evening I tried shutting it down again (7:34pm) & restoring a backup from 12:05am,
(3 daily Cron backups still configured for testing),
Restarted & don't see any data being downloaded & now have the data gap from midnight to 7:35pm.

Have I missed something or done something wrong?
Handful of Diags files attached, all events should be covered within the scope of my settings.

Cheers

Phil.

Edit:- Was actually before 4:00pm I noticed, as Diags files shows a restart of MX at 3:54pm.
Have had a quick browse, but not sure what sticks out apart from a firmware version error when attempting to connect to the GW.
Possibly explains the gap in data, but not why I can't retrieve with the API.

Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached

cmx screenshot

Still haven't looked that far, but the 9:05am restart is where the problem started.
Will try again from the backups again later in the day when I have time to check everything.
Backups are available from 9, 17 & midnight for a few days back.

Edit:- Might run it in a Terminal Session after the restore, so I can see the responses as I start it.
Looking at one of the logs, I may not have restored correctly as after the restore it started with this....

Just Restored my 9:05am backup from 15/03 & restarted MX, but still seeing the data gap.
It extends from 9:00am yesterday up to the restart time.

Restarted in a Terminal session & output is MX Diags file is a copy I grabbed while it's still running in the Terminal Session.

Cheers.

Edit:- Did also Try running an instance of FTP Now to see it that Re-generated the Graphs.

Not sure what I should expect to see in this area of the log.

Box wrote: ↑Tue 15 Mar 2022 9:18 pm Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached

But the log files show a run that started on 15 Mar @ 20:23
The last data at that point was 10 Mar @ 07:50

The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.

That debug build appears to be working now for you.

Phil23 wrote: ↑Wed 16 Mar 2022 1:37 am Not sure what I should expect to see in this area of the log.

Code: Select all

2022-03-16 11:46:17.401 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:17.426 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:17.426 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-15%2009:01:00&end_date=2022-03-16%2009:01:00&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:19.775 This Cumulus MX instance is running the latest version
2022-03-16 11:46:21.752 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:22.419 API.GetHistoricData: Exception:
2022-03-16 11:46:22.419 GetHistoricData: Starting Historic Data Process
2022-03-16 11:46:22.423 API.GetHistoricData: Get Ecowitt Historic Data
2022-03-16 11:46:22.424 Ecowitt URL = https://api.ecowitt.net/api/v3/device/history?application_key=<<App-key>>&api_key=<<User-key>>&mac=FC:F5:C4:8D:43:C9&start_date=2022-03-16%2009:02:00&end_date=2022-03-16%2011:46:22&call_back=indoor,outdoor,wind,pressure,solar_and_uvi,temp_and_humidity_ch1,temp_and_humidity_ch2,temp_and_humidity_ch3,temp_and_humidity_ch4,temp_and_humidity_ch5,temp_and_humidity_ch6,temp_and_humidity_ch7,temp_and_humidity_ch8,soil_ch1,soil_ch2,soil_ch3,soil_ch4,soil_ch5,soil_ch6,soil_ch7,soil_ch8,temp_ch1,temp_ch2,temp_ch3,temp_ch4,temp_ch5,temp_ch6,temp_ch7,temp_ch8,leaf_ch1,leaf_ch2,leaf_ch3,leaf_ch4,leaf_ch5,leaf_ch6,leaf_ch7,leaf_ch8,indoor_co2,co2_aqi_combo,pm25_ch1,pm25_ch2,pm25_ch3,pm25_ch4&cycle_type=5min
2022-03-16 11:46:23.158 API.GetHistoricData: Ecowitt API Historic Response code: 200
2022-03-16 11:46:23.182 API.GetHistoricData: Exception:
2022-03-16 11:46:23.182 Lock: Station releasing the lock

You have the same error as @box was getting "API.GetHistoricData: Exception:"
I have a new build to fix this.

mcrossley wrote: ↑Wed 16 Mar 2022 2:47 pm

Box wrote: ↑Tue 15 Mar 2022 9:18 pm Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached

But the log files show a run that started on 15 Mar @ 20:23
The last data at that point was 10 Mar @ 07:50

The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.

That debug build appears to be working now for you.

Maybe I am not uncerstanding how this works , what data backup would I need to get the gap filled? I thought I had used a data backup from 22 Feb, so surely it would have filled in the gap on 8-10 Mar?

Box wrote: ↑Wed 16 Mar 2022 3:01 pm

mcrossley wrote: ↑Wed 16 Mar 2022 2:47 pm

Box wrote: ↑Tue 15 Mar 2022 9:18 pm Ok

Build 3172 loaded and ran fine

I found a data backup from before the gap and ran it

The CMX screen told me it had processed historry data between 22 Feb and today

I then opened the data log viewer and the gap remains, I have checked on the ecowitt site and there is definitely datan from this period

MX diags and mar22log.txt attached

But the log files show a run that started on 15 Mar @ 20:23
The last data at that point was 10 Mar @ 07:50

The diags log and monthly log file show that the catch-up worked perfectly, all data from 10 Mar 07:50 to 15 Mar 20:25 was received OK, processed OK, and added to the monthly log file OK.

That debug build appears to be working now for you.

Maybe I am not uncerstanding how this works , what data backup would I need to get the gap filled? I thought I had used a data backup from 22 Feb, so surely it would have filled in the gap on 8-10 Mar?

Should I doctor the Mar22log.txt file so that the last recorded data is before the current gap on 8 Mar and ensure it is all filled in (including data that is already there)?

You would need to go back to a backup from before "the gap". The logs you posted were from after.
The screen shot showed going back to 22 Feb which was before, but there were no logs from that run.