CMX 3.15.2 (3171) not downloading/archiving the Ecowitt.net data *solved w/ 3173*

[Gyvate]

scenario:

- CMX stopped recording today (18-Mar-2022) 01:40

- after restart I realized the Ecowitt API option wasn't set, so provided the credentials + MAC

- after restart no download happened - I suppose it was because of the last update timestamp in today.ini, so I changed that to 01:40

and cleaned the Mrz21log.txt entries after 01:40

CMX(48)_20220318.JPG

- after restart data as per MXdiags seemed to be downloaded but were never transferred into Mrz21log.txt
(locale DE !)


18.03.22;01:38;5,8;74;1,5;5,32;14,04;1;0,0;0,0;1034,2;248,7;18,4;52;3,60;4,9;5,8;0,0;0;0,01;35,51;3,0;0;0,0;15;0,0;0,0;4,9;5,8
18.03.22;01:39;5,8;74;1,5;5,12;14,04;4;0,0;0,0;1034,3;248,7;18,4;52;4,32;4,9;5,8;0,0;0;0,01;35,51;3,1;0;0,0;44;0,0;0,0;4,9;5,8
18.03.22;01:40;5,8;74;1,5;5,00;14,04;7;0,0;0,0;1034,1;248,7;18,5;52;3,60;5,0;5,8;0,0;0;0,01;35,51;3,1;0;0,0;31;0,0;0,0;5,0;5,8
18.03.22;11:46;10,0;64;3,5;3,60;11,52;53;0,0;0,0;1037,0;248,7;19,5;48;3,60;10,0;10,0;1,0;274;0,10;35,61;7,9;607;0,0;53;0,0;0,0;10,0;8,8
18.03.22;11:47;10,0;64;3,5;9,05;21,24;174;0,0;0,0;1036,9;248,7;19,5;48;14,04;10,0;10,0;1,0;274;0,10;35,61;6,8;608;0,0;188;0,0;0,0;8,8;8,8
18.03.22;11:48;10,1;64;3,6;7,79;21,24;169;0,0;0,0;1036,8;248,7;19,4;48;7,92;10,1;10,1;1,0;269;0,10;35,61;7,2;609;0,0;177;0,0;0,0;9,2;8,9
18.03.22;11:49;10,1;64;3,6;7,16;21,24;166;0,0;0,0;1036,9;248,7;19,4;48;4,32;10,1;10,1;1,0;271;0,10;35,61;7,3;610;0,0;41;0,0;0,0;9,3;8,9

what am I doing wrong ?
or maybe some bug in the "experimental" feature ?

[HansR]

Better would be to restore a backup (from the 17th?) but not the Cumulus.ini because you just changed the credentials and hopefully nothing else.
I believe you will get 5 minute data because Ecowitt only has 5 minute interval.
It probably is not a bug in an experimental feature because it is working for others.

[Gyvate]

There are many workarounds.
I also have a mirror installation and could just copy the files (or entries) over.
But this doesn't explain why the data is read (see MXdiags) but not stored.

I have tried the backfill option before and was ß-testing the feature.
So it should work from where the logging stopped.

Of course I can take a backup from the day before, see if all is being filled,
and fill in the data from 00:00 till 01:40 manually from my old (saved!) month file.
(2nd workaround).
But, that's not the expected behaviour.

If someone is not running CMX 24/7, they cannot each time take some backup.
Starting CMX again should initiate and provide the backfill.

Something is simply wrong - either on my end or at the CMX end.

And it's good to know what that is/was and how it can be avoided in the future.
In professional terms: I'm not looking for incident management but for problem management.

But, nonetheless, thanks for proposing the earlier backup approach.

[freddie]

It's because you did the initial restart when the credentials and MAC weren't set. This would've populated a later time in your monthly log than the data gap interval. MX assumes continuous record from the latest entry in the monthly log, so wouldn't backfill the missing data. Hence the suggestion from Hans to restore complete data file set (apart from Cumulus.ini) from backup, rather than faffing around with partial edits.

[PaulMy]

I presume you noticed the errors in MXdiags and have or tried to fix them:

Code: Select all

2022-03-18 11:44:59.818 LoadDayFile: Attempting to load the day file
2022-03-18 11:45:00.349 ParseDayFileRec: Error at record 1 - Die Eingabezeichenfolge hat das falsche Format.
2022-03-18 11:45:00.349 LoadDayFile: Error at line 406 of data\dayfile.txt : Error at record 1 = "24.01.22,25,56,287,11:10,-0,6,23:45,6,5,16:04,1030,3,14:27,1032,2,00:04,0,0,00:00,0,0,2,3,70,0,9,72,12:31,72,16:06,96,22:00,0,27,3,5,6,5,16:04,4,2,16:06,-3,0,23:30,0,0,00:00,-1,2,07:27,2,0,16:04,-1,2,23:45,242,16,0,0,0,176,13:05,0,0,00:00,6,5,16:04,-1,2,07:27,6,5,16:04,1834,1" - Die Eingabezeichenfolge hat das falsche Format.
2022-03-18 11:45:00.349 Please edit the file to correct the error
2022-03-18 11:45:00.396 LoadDayFile: Dayfile parse = 582 ms
2022-03-18 11:45:00.396 LoadDayFile: Loaded 457 entries to recent daily data list

and:

Code: Select all

2022-03-18 11:45:09.302 Loading last N hour data from data logs: 18.03.2022 01:40:00
2022-03-18 11:45:09.333 LoadRecent: Attempting to load 7 days of entries to recent data list
2022-03-18 11:45:16.724 Error parsing log file record: Die Eingabezeichenfolge hat das falsche Format.
2022-03-18 11:45:16.740 Log record: 13.03.22;10:45;11,8;44;-0,1;4,03;12,60;122;0,0;0,1;1014,5;234,2;18,2;43;3,60;11,8;11,8;3,0;604;0,18;38,16;9,0;493;2,3;116;0,0;0,1;11,5;9,613.03.22;10:39;11,0;46;-0,2;5,20;17,28;115;0,0;0,1;1014,9;234,2;18,1;44;4,32;11,0;11,0;3,0;590;0,18;38,16;8,0;482;2,2;143;0,0;0,1;10,6;8,813.03.22;10:47;11,1;50;1,1;4,19;9,72;199;0,0;0,0;1014,7;234,2;19,1;43;3,96;11,1;11,1;2,0;390;0,16;32,45;8,5;497;2,4;29;0,0;0,0;10,9;9,2
2022-03-18 11:45:16.740 LoadRecent: Error at line 17878 of data\Mrz22log.txt : Die Eingabezeichenfolge hat das falsche Format.
2022-03-18 11:45:16.740 Please edit the file to correct the error
2022-03-18 11:45:19.396 LoadRecent: Loaded 0 new entries to recent database

I am not fully familiar with the Ecowitt historical download but there seams to have been a significant import from 11:45:52 to 11:45:57.

Enjoy,
Paul

[Gyvate]

That's not a fully satisfactory explanation and it doesn't reflect the course of events.

Where would CMX remember that I had made this change later ? It has to get that from somewhere - as it would do from the backup.
the today.ini entry was 01:40 as you can also see from the screen shot.

Is there an update check on the time stamp of Cumulus.ini (and what sense would this make and the file can be updated for plenty of reasons) ?
My monthly logs (normal + extra) stopped at 01:40.

There was not further entry. The later entries you see were created after the restart and the unsuccessful backfill.
You can cross-check the MXdiags files and time stamps against when the logging started again.

[Gyvate]

@PaulMy
yes, I've seen the messages and corrected the file (there was a LF/CR too much).
But such errors do not keep CMX from continuing writing the minute log entries.
Especially an error in the dayfile.txt
And, the month file error was an entry from 5 days earlier.
So, I'm wondering, why it would keep the backfill from happening

[freddie]

That's not a fully satisfactory explanation and it doesn't reflect the course of events.

Where would CMX remember that I had made this change later ? It has to get that from somewhere - as it would do from the backup.
the today.ini entry was 01:40 as you can also see from the screen shot.

Is there an update check on the time stamp of Cumulus.ini (and what sense would this make and the file can be updated for plenty of reasons) ?
My monthly logs (normal + extra) stopped at 01:40.

There was not further entry. The later entries you see were created after the restart and the unsuccessful backfill.
You can cross-check the MXdiags files and time stamps against when the logging started again.

Apologies for not noticing that a logging entry hadn't been made - my mistake. I don't know why the second restart didn't pick up your archived data.

There isn't a timestamp check done on Cumulus.ini - the reason why this file shouldn't have been copied from backup was the fact that you had entered the Ecowitt API credentials and MAC since the 01:40 stop, and to copy the file from backup would have meant those entries disappeared again.

[Gyvate]

I've repeated the whole exercise - with a backup from 18-Mar-2022 00:00 as suggested by @HansR and @freddie
result: the same - all data read as per MXdiags log - no data written
17.03.22;23:57;5,7;79;2,3;3,35;8,28;6;0,0;0,0;1034,0;248,7;18,6;52;4,32;5,7;5,7;0,0;0;0,40;35,50;3,4;0;0,0;1;0,0;0,0;5,7;5,7
17.03.22;23:58;5,7;79;2,3;3,25;6,12;6;0,0;0,0;1034,1;248,7;18,6;52;2,16;5,7;5,7;0,0;0;0,40;35,50;3,5;0;0,0;12;0,0;0,0;5,7;5,7
17.03.22;23:59;5,7;79;2,3;3,12;6,48;7;0,0;0,0;1034,1;248,7;18,6;52;4,68;5,7;5,7;0,0;0;0,41;35,50;3,5;0;0,0;2;0,0;0,0;5,7;5,7
18.03.22;00:00;5,7;79;2,3;3,44;9,72;5;0,0;0,0;1034,1;248,7;18,6;52;2,52;5,7;5,7;0,0;0;0,00;35,50;3,4;0;0,0;352;0,0;0,0;5,7;5,7
19.03.22;16:51;12,4;33;-3,4;14,11;28,44;192;0,0;0,0;1030,4;248,7;22,3;36;18,00;12,4;12,4;0,0;161;0,00;35,50;7,2;196;8,1;188;0,0;0,0;10,4;9,5

I have a suspicion though - this instance runs on locale = DE, i.e. 01.01.2022 instead of 01/01/2022, "," instead of ".", ";" instead of ","
could it be that the data from ecowitt.net, which are locale=en cannot be properly transformed into locale=DE data ? the ecowitt data is decimal point = "."
However the MDdiags only shows an empty exception line:
"2022-03-19 16:50:19.178 API.GetHistoricData: Exception:"

[freddie]

However the MDdiags only shows an empty exception line:
"2022-03-19 16:50:19.178 API.GetHistoricData: Exception:"

Ah @mcrossley spotted an issue with the latest release that caused that exception for some users, and gave them a new version to test out. As far as I know the testing was successful. He probably hasn't created a new release yet as he has been unwell this week. There's more about it in this topic:
viewtopic.php?f=40&t=20238

[Gyvate]

OK, let me see when 3172 or higher will be released. I had read the other thread to some point but didn't make the connection to my scenario as my earlier (successful) tests were done with the CO2 sensor active.
I will keep the files of my scenario to verify.
Meanwhile I have backfilled the actual log files from my mirror instance, but will be able to go back to the date in question (18-Mar-2022 01:40) any time.

[mcrossley]

You seem to have the same issue. Ecowitt are not returning some data in the requests that was expected (and they appeared to return in the past), the new release is now more robust and resilient to single categories of expected data being missing.

[mcrossley]

Please try version 3.15.3 I have just released.

[Gyvate]

3173 worked in my case !

However, when I look at the monthfile now there is a rather big gap left

20.03.22;20:10;9,5;72;4,7;0,89;2,90;5;0,0;0,0;1027,4;248,7;20,3;46;0,97;9,5;9,5;0,0;0;1,30;40,70;8,1;0;2,8;5;0,0;0,0;9,5;9,5
20.03.22;20:15;9,3;73;4,7;1,13;3,22;351;0,0;0,0;1027,4;248,7;20,3;46;0,64;9,3;9,3;0,0;0;1,30;40,70;7,9;0;2,8;331;0,0;0,0;9,3;9,3
20.03.22;20:20;9,2;73;4,6;1,61;3,22;348;0,0;0,0;1027,4;248,7;20,3;46;0,32;9,2;9,2;0,0;0;1,30;40,70;7,7;0;2,8;24;0,0;0,0;9,2;9,2
*** end of ecowitt.net download - start of normal CMX logging ***
20.03.22;20:33;9,0;74;4,6;0,36;2,16;0;0,0;0,0;1027,4;248,7;20,2;46;0,00;9,0;9,0;0,0;0;1,30;40,70;7,7;0;2,1;52;0,0;0,0;9,0;9,0
20.03.22;20:34;9,0;74;4,6;0,13;2,16;0;0,0;0,0;1027,5;248,7;20,2;46;0,00;9,0;9,0;0,0;0;1,30;40,70;7,8;0;2,1;55;0,0;0,0;9,0;9,0

2022-03-20 20:09:44.123 Last update=2022-03-18T01:40:00
2022-03-20 20:09:54.904 Connected to station
2022-03-20 20:10:35.482 Processing history data from 2022-03-18 01:41 to 2022-03-19 01:41...
2022-03-20 20:20:29.038 Processing history data from 2022-03-19 01:41 to 2022-03-20 01:41...
2022-03-20 20:29:27.979 Processing history data from 2022-03-20 01:41 to 2022-03-20 20:29...

Therefore I have a suggestion to make:
If CMX has finished its last (daily) download, could another history check be made to cover the time passed since the last daily download ?
Like this the gap would be minimized and end up to be maximum 5 minutes while it is now in the above example 13 minutes.
Thanks for considering.

[mcrossley]

I have noticed that Ecowiit often do not return the last record you would expect, it can be a many minutes into an interval and the previous one is still not available

You can see you last request was made at 20:29:28, for records up to 20:29, you would expect the 20:25 record to be available at that point, but nope, it's the luck of draw

Ecowitt seem to use a timestamp of the start of the interval rather than the end as you might expect - I did, every other API uses the end time. So v3.15.3 should avoid those many 10 minute "gaps" at the end and have the expected last 5 minute, but often that last record is just not returned.