Repairing Rain

[HansR]

I installed a new GW1100 and unfortunately I did transfer the old rain settings from the previous GW1100. This resulted in a doubling of the rain and therefor erroneous records and erroneous rain values in the jan24log.txt. I managed to reset it all and the last thing I did was removing the entries in jan24log.txt so I only had entries from the old and new GW1100. That seemed to work fine.

But at startup (and likely at rollover, I'll see that tonight) I get the erroneous values back, this time only in the charts after a power failure (not recorded by my home system btw so it could also be a spontaneous reboot of the pi). So I fear that at rollover that value will actually propagate through everything again.

The big question is: where does that value (111.9 mm) come in the charts from?
It is not in the log, not in the records and currently I only find it in today.ini and yesterday.ini.

I do find it :

1. In today.ini for High24h and HourlyHigh at the moment of the power dip

   Code: Select all

   [Rain]
   High=0
   HTime=00:00
   HourlyHigh=111.90000000000001
   HHourlyTime=05:05
   High24h=111.90000000000001
   High24hTime=05:05
   Start=3.3999999999999999
   Yesterday=0
   LastTip=2024-01-28 05:05
   ConsecutiveRainDays=0
   ConsecutiveDryDays=1
   RG11Today=0
   Midnight=3.3999999999999999
   Last=3.3999999999999999
   

2. In yesterday.ini for HourlyHigh (Note the 7:15 time of the peak i.s.o. 5:05 we see in today.ini)

   Code: Select all

   [Rain]
   High=0
   HTime=00:00
   HourlyHigh=111.90000000000001
   HHourlyTime=07:15
   RG11Yesterday=0
   High24h=0
   High24hTime=07:20
   

Now, checking the logfile I find:

Code: Select all

2024-01-28 05:11:44.382 891 web tags initialised
2024-01-28 05:11:44.392 GetHistoricData: Starting Historic Data Process
2024-01-28 05:11:44.402 API.GetHistoricData: Get Ecowitt Historic Data
2024-01-28 05:11:44.412 API.GetHistoricData: Processing history data from 2024-01-28 05:01 to 2024-01-28 05:16...
2024-01-28 05:11:46.249 Processing data for 28-1-2024 05:05:00
2024-01-28 05:11:46.393 New all-time record: New time = 2024-01-28 05:05, new value = 111,900 "High daily rain" prev time = 2021-06-18 00:00, prev value =  45,600
2024-01-28 05:11:46.437 New monthly record: month = 01: New time = 2024-01-28 00:00, new value = 111,900 "High daily rain" prev time = 2023-01-04 00:00, prev value = 25,200
2024-01-28 05:11:46.502 Windrun: 7,9km/h for 5 minutes = 0,7km
2024-01-28 05:11:46.530 DoLogFile: Writing log entry for 28-1-2024 05:05:00
2024-01-28 05:11:46.559 DoLogFile: log entry for 28-1-2024 05:05:00 written
2024-01-28 05:11:46.562 Writing today.ini, LastUpdateTime = 28-1-2024 05:05:00 raindaystart = 3,40 rain counter = 115,30
2024-01-28 05:11:46.848 New all-time record: New time = 2024-01-28 05:05, new value = 111,900 "High hourly rain" prev time = 2024-01-27 07:15, prev value =  25,000
2024-01-28 05:11:46.876 New monthly record: month = 01: New time = 2024-01-28 05:05, new value = 111,900 "High hourly rain" prev time = 2024-01-27 07:15, prev value = 25,000
2024-01-28 05:11:46.918 API.GetStationList: Get Ecowitt Station List
2024-01-28 05:11:47.584 Starting Ecowitt Local API
2024-01-28 05:11:47.589 Starting Extra Sensors

So AHA... it comes from the history on Ecowitt.net...
That is interesting: how to correct that value (note: all values in CMX now seem correct for the moment but I really fear it all comes back at rollover time.

Any suggestion on how to remove that value from Ecowitt? Why does CMX take a record value from Ecowitt if it does through so much trouble keeping track of records. I think if CMX has no control over Ecowitt record values, it should not use them for setting records in CMX.

Anyway... how to deal with this?

[mcrossley]

cumulus just uses the year rain counter. So, you had the gateway upload to Ecowitt a zero value, then edited the value on the gateway. Cumulus sees the counter increase and records it as rain**. If you had edited the gateway value before starting Cumulus, then some issues would have been avoided, but really it needed to edited "off-line" so the zero value was never sent to Ecowitt.

** It isn't quite working as intended, Cumulus is supposed (according to the C1 FAQ) ignore large increments in the rainfall counter. Afaik it only accounts for large decreases, so it is either something C1 does that got missed in the MX code, or C1 never really did it either!

I'll add a ToDo to look at that...

[HansR]

I could not avoid... As I was unaware of anything wrong... My bad

Oh and the edits were done with CMX down
It comes back during startup.

But I would appreciate if something can be done before it propagates to unreparable. It comes back to the dayfile but is not seen in the logfile.

[HansR]

So, you had the gateway upload to Ecowitt a zero value, then edited the value on the gateway. Cumulus sees the counter increase and records it as rain**.

Actually: I did set the rain on the GW1100 the old values so after a factory reset. I uploaded definitely not zero. That is how my rain value doubled. Then I installed a new GW1100 with zero values and was not able to repair (several times during two days or so, it kept coming back). That is when I posted!

[HansR]

** It isn't quite working as intended, Cumulus is supposed (according to the C1 FAQ) ignore large increments in the rainfall counter. Afaik it only accounts for large decreases, so it is either something C1 does that got missed in the MX code, or C1 never really did it either!

And actually it was a large decrease: all in all it went from 111 to 0 but it keeps adding the 111 to the dayfile (fortunately not the daily log afaics).

[HansR]

OK, everything is corrected. Waiting at least two days - or where today.ini and yesterday.ini no longer show the erroneous amount - stabilised the registration so the deletion of the records from dayfile and the logfile did not trigger the download from Ecowitt anymore. The order of action then is (I think):

1. Remove the lines from the standard logfile(s) which have the erroneous values in them. You lose some data but OK, you should not wait long after detection. This is an important step.
2. Remove the line(s) from dayfile containing the error
3. Run CreateMissing to recreate dayfile
4. Correct the records in the initfiles (use the CMX editors, be aware which records you use! there will be differences)

But: it seems to me CMX needs to ignore irregular amounts + and - (@MArk: you said it worked only for the -) as it is very easy to err with a new Ecowitt GWxxyy and probably with any Ecowitt device where you give the rain balance in the device.

Yes, it was my error: first I upped some 111 mm then regognising the error in correction I downed 111 mm. Life is tough with rain.

[HansR]

Unfortunately I was too fast apparently, after startup I got this (where it should have been 5.9 or something like that):
Forget it, I was looking at the wrong CMX...

[mcrossley]

When you get a large amount of rainfall like this (cleaning gauge, or counter reset, then restore) the thing to do is correct that days rainfall in the editor. Do not edit any ini files. Then wait for the day rollover. The following day you can clean up any other records that have been set (eg rainfall in an hour etc).

If you edit the ini files you will end up chasing your tail unless you have a deep understanding of how MX calculates (and corrects) the rainfall.

[HansR]

When you get a large amount of rainfall like this (cleaning gauge, or counter reset, then restore) the thing to do is correct that days rainfall in the editor. Do not edit any ini files. Then wait for the day rollover. The following day you can clean up any other records that have been set (eg rainfall in an hour etc).

If you edit the ini files you will end up chasing your tail unless you have a deep understanding of how MX calculates (and corrects) the rainfall.

But I think that is exactly what I did (see two posts back). I waited two or three day between the actual error made, I corrected through the editors. The only manual thing I did was changing dayfile and the monthly log. And then CreateMissing.
But still the 117 mmm comes back.

[SamiS]

When you get a large amount of rainfall like this (cleaning gauge, or counter reset, then restore) the thing to do is correct that days rainfall in the editor. Do not edit any ini files. Then wait for the day rollover. The following day you can clean up any other records that have been set (eg rainfall in an hour etc).

If you edit the ini files you will end up chasing your tail unless you have a deep understanding of how MX calculates (and corrects) the rainfall.

But I think that is exactly what I did (see two posts back). I waited two or three day between the actual error made, I corrected through the editors. The only manual thing I did was changing dayfile and the monthly log. And then CreateMissing.
But still the 117 mmm comes back.

Your post says that first you remove the whole lines with wrong data from standard logfiles and that you waited over rollover before making any corrections. Mark says that you should use data editor (not to edit logfiles directly) to remove the bad data before rollover and then after rollover fix only erraneous records via editor. In my eyes there is a great difference between your and Mark’s saying.

[HansR]

Your post says that first you remove the whole lines with wrong data from standard logfiles and that you waited over rollover before making any corrections. Mark says that you should use data editor (not to edit logfiles directly) to remove the bad data before rollover and then after rollover fix only erraneous records via editor. In my eyes there is a great difference between your and Mark’s saying.

Indeed he says. I have always edited the dayfile and logfiles manually (only when needed, I don't do that daily ) with CMX down. Never had an issue. But OK, next time I will do through the editor. @Mark: does that have a data effect other than manual deletion? Point is that multiple line deletion does not work (last time I tried/tested) so have to do that line by line which is OK for the dayfile but can be a nuisance for the logfile.

And now btw I see what has happened: I currently have three GW1100 operating for three different CMX's but all having the same sensors. And i missed the point that they also had that erroneous amount picked up. So I was looking at the wrong CMX. My bad. Having a colour coded CMX interface would be a blessing. I corrected my post above.

To conclude: My operational is OK now, I just have to repair my test CMX's as well and check better which CMX I am looking at
Sorry for the confusion.

[mcrossley]

The correction to "today's" rainfall makes several internal amendments to the calculation (including amending values in the SQLite database). It does not amend the log file for the current day, but uses the correction in the log file at the start of the following day.

It's not very clean and does leave some residual effects to records, that days max rainfall in an hour etc that need manually resolving, but the basic total rainfall for each day should be correct.