Cumulus MX Rainfall Internal Counter

[HansR]

MC: I HAVE SPLIT OFF THIS DISCUSSION ABOUT THE MX INTERNAL RAINFALL COUNTER INTO A NEW TOPIC

The advantage Ecowitt has is that their stations do record weekly data in the consoles and gateways where the start of the week is defined. However CMX does not use this data but collects and collates it own.

Stuart

And above that, CMX is generic so what it offers for one stationtype, it must offer for all stationtypes. Also for the ones that do not have weekly data which means it has to be created by CMX. Feature of one station does not imply that feature on another station.

[SamiS]

I’m not a programmer, but I think that the complexity of the task greatly depends on how CMX now handles rain internally. Basically it shouldn’t be too hard to query and sum up the cumulative daily rain amounts from dayfile backwards from today to last sunday or monday depending on the user setting. To lessen cpu load and disk i/o this could be only on startup and after that the calculation logic could happen in memory. But as said, I do not know the internals of CMX, so this is speculation.

[HansR]

I’m not a programmer, but I think that the complexity of the task greatly depends on how CMX now handles rain internally. Basically it shouldn’t be too hard to query and sum up the cumulative daily rain amounts from dayfile backwards from today to last sunday or monday depending on the user setting. To lessen cpu load and disk i/o this could be only on startup and after that the calculation logic could happen in memory. But as said, I do not know the internals of CMX, so this is speculation.

No doubt it will make correcting rain even more complex.
Might be a reason to review/overhaul the whole rain system (I did not say that )

[broadstairs]

First choice for me would be an ever increasing counter, but the bug bear is always those times users/stations reset counters or subtract from crazy counts caused by hardware faults or cleaning rain tippers.

Surely that is tricky because totals have to be reset and month and year end and with some locations having very large rain amounts you counter could get quite large. Every station I suspect does actually reset its counters at least at month and year end!

Stuart

[mcrossley]

Surely that is tricky because totals have to be reset and month and year end and with some locations having very large rain amounts you counter could get quite large. Every station I suspect does actually reset its counters at least at month and year end!

I think the counter will cope with amount of rainfall in the age of the solar system. And yes, the counters from the stations reset periodically (and randomly) and MX tries to cope along with user applied corrections. But are there better ways of doing this?

[HansR]

Wouldn't it be a better idea if somebody would propose a new design (including user correction method, crash sturdiness and other things) for the rain counter system and have all shoot holes in it? Rather than this pluŕiformous design discussion?

I'll be absent for some two weeks

[freddie]

I think the counter will cope with amount of rainfall in the age of the solar system. And yes, the counters from the stations reset periodically (and randomly) and MX tries to cope along with user applied corrections. But are there better ways of doing this?

I think it is a no-brainer for MX to have an internal counter that is completely independent of the station, and that is only updated by rainfall reports (including reading archive records) from the station and user corrections. It should start at zero from the time of earliest data. It should be persisted frequently to guard against crashes or sudden power loss. Each reading persisted should have an associated timestamp. Historic edits to rainfall data would be easy if there are timestamps, with the associated change propagated forward simply to the current data timestamp.

What MX really needs is an internal database. It cries out for it, and would make data edits (and the promulgation of those edits to external sources) consistent and simple. By external sources, I mean user's own databases and any logging done to files.

[freddie]

Wouldn't it be a better idea if somebody would propose a new design (including user correction method, crash sturdiness and other things) for the rain counter system and have all shoot holes in it? Rather than this pluŕiformous design discussion?

@hansr good designs come from much discussion - it helps to capture all the use cases. I would very much encourage contributions to this topic.

[HansR]

@freddie: agreed but a rational starting point will give direction. Discussion will follow. That's better than a 360 degree discussion. But anyway... good subject to talk about

[broadstairs]

I think it is a no-brainer for MX to have an internal counter that is completely independent of the station, and that is only updated by rainfall reports (including reading archive records) from the station and user corrections. It should start at zero from the time of earliest data. It should be persisted frequently to guard against crashes or sudden power loss. Each reading persisted should have an associated timestamp. Historic edits to rainfall data would be easy if there are timestamps, with the associated change propagated forward simply to the current data timestamp.

What MX really needs is an internal database. It cries out for it, and would make data edits (and the promulgation of those edits to external sources) consistent and simple. By external sources, I mean user's own databases and any logging done to files.

That's fine for someone starting from scratch with CMX. My concern on that database would be how would it cope with someone like me wanting to import 17 years of previous data from another software when moving to CMX?

Stuart

[freddie]

That's fine for someone starting from scratch with CMX. My concern on that database would be how would it cope with someone like me wanting to import 17 years of previous data from another software when moving to CMX?

Stuart

It's all about the timestamps Quoting myself:

Historic edits to rainfall data would be easy if there are timestamps, with the associated change propagated forward simply to the current data timestamp

This can be applied to imports of old data. If you have monthly files then you have timestamps for rainfall events. Therefore you could in theory move the start point of your timer backwards on the timeline and apply retrospective changes forward in time. It sounds complex but is achievable with an internal database and an import utility that will create database records by reading the existing monthly files.

[Polin79]

I'll add the idea to the list of things to look at ...

MX does already allow for defining when the rain year starts.

@freddie - or anyone - want to have a go at defining a revised rain counting system? I'd love it to be simplified! The main thing is finding something that is workable with all stations.

First choice for me would be an ever increasing counter, but the bug bear is always those times users/stations reset counters or subtract from crazy counts caused by hardware faults or cleaning rain tippers.

I meant the same thing as per the rain year start, where the user can define the week start from sunday or from monday. I'm undestanding that it's complicated because of the CMX data store

[HansR]

I think it is a no-brainer for MX to have an internal counter that is completely independent of the station, and that is only updated by rainfall reports (including reading archive records) from the station and user corrections. It should start at zero from the time of earliest data. It should be persisted frequently to guard against crashes or sudden power loss. Each reading persisted should have an associated timestamp. Historic edits to rainfall data would be easy if there are timestamps, with the associated change propagated forward simply to the current data timestamp.

What MX really needs is an internal database. It cries out for it, and would make data edits (and the promulgation of those edits to external sources) consistent and simple. By external sources, I mean user's own databases and any logging done to files.

The internal counter with timestamp sounds logical enough.

But the internal database? Apart from the efforts by Mark in some alpha testing some years ago, where he made an SQLite database where all data went. For my, then three year or so, dataseries that already became huge. But you might look into some timeseries database without relational/SQL access. That might indeed lead to something.

Apart from that, binary storage could be promising. You would also require some user API data acces to actually access (and possibly modify?) those data using some form of block or record access or even retrieving the whole series one way or another. Unless you would not want users or actually do data analysis themselves which would make CMX less popular I guess.

A non-ascii (binary) internal rain counter would be logical as this number typically must be protected from direct user interference. And yes, the derivatives (rainfall on a day, per month, year etc.... should be easily - and automatically - modifiable when changing the internal counter on some time.

You might think of some hybrid data storage: rain in a (binary) timeseries db while maintaining the ascii db for easy access (the access of the inifiles for records I count outside the db storage). CMX could then ignore the ascii rain values currently in the ascii db and only use the binary. Hybrid is always confusing though and the idea might better be ignored.

That's fine for someone starting from scratch with CMX. My concern on that database would be how would it cope with someone like me wanting to import 17 years of previous data from another software when moving to CMX?

Import data would be some additional tool for that database of course. That would not be a user problem. And importing should only be permitted for data outside the period already in that binary CMX database: when importing, the existing data takes precedence. I do not see a problem there (for the user that is).

In general a new internal database sounds great - given some size and acces speed constraints - but it would require a rebuild of CMX - almost? - from the ground up.

[...] but the bug bear is always those times users/stations reset counters or subtract from crazy counts caused by hardware faults or cleaning rain tippers.

Hardware faults and power surges etc... are beyond a requirement as they would need some additional hardware to the weathercomputer. But they may need a good and easy correction facility (also for other sensors).

[mcrossley]

Correcting invalid rainfall is non-trivial. To correct the current days rainfall to remove spurious tips, MX (and C1) amend the counter value recorded for the start-of-day to subtract the extra amount, and that fixes the total going forward. That makes the total that goes into the day file at the end of day correct.

What that does not do is amend all the records in the interval records that have "bad" counter values, and there could be many of them with 1 minute logging. If the counter is too be truly useful you should be able to use all the values from the interval data. But to do that is going to be a manual process afaics, any program will have no idea which increments during that day are real and which are spurious for any automated form of correction.

The station resetting whatever counter MX is using is already catered for, MX ignores big changes in the counter (up or down) between readings until they have been read three times to confirm them, then makes the corresponding adjustment to it's internal counter to match. It also anticipates year end resets for annual counters.

And remind never to compose long posts on my phone again. Autocorrect and pecking doors errors are too painful