ET and Monthly log file

[BCJKiwi]

Have noticed an apparent anomaly in the monthly log file.

Don't know if this is the Davis console playing up or something in Cumulus.

The first record for the day (midnight in the log) still contains the total ET from the day before and the corresponding Annual ET value.

The second record reverts to the new value for the new day and the Annual ET value is increased appropriately.

The other data fields that are carrying running totals for the day (e.g. total rainfall and sunshine hours so far today) are zeroed at the first (midnight) entry for the day.

Any suggestions as to what may be causing this?

Thanks.

[steve]

You could argue that the ET one is 'correct' and it is the others which shouldn't have zeroed in the midnight entry, but I agree that either way, it should really be consistent. The new version of Cumulus is consistent in this respect (and the midnight entry contains the last figures of the day). I may have a look at making the current version consistent, but it is not code that I would be comfortable tinkering with, as it is already very complicated, having to cater for a large number of different situations.

[BCJKiwi]

OK,
It is not a pressing issue as the subsequent records seem to be OK and it does not affect anything long term that I can see.

[colinpb]

I noticed the same thing but I assume Davis use 00.00 as the end of day and enter ET, whereas Cumulus uses 00.00 as the start of the day so no ET. However for me the time difference is unimportant as I would never use the daily running total as it has a bad habit of zeroing during the day. This only happens rarely, but in October it happened 6 times resulting in Cumulus reporting zero ET in the dayfile (also appears in monthly log file). To find todays ET I would always use the running annual total.

As an aside, another oddity of the console software is negative ET. The inference from using version 3.12 was that this had been changed to show zero if negative. This is not happening as Cumulus still reports negative ET

Colin

[steve]

There are bugs in the Davis DLL with the ET figure (or at least, I seem to remember that it's the DLL rather than the console firmware). I could change Cumulus to show zero instead of a negative value, but the zero would be incorrect anyway, and at least with a negative value it is obvious that the value is invalid.

[BCJKiwi]

The way it is now, web pages which indicate daily totals are zeroed at midnight - i.e. when the website refreshes a page at midnight, the day so far totals are all zero except ET. ET still shows the previous days total until ET is updated in Cumulus to the new starting value for the day and that is uploaded to the website and the page is subsequently refreshed.

So the way the other records are done (zeroed at midnight) is correct, and, ET is not.
i.e. the midnight record should be zeroed as it is now.

[BCJKiwi]

Also meant to mention that the time of 00:00 is associated with the first record of the new date so should one would expect it to show the lowest number for the new day.

[steve]

One argument for having the midnight entry in the log files show the final values of the day that's ending rather than the initial values of the day that's beginning, is that otherwise, the final values for the day never appear in the log file, only in dayfile.txt.

[mcrossley]

Using 00:00 for the last entry of the day makes querying databases a bit awkward, you cannot simply group by date, you would have to make the grouping by date minus 1 minute.

[steve]

Yes; and Cumulus does regard the 00:00 entry as the first of the day rather than the last anyway. I think that the fact that the final totals don't appear in the log files isn't really an issue as they are in dayfile.txt, which is one of the reasons that file exists in the first place.

Having looked at the code, I've realised that it's actually trivial and low risk to set the ET to zero explicitly so it's consistent with rainfall etc.

And I guess I should look at changing the new code so that it's consistent with the old code...

[freddie]

Yes; and Cumulus does regard the 00:00 entry as the first of the day rather than the last anyway.

I think one of the issues here is that you are referencing accumulations/extremes and instantaneous/point values in the same way. I would argue the following:

• At the 00:00 data time for accumulations and extremes, the values should be part of the day just ended.
  At the 00:00 data time for instantaneous values, the values should be part of both days!

Obviously you can't really have a value that is part of both days. This is only really a problem if you have a change in value as the day threshold is crossed. But that should be picked up by the "max/min since last reading". For that reason, I think the 00:00 values should be attributed to the previous day. If you attribute them to the day just started then you lose data (any extremes since the previous reading would have to be discarded as they are actually for the previous day, and totals - which may included data from events since the previous reading - are reset to zero).

I think it's best to discard the idea of instantaneous data readings, and store the data as a "period" - i.e. with a data start date and end date - and have your day's threshold coincident with a data reading end date/time. Things that require instantaneous data points (graphs) can still be done with this approach - just use the value from the end of the period. As long as the length of the period is small (a second or so) and consistent then this approach works.

[steve]

I think the 00:00 values should be attributed to the previous day. If you attribute them to the day just started then you lose data (any extremes since the previous reading would have to be discarded as they are actually for the previous day, and totals - which may included data from events since the previous reading - are reset to zero).

They aren't discarded, though; they go in dayfile.txt.

I think it's best to discard the idea of instantaneous data readings, and store the data as a "period" - i.e. with a data start date and end date

I am not proposing to change the way Cumulus stores its data, so this is not an option. When I add SQL database storage alongside the plain text files, we can perhaps look at how to do it better for that. The database which Cumulus 2 used worked in a somewhat similar way to what you're suggesting - each entry effectively covered a "period".

[freddie]

They aren't discarded

Sorry, I should've said "stored elsewhere".

I am not proposing to change the way Cumulus stores its data, so this is not an option. When I add SQL database storage alongside the plain text files, we can perhaps look at how to do it better for that. The database which Cumulus 2 used worked in a somewhat similar way to what you're suggesting - each entry effectively covered a "period".

Before I discovered Cumulus, I had written my own (Java and MySql) solution for extracting and storing data from a Vantage Pro 2 station, as I didn't get on with the Davis software. So I've been there and done this. I never got round to the display side of things, though, so binned it (not literally) and used Cumulus instead.