Cumulus Support

Just upgraded to b3141 and started up with no problems - all seems to be working just fine. I did scan quickly through the MXDiags file after the restart and saw the following:
I'm sure it's nothing to worry about, but wanted to bring it to your attention.

Nothing to worry about, 3.12 now uses a new temporary database to store recent data. On the first run this does not exist and gets created.

Fab, thanks for the detail. I was sure it was nothing to worry about as everything was running sweet as a nut

A technical question related to your persistent database mentioned in your Updates.txt, that gave the message that Freddie spotted:

A SQLite3 database uses the row number as the primary key in any database table,

When the "clocks go back", there will be an hour of duplicated minutes, will MX (from 3.12.0) now be able to retain all the entries because in some way it makes use of that primary key, or (as I suspect might be the way you have coded it) does MX use the datetime column as its key and therefore overwrite the entries for the past hour?

When the "clocks go forward", obviously we lose an hour of values for the clock times that do not exist, so 3.12.0 does not change the need to be careful with selection of input modification parameters when seeking values on the day involved.

If it was my design, then internally I would use a fixed time zone with no DST and convert on-the-fly when making the data available externally. Something I've been doing for years. When storing and manipulating timeline-based data such as meteorological readings then artificial adjustments such as DST have no place in the storage layer.

freddie wrote: ↑Sun 01 Aug 2021 6:45 am If it was my design, then internally I would use a fixed time zone with no DST and convert on-the-fly when making the data available externally.

That has been suggested many times in this support forum over the years.

Of course, in response, Steve Loft's Cumulus 2 was coded to do just that, using UTC internally, and in log files, just converting to local time zone on outputs.
Steve apparently found it impossible to make it work, I don't recall him explaining why he struggled with the UTC to local time conversions. Perhaps it was because of the complications for rollover processing time or because his C# skills were limited?
Anyway, he abandoned that idea when his MX beta took some ideas from both Cumulus 1 and Cumulus 2.

I have an idea that Mark has, at times, considered making MX incompatible with the legacy by various internal standardisations (with external presentational conversions) to make coding easier. In my opinion MX has already diverged so much from the legacy, I don't think compatibility between them matters any more, especially as it is possible to write one-off conversion utilities.

freddie wrote: ↑Sun 01 Aug 2021 6:45 am Something I've been doing for years. When storing and manipulating timeline-based data such as meteorological readings then artificial adjustments such as DST have no place in the storage layer.

I am sure I recall several other Cumulus users in the past, not just you, saying they had adopted a fixed time-zone, either for easier comparison with official data or because they have an international interest in their weather data.

I think using UTC for all internal storage is a great idea, and I do not see any great reason why it would not work. But...

The issue is the effort involved in converting all the existing code. Then writing all the code required for converting all the various existing data files. Then all the support required hand holding people migrating and getting into problems.

The short answer - yes the new database still has all the issues with DST that the original code had.

Background...

Before this release MX kept recent data (used for the recent graphs etc) in in-memory structures. So if you restarted the graphs reverted to using whatever log interval you used until the in-memory storage buffered eventually held the 7 days or whatever you specified.

For this release I just moved that recent data from in-memory to an on-disk SQLite data base so it persists across runs of MX.

Steve Loft, for anyone interested, mentions this in viewtopic.php?p=100098#p100098.