BST to GMT Experiment

[sfws]

THIS IS NOT A REQUEST FOR ASSISTANCE - just sharing an experiment I did.

My computer has been poorly all this year for reasons unconnected with Cumulus. Yesterday it gave the blue screen of death, and I rebuilt it from a several days old backup finishing just after midnight. Knowing the clock change was due soon, I decided on a Cumulus related experiment.
1. I started Cumulus (the 1083 build of 1.9.4 beta) and left it running.
2. When I got up for breakfast looking at the trend graphs, there was evidence of extra variation between 1am and 2am, telling me the graphs included both BST and GMT measurements in the 1am to 2am plotting period.

-----------------
ORIGINAL 3.
Looking at the oct2013.txt monthly log, only one set of readings were stored every ten minutes from 1am to 2am, subsequent analysis revealed these related to GMT, although looking in the diags had led me to expect to see 1am to 2am BST followed by 1am to 2am GMT in the monthly log. In my mind I was recalling https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=8513 that concluded both are logged at end of DST.

REVISED(29 OCT 2013) 3.
Looking at the oct2013.txt monthly log, looking in the diags had led me to expect to see 1am to 2am BST followed by 1am to 2am GMT in the monthly log. In my mind I was recalling https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=8513 that concluded both are logged at end of DST. FOLLOWING EXCHANGES IN THIS THREAD I CAN CONFIRM CUMULUS BEHAVES AS EXPECTED
-----------------

4. Cumulus was stopped and my computer switched off while I was out for the day.
5. On my return this afternoon, I backed up my data sub-folder into a newly created sub-folder (BST to GMT).
6. Next I selected the backup folder created when I started Cumulus prior to the clock change, copied the ini files and oct2013.txt files from that backup across, and restarted Cumulus so it read the Fine Offset memory blocks to catch up.
7. I had changed the console clock yesterday because my model does not have radio controlled clock.

8. So Cumulus started at then (GMT) time and worked backwards through memory blocks assigning them to GMT time-stamps at 15 minute intervals (the station logging interval that does not exactly fall on quarter hours) but it actually finished an hour after the end time X in the back-up Oct2013.txt because it finished at just after X GMT did not include the extra hour needed to finish just after X BST.
EDIT 29 OCT - this is in line with what the Wiki now says about DST and Fine Offset

9. After a while, I closed Cumulus again.
10. Using 'Calc', I compared the two monthly log files and created a combined replacement one merging them. Essentially, for the one from the station logger, I worked backwards and added one hour to the timestamps between 1am in its monthly log file until just after the time X in the today.ini and oct2013.txt from the backup sub-folder (so creating an hour gap in the data after X). This gave me reading forwards; a change to 1am (GMT on Cumulus running log), after the edited Fine Offset times inbetween 1am (BST) and 2am (BST) that merged with the cumulus running monthly log for those hours, then readings from both logs until another 2am (GMT) and later times from the Cumulus running log. Since the Cumulus running measurements logged fitted inbetween the Fine Offset logged values for both hours, it was easy to confirm that I now had both sets of measurements with correct time-stamps according to computer clock.
11. Thus I have ended up with a monthly log file that runs until 2am then jumps back to 1am and continues.
12. Cleverly, Steve has released 1.9.4 as a new build today, so I've installed that in a different location to the beta I was using (reversing my procedure when I first swapped to a beta), enabling me to test running Cumulus b.1085 carrying on with the new monthly log with the option to go back to the previous had it all gone wrong.
13. My rollover is 9am GMT, so I deleted the record for the previous meteorological day in dayfile.txt in the new location, and used 'create missing' to recreate yesterday from the monthly log. Comparing the old and new versions of the last row in the daily summary log I am now convinced that I have all the 1am to 2am GMT and all the 1am to 2am BST figures now included (including the coldest part of the night for example).

Since the experiment I have seen https://cumulus.hosiene.co.uk/viewtopic.php?f=6&t=10780 and so I thought I would share my Cumulus 1.9.4 with Fine Offset WH1081 station experience.

[steve]

If Cumulus was running, then regardless of the station type, and assuming your PC changed the time correctly at 0200 BST, then I would expect to see two lots of data in the logs with timestamps between 0100 and 0200 (just as you saw with the graphs). Are you sure that isn't what you had? It's hard to see how it could be any other way, assuming your PC clock did change at the right time. And the diags file should confirm this. And it's what I see in my logs.

If Cumulus wasn't running, because Fine Offset stations don't timestamp the data, then I would expect the logs to appear as if nothing had happened, because Cumulus would simply go back far enough through the logger entries to go back for what it believed to be the correct number of hours. Whereas in fact it didn't really go back far enough and effectively the first hour of data after you closed Cumulus down last night would have been ignored.

[sfws]

If Cumulus wasn't running, because Fine Offset stations don't timestamp the data, then I would expect the logs to appear as if nothing had happened, because Cumulus would simply go back far enough through the logger entries to go back for what it believed to be the correct number of hours.

Agreed, and my initial typing used most of those words for step 8, but on reviewing them I changed it to how it currently reads.

effectively the first hour of data after you closed Cumulus down last night would have been ignored

Agreed, my step 10 says this in my words, but to use alternate words.
1. The merged Oct2013.txt starts with the one from the backup sub-folder - this was same as the start of both the Cumulus running and station logger derived logs.
2. The next hour was missing in the station logger version after I added one hour to the entries (rows) from that time until just before 1am.
3. So the missing hour was plugged from the Cumulus left running log that matched up for its next few hours to the station logger version after its timestamps had been adjusted.

assuming your PC changed the time correctly at 0200 BST

I was asleep at time, but my matching of the two logs confirmed this. That is not to say that my PC is working perfectly, I am still noticing problems with some other software and intend to re-visit a local computer repair place when I have some spare time.

I would expect to see two lots of data in the logs with timestamps between 0100 and 0200 (just as you saw with the graphs). Are you sure that isn't what you had?

The wind is now increasing rapidly here, so I will stop now and answer this when I next have some free time.

[sfws]

The wind is now increasing rapidly here

Actually, the maximum gust stayed below that recorded last March, so there were no problems on my property. However, there were several trees down in nearby public areas and I conducted a risk assessment for a public walk I was involved with this morning. Sorry for the delay in completing my reply.

you saw with the graphs

Yes, I did not back those up, so I cannot share the evidence, but you will know what it looks like when alternating between the higher and lower values of one hour apart overlapped onto same time.

I would expect to see two lots of data in the logs with timestamps between 0100 and 0200. Are you sure that isn't what you had?

That was my expectation, but what I had in the log while Cumulus left running was as my step 3 said:
27/10/13 00:50 14.4 99 14.3 1.9 15.9 193 0 0.3 29.41 1003.8 117.5 58 5.4 14.4 14.4 0 0 0 0 15.2 0 0 180 0 0
27/10/13 01:00 14.3 99 14.2 5.4 8.3 170 0 0.3 29.41 1003.8 117.5 58 7.6 14.3 14.3 0 0 0 0 13.9 0 0 225 0 0
27/10/13 01:10 14.2 99 14.1 1.9 12.1 224 0 0.3 29.4 1003.8 117.5 58 5.4 14.2 14.2 0 0 0 0 14.9 0 0 270 0 0
27/10/13 01:20 14.2 99 14.1 1.9 8.3 228 0 0.3 29.41 1003.8 117.4 58 4.5 14.2 14.2 0 0 0 0 14.9 0 0 270 0 0
27/10/13 01:30 14.2 99 14.1 0 8.3 191 0 0.3 29.41 1003.8 117.4 58 1.6 14.2 14.2 0 0 0 0 15.5 0 0 225 0 0
27/10/13 01:40 14.1 99 14 0 5.4 148 0 0.3 29.4 1003.8 117.3 58 1.6 14.1 14.1 0 0 0 0 15.3 0 0 225 0 0
27/10/13 01:50 13.9 99 13.8 0 4.5 216 0 0.3 29.4 1003.8 117.2 59 0.7 13.9 13.9 0 0 0 0 15.1 0 0 225 0 0
27/10/13 01:00 13.7 99 13.6 0 12.1 207 3.6 0.6 29.38 1004.1 117.2 58 1.6 13.7 13.7 0 0 0 0 14.8 0 0 360 0 0.3
27/10/13 01:10 13.3 99 13.2 0 12.1 220 3.6 0.6 29.39 1004.1 117.1 59 0.7 13.3 13.3 0 0 0 0 14.3 0 0 270 0 0.3
27/10/13 01:20 13 99 12.9 0 8.3 214 3.6 0.6 29.39 1004.1 117.1 59 0.7 13 13 0 0 0 0 13.9 0 0 270 0 0.3
27/10/13 01:30 12.9 99 12.8 0 8.3 202 7.2 0.9 29.39 1004.4 117 59 1.6 12.9 12.9 0 0 0 0 13.8 0 0 135 0 0.6
27/10/13 01:40 12.7 99 12.6 1.9 6.9 226 7.2 0.9 29.39 1004.4 117 59 6.9 12.7 12.7 0 0 0 0 12.9 0 0 315 0 0.6
27/10/13 01:50 12.5 99 12.4 0.8 9.2 215 7.2 0.9 29.39 1004.4 117 59 4.5 12.5 12.5 0 0 0 0 13 0 0 225 0 0.6
27/10/13 02:00 12.2 99 12.1 0.8 8.3 260 7.2 0.9 29.39 1004.4 116.9 59 4.5 12.2 12.2 0 0 0 0 12.6 0 0 270 0 0.6
27/10/13 02:10 12 99 11.9 0 4.5 252 0 0.9 29.38 1004.4 116.9 59 0.7 12 12 0 0 0 0 12.6 0 0 248 0 0.6

assuming your PC clock did change at the right time. And the diags file should confirm this.

The diags contain the creating of the backup files I used in my step 6:
27/10/2013 00:04:13.792 : UseDataLogger=1
27/10/2013 00:04:13.792 : Station type: Fine Offset
27/10/2013 00:04:13.792 : COM port = 0
27/10/2013 00:04:13.792 : Short date format: 27/10/2013
27/10/2013 00:04:13.792 : Long date format: 27 October 2013
27/10/2013 00:04:13.792 : Short time format: 00:04
27/10/2013 00:04:13.792 : Long time format: 00:04:13
27/10/2013 00:04:14.241 : Creating backup folder S:\Cumulus_beta\backup\20131027000413\
27/10/2013 00:04:14.461 : 622 web tags initialised
27/10/2013 00:04:14.464 : 00:04:14 EWUSB: Creating EW USB

They do confirm the clock change was at 2am BST as 1:50:01 is followed by 01:00:00, the key lines are:
27/10/2013 01:50:01.003 : Writing today.ini, LastUpdateTime = 27/10/2013 01:50:00 raindaystart = 1003.5 rain counter = 1003.79998779297
27/10/2013 01:50:01.003 : Latest reading: D150: Data: 0A 3B AC 00 63 8B 00 70 26 00 03 00 0A 12 0D 00
27/10/2013 01:00:00.756 : Writing today.ini, LastUpdateTime = 27/10/2013 01:00:00 raindaystart = 1003.5 rain counter = 1004.09997558594
27/10/2013 01:00:00.756 : Latest reading: D160: Data: 05 3A AC 00 63 89 00 6A 26 00 07 00 00 13 0D 00
and some lines later are entries typical of those that continued until I closed Cumulus:
27/10/2013 01:50:00.211 : Writing today.ini, LastUpdateTime = 27/10/2013 01:50:00 raindaystart = 1003.5 rain counter = 1004.40002441406
27/10/2013 01:50:00.211 : Latest reading: D190: Data: 0A 3B AA 00 63 7D 00 6D 26 03 14 00 0A 14 0D 00
27/10/2013 02:00:00.850 : Writing today.ini, LastUpdateTime = 27/10/2013 02:00:00 raindaystart = 1003.5 rain counter = 1004.40002441406
27/10/2013 02:00:00.850 : Latest reading: D1A0: Data: 05 3B A9 00 63 7A 00 6C 26 03 14 00 0C 14 0D 00

It's hard to see how it could be any other way

In conclusion the diags file provides the evidence is that Cumulus was operating the same throughout the time it was running, there is nothing to show why the monthly log missed the hour, and as it is the GMT logs that are stored, I can only guess that the monthly log somehow overwrote the records for 1am to 2am BST with the GMT ones.

And it's what I see in my logs.

Glad both BST and GMT are in your logs and only I ended up having to insert a missing hour.
(Ends).

[steve]

I would expect to see two lots of data in the logs with timestamps between 0100 and 0200. Are you sure that isn't what you had?

That was my expectation, but what I had in the log while Cumulus left running was as my step 3 said:
27/10/13 00:50 14.4 99 14.3 1.9 15.9 193 0 0.3 29.41 1003.8 117.5 58 5.4 14.4 14.4 0 0 0 0 15.2 0 0 180 0 0
27/10/13 01:00 14.3 99 14.2 5.4 8.3 170 0 0.3 29.41 1003.8 117.5 58 7.6 14.3 14.3 0 0 0 0 13.9 0 0 225 0 0
27/10/13 01:10 14.2 99 14.1 1.9 12.1 224 0 0.3 29.4 1003.8 117.5 58 5.4 14.2 14.2 0 0 0 0 14.9 0 0 270 0 0
27/10/13 01:20 14.2 99 14.1 1.9 8.3 228 0 0.3 29.41 1003.8 117.4 58 4.5 14.2 14.2 0 0 0 0 14.9 0 0 270 0 0
27/10/13 01:30 14.2 99 14.1 0 8.3 191 0 0.3 29.41 1003.8 117.4 58 1.6 14.2 14.2 0 0 0 0 15.5 0 0 225 0 0
27/10/13 01:40 14.1 99 14 0 5.4 148 0 0.3 29.4 1003.8 117.3 58 1.6 14.1 14.1 0 0 0 0 15.3 0 0 225 0 0
27/10/13 01:50 13.9 99 13.8 0 4.5 216 0 0.3 29.4 1003.8 117.2 59 0.7 13.9 13.9 0 0 0 0 15.1 0 0 225 0 0
27/10/13 01:00 13.7 99 13.6 0 12.1 207 3.6 0.6 29.38 1004.1 117.2 58 1.6 13.7 13.7 0 0 0 0 14.8 0 0 360 0 0.3
27/10/13 01:10 13.3 99 13.2 0 12.1 220 3.6 0.6 29.39 1004.1 117.1 59 0.7 13.3 13.3 0 0 0 0 14.3 0 0 270 0 0.3
27/10/13 01:20 13 99 12.9 0 8.3 214 3.6 0.6 29.39 1004.1 117.1 59 0.7 13 13 0 0 0 0 13.9 0 0 270 0 0.3
27/10/13 01:30 12.9 99 12.8 0 8.3 202 7.2 0.9 29.39 1004.4 117 59 1.6 12.9 12.9 0 0 0 0 13.8 0 0 135 0 0.6
27/10/13 01:40 12.7 99 12.6 1.9 6.9 226 7.2 0.9 29.39 1004.4 117 59 6.9 12.7 12.7 0 0 0 0 12.9 0 0 315 0 0.6
27/10/13 01:50 12.5 99 12.4 0.8 9.2 215 7.2 0.9 29.39 1004.4 117 59 4.5 12.5 12.5 0 0 0 0 13 0 0 225 0 0.6
27/10/13 02:00 12.2 99 12.1 0.8 8.3 260 7.2 0.9 29.39 1004.4 116.9 59 4.5 12.2 12.2 0 0 0 0 12.6 0 0 270 0 0.6
27/10/13 02:10 12 99 11.9 0 4.5 252 0 0.9 29.38 1004.4 116.9 59 0.7 12 12 0 0 0 0 12.6 0 0 248 0 0.6

I'm confused now, because what I can see there is as I would expect, two lots of data with timestamps between 0100 and 0200. What am I not understanding?

[sfws]

Apologies.
It seems my brain was changing what my eyes saw. The pasting in my last post was from the file in my step 5. Reading it normally looking at the full file working downwards, I didn't see the repeat however many times I looked (whether using the cumulus viewer or viewing externally), I think my brain must be deciding wrongly where to restart scanning as I move to next line and skipping the repeated lines.

If I now focus in on an individual value in each single line and then look left to the time, or look at that shorter extract I posted I can see the time repeats.

By the way I approve of the way the log viewer in the new release makes it clear it is not for viewing the dayfile! Seeing all the forum discussions on this I wondered if the menu item 'Data logs' needed to include the word 'monthly', but I think your solution is better.

[sfws]

I take slight comfort from knowing I am not the only person who makes a fool of himself on the forum.
Anyway to complete my experiment I have run Easyweather instead of running Cumulus so it processed observations from the console for the period including the end of BST. It may be nobody is interested in this?

I have pasted in an extract from the database for the same period around clock change as in my earlier posting.
5832, 2013-10-29 07:43:38, 2013-10-27 00:47:13, 15, 58, 17.5, 99, 14.5, 14.4, 14.5, 984.2, 995.8, 1.7, 2, 4.1, 3, 14, NW, 3346, 1003.8, 0.0, 0.0, 0.3, 19.8, 44.7, 84.0, 0, 0, 0, 0, 0, 0, 0, 0, 00D100, 0F 3A AF 00 63 91 00 72 26 11 29 00 0E 12 0D 00 ,

5833, 2013-10-29 07:43:38, 2013-10-27 01:02:13, 15, 58, 17.5, 99, 14.4, 14.3, 14.4, 984.3, 995.9, 0.3, 1, 1.0, 1, 2, NE, 3346, 1003.8, 0.0, 0.0, 0.3, 19.8, 44.7, 84.0, 0, 0, 0, 0, 0, 0, 0, 0, 00D110, 0F 3A AF 00 63 90 00 73 26 03 0A 00 02 12 0D 00 ,

5834, 2013-10-29 07:43:38, 2013-10-27 01:17:13, 15, 58, 17.5, 99, 14.3, 14.2, 14.3, 984.2, 995.8, 1.4, 1, 2.4, 2, 6, SE, 3346, 1003.8, 0.0, 0.0, 0.3, 19.8, 44.7, 84.0, 0, 0, 0, 0, 0, 0, 0, 0, 00D120, 0F 3A AF 00 63 8F 00 72 26 0E 18 00 06 12 0D 00 ,

5835, 2013-10-29 07:43:38, 2013-10-27 01:32:13, 15, 58, 17.4, 99, 14.2, 14.1, 14.2, 984.2, 995.8, 1.0, 1, 3.1, 2, 8, S, 3346, 1003.8, 0.0, 0.0, 0.3, 19.8, 44.7, 84.0, 0, 0, 0, 0, 0, 0, 0, 0, 00D130, 0F 3A AE 00 63 8E 00 72 26 0A 1F 00 08 12 0D 00 ,

5836, 2013-10-29 07:43:38, 2013-10-27 01:47:13, 15, 58, 17.3, 99, 14.1, 14.0, 14.1, 984.0, 995.6, 0.3, 1, 1.0, 1, 4, E, 3346, 1003.8, 0.0, 0.0, 0.3, 19.8, 44.7, 84.0, 0, 0, 0, 0, 0, 0, 0, 0, 00D140, 0F 3A AD 00 63 8D 00 70 26 03 0A 00 04 12 0D 00 ,

5837, 2013-10-29 07:43:38, 2013-10-27 01:02:13, 15, 59, 17.2, 99, 13.9, 13.8, 13.9, 984.0, 995.6, 0.0, 0, 0.3, 1, 10, SW, 3346, 1003.8, 0.0, 0.0, 0.3, 19.8, 44.7, 84.0, 0, 0, 0, 0, 0, 0, 0, 0, 00D150, 0F 3B AC 00 63 8B 00 70 26 00 03 00 0A 12 0D 00 ,

5838, 2013-10-29 07:43:38, 2013-10-27 01:17:13, 15, 59, 17.1, 99, 13.3, 13.2, 13.3, 983.9, 995.5, 0.0, 0, 1.0, 1, 14, NW, 3347, 1004.1, 0.3, 0.3, 0.6, 20.1, 45.0, 84.3, 0, 0, 0, 0, 0, 0, 0, 0, 00D160, 0F 3B AB 00 63 85 00 6F 26 00 0A 00 0E 13 0D 00 ,

5839, 2013-10-29 07:43:38, 2013-10-27 01:32:13, 15, 59, 17.1, 99, 12.9, 12.8, 12.9, 984.0, 995.6, 0.3, 1, 1.4, 1, 4, E, 3347, 1004.1, 0.0, 0.3, 0.6, 20.1, 45.0, 84.3, 0, 0, 0, 0, 0, 0, 0, 0, 00D170, 0F 3B AB 00 63 81 00 70 26 03 0E 00 04 13 0D 00 ,

5840, 2013-10-29 07:43:38, 2013-10-27 01:47:13, 15, 59, 17.0, 99, 12.8, 12.7, 12.8, 983.8, 995.4, 0.0, 0, 1.0, 1, 11, SWW, 3348, 1004.4, 0.3, 0.6, 0.9, 20.4, 45.3, 84.6, 0, 0, 0, 0, 0, 0, 0, 0, 00D180, 0F 3B AA 00 63 80 00 6E 26 00 0A 00 0B 14 0D 00 ,

5841, 2013-10-29 07:43:38, 2013-10-27 02:02:13, 15, 59, 16.9, 99, 12.4, 12.3, 12.4, 983.6, 995.2, 0.0, 0, 1.0, 1, 12, W, 3348, 1004.4, 0.0, 0.6, 0.9, 20.4, 45.3, 84.6, 0, 0, 0, 0, 0, 0, 0, 0, 00D190, 0F 3B A9 00 63 7C 00 6C 26 00 0A 00 0C 14 0D 00 ,
Here's the format of the 'easyweather.dat' file taken from the Wiki: 0 - Record no; 1 - Transfer date; 2 - Transfer time; 3 - Reading date; 4 - Reading time; 5 - Interval; 6 - Indoor Hum; 7 - Indoor Temp; 8 - Outdoor Hum; 9 - Outdoor Temp; 10 - dew point; 11 - wind chill; 12 - absolute pressure; 13 - rel pressure; 14 - wind average; 15 - wind average bft; 16 - wind gust; 17 - wind gust bft; 18 - wind direction number; 19 - wind direction text (N, ENE etc, converted to a bearing as an integer);
20 - rain tip counter; 21 - rain total; 22 - rain since last reading; 23 - rain in last hour (used as rain rate); 24 - rain last 24 hours; 25 - rain last 7 days; 26 - rain last 30 days; 27 - rain last year (used as rain 'counter' to determine other totals)

Obviously easyweather.dat has assigned the time when I was running the software to fields 1 and 2 of each database entry as it was read from its console memory (note this time is computer time, although that is approximately what I have set console time to, remember I changed the console time the evening before the clocks changed). The software correctly inserts the time for filling the memory location into each database row at fields 3 and 4, note that it identifies when the clocks change (see second red time) so it is not simply adding 15 (field 5) minutes to previous time. I'm guessing easyweather adds 15 minutes each time but calls some routine that returns the new time making allowance for any clock change.


As a passing comment, Cumulus is obviously designed to be left running. Hence, although it works through the same console memory and adds the equivalent of the field 5 interval time to the cumulus monthly log, it does not when doing such (not expected to be normal Cumulus operation) catch up, make allowance for end/start of DST. Remember how Cumulus works is a matter of historical development sometimes making a compromise to minimise coding for individual station makes.

I'm confused now

I have edited my first post correcting steps 3 and 8, and realise my step 10 is okay, but contradicted the original step3. So I obviously was confused when I first typed that first post.

[steve]

What EW has done there is quite clever, and I suppose Cumulus could do the same. But unless there's some obvious easy algorithm that I haven't thought of, it would mean checking explicitly for DST start and end on every logger download, which is somewhat inefficient.