Recent historical data problem

[logjam]

I'd not seen any problems with the 'recent historical data' figures until this morning when I had to restart CumulusMX. I then noticed that although the times and figures were correct from midnight onwards, the figures before midnight were just repeats of the midnight value.

I checked the logger data, and everything is ok there, but the 'Recent' tags (including <RecentTS>), before midnight, just return the midnight value. I'm only using values for the last 24 hours so I don't know if values later than 24 hours are affected.

[steve]

This was just a 'quick' restart, yes, so no logger data involved from overnight? It sounds like a bug in pre-loading the recent data list from the log files. I'll check the code - could you attach the diags file from the restart, please?

[steve]

I've just realised what the date is. For some reason, it's not going back into the previous month's log when pre-loading the recent data list, that shouldn't be too hard to fix.

[logjam]

I'm regularly backing up the data from the Pi to my laptop so in this instance I checked the logger data by starting Cumulus 1 on the laptop and letting it read the data. (Lazy - I know!) I then just looked at the graphs to see if the data was showing over the past 24 hours. If I had thought to look at the actual files instead I might have realised that the data before and after midnight were on a different month files!

Do you still need the diag file?

[steve]

Yes, please, because having looked at the code it's not obvious what the problem is. I've just restarted mine, and it's loaded data from last month. It seems to be doing the correct thing, so perhaps there's a problem in your July log.

[logjam]

I've zipped them all up. I restarted Cumulus a second time today before I wrote this thread in case it hadn't read the log correctly. It made no difference, but that explains the additional file today.

EDIT: I'm using several historical tags, but as an example if I use <#RecentOutsideTemp h=X>, where X are the numbers 24 to 1, I get these figures generated after processing. (as of 16:45 today)

13.2, 13.2, 13.2, 13.2, 13.2, 13.2, 13.2, 13.2, 12.7, 12.5, 11.6, 11.0, 10.9, 11.1, 11.8, 13.1, 15.3, 16.7, 18.6, 20.0, 19.6, 19.9, 20.7, 21.3

[steve]

2015-08-01 12:29:26.870 Error at line 35154 of data/Jul15log.txt : Constraint
2015-08-01 12:29:26.871 Please edit the file to correct the error
2015-08-01 12:29:30.133 Error at line 674 of data/Aug15log.txt : Constraint
2015-08-01 12:29:30.134 Please edit the file to correct the error

The 'constraint' error occurs when adding items to the recent data database and usually means a duplicate timestamp in the log file. This probably isn't indicative of a real problem, it's just a side effect of the way the Fine Offset logger doesn't timestamp entries. I'll see if I can persuade the database to not complain about duplicate timestamps.

[logjam]

As you predicted there were duplicate entries at those points. I stopped CumulusMX and removed the duplicates, and restarted. Of course the program then found further duplicates and complained about them too. It is hard to spot any duplicates by eye. I'll await developments!

I imagine my immediate problem will be ironed out at midnight tonight as the last of the errors move off the part of the array I'm using - until I re-start the program again, of course.

[steve]

I think you're using a 1-minute logger interval, and so, with a Fine Offset station, you are likely to get a duplicate timestamp pretty much every time you restart Cumulus.

[logjam]

I must admit that logging every minute is not really necessary. Once every 2 or 3 minutes would be just as effective. I'll change that before I next restart.

[logjam]

I just upgraded to 3027. Unfortunately upon restart not only has it 'forgotten' everything before midnight, but also everything after it as well.

In anticipation, I've included the relevant diag file.

btw I did increase the logging interval to 2 minutes using 'setlogger'. EDIT: (Having said that, I notice it is still logging every minute )

[steve]

I just upgraded to 3027. Unfortunately upon restart not only has it 'forgotten' everything before midnight, but also everything after it as well.

Changing from 'Insert' to 'InsertOrReplace' doesn't seem to have had the effect that the documentation promised it would have:

2015-08-05 16:32:07.037 Error at line 748 of data/Aug15log.txt : Constraint

Can you have a look at that line and confirm that it has the same timestamp as a previous entry? Most likely the one immediately before it.

btw I did increase the logging interval to 2 minutes using 'setlogger'. EDIT: (Having said that, I notice it is still logging every minute )

It's logging every two minutes:

16:32:08 Read logger entry for 05/08/2015 16:31:07 address 66B0: 02 25 19 01 37 D1 00 76 27 25 36 00 08 D2 0E 00
16:32:08 Read logger entry for 05/08/2015 16:29:07 address 66A0: 02 25 1A 01 37 D1 00 76 27 11 1B 00 0A D2 0E 00
16:32:08 Read logger entry for 05/08/2015 16:27:07 address 6690: 02 25 1A 01 37 D1 00 75 27 18 25 00 0A D2 0E 00
16:32:08 Read logger entry for 05/08/2015 16:25:07 address 6680: 02 25 1B 01 37 D1 00 76 27 18 29 00 0C D2 0E 00

You have Cumulus set to log every minute, but that's nothing to do with the station logger, or what the SetLogger utility does.

[logjam]

Yes line 748 and 747 have identical time stamps.

[steve]

I'll change it to ignore that error.

[logjam]

I'll look out for that change.

Yes, now I've worked it through, I can see that the console logger interval only determines the rate at which data is placed into its memory, and it will only be apparent to Cumulus for the period when it 'catches up' after a restart. It is Cumulus' own log interval which would need changing.

I'll keep it at 1 min for now at least, since that will ensure a supply of duplicate time stamps, to check the effect of any changes you might make!