Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Downloading Archive data shows very large spikes

Discussion specific to Fine Offset and similar rebadged weather stations
Post Reply
GeorgeF
Posts: 13
Joined: Wed 14 Apr 2021 12:41 pm
Weather Station: Davis VP2 + solar sensor
Operating System: Windows 7 Starter

Downloading Archive data shows very large spikes

Post by GeorgeF »

MX sometimes loses communication with the weather station which I handle by shutting down MX then restarting weather station - the reload of archived data seems to load some very bad values which should be totally ignored - the corrupted data of course then corrupts all the various data files and it is a long job to correct them all by hand.

I have attached the logfiles of the latest problem on 14-04-21

Regards George
You do not have the required permissions to view the files attached to this post.
sfws
Posts: 1183
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Chas O, Maplin N96FY, N25FR
Operating System: rPi 3B+ with Buster (full)

Re: Downloading Archive data shows very large spikes

Post by sfws »

I see you are already using spike removal and I thought that should have helped.

I believe a future MX release will handle the loss of communication better, see this topic.
User avatar
mcrossley
Posts: 12686
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Downloading Archive data shows very large spikes

Post by mcrossley »

Thanks, I found an issue with the processing of historic data from FineOffset stations that allowed spikes through. I'll add a fix to the next release.
GeorgeF
Posts: 13
Joined: Wed 14 Apr 2021 12:41 pm
Weather Station: Davis VP2 + solar sensor
Operating System: Windows 7 Starter

Re: Downloading Archive data shows very large spikes

Post by GeorgeF »

Sorry there still seems to be a problem on rainfall as well. I attach the webpage as text data and the logs of the failure. Things which I have noted the last rainfall is on 18/05/21 but 0.4 mm has been recorded today 19/05/21 and the total rainfall would indicate that the oceans levels have probably risen!

Hope you can track down what is going wrong

Regards

George
You do not have the required permissions to view the files attached to this post.
User avatar
mcrossley
Posts: 12686
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Downloading Archive data shows very large spikes

Post by mcrossley »

Your station keeps locking up, and sending odd data. Cumulus is coping the best it can, but with rubbish input you tend to get rubbish output.

On the 18th CMX was started at 19:19, there was no history to read. It started running OK..
Until 19:33, when CMX received bad data that it ignored...

Code: Select all

2021-05-18 19:33:56.835 Ignoring bad data: pressure = 510.5
2021-05-18 19:33:56.835                      offset = 22.6
2021-05-18 19:33:56.836 Ignoring bad data: gust = 407.5
2021-05-18 19:33:56.837 Warning: large difference in rain gauge tip count: 46639
After that from 19:34 onwards there was no response from the station until you stopped CMX at 09:09 in the 19th.

BUT, because the station was locked up and no data coming in, the midnight rollover processing for the 18th never happened.


On the 19th, when you started MX at 09:11, the station was reporting a rain counter value of 211.5. The counter value is read from the previous midnight entry in the dayfile. But because the rollover did not happen on the 18th, the only one it could use was the value from the 17th (which was 211.2), so 0.3mm from the 18th was recorded again on the 19th.

As the normal reading of the station stared, the rain counter jumped up 51816 ticks (15544.8 mm), it then stayed at this value, after 6 readings CMX adjusted it's internal counter to this new value until the log ends at 19:09.


What you need to do when you get a lockup is to perform a rollback to either the previous successful midnight backup, or the last time CMX was started. Then the missing rollovers will be completed using historic data from the station logger.

What might help you would be an option to shutdown CMX if it is in a Data Stopped condition for a period. I'll have a think about adding that capability.

However your station is locking up so frequently I think you need to seriously consider replacing it.
GeorgeF
Posts: 13
Joined: Wed 14 Apr 2021 12:41 pm
Weather Station: Davis VP2 + solar sensor
Operating System: Windows 7 Starter

Re: Downloading Archive data shows very large spikes

Post by GeorgeF »

The lock ups seemed to be related USB data transfers which although the cable is shielded seem to happen at random intervals - This is a known problem with this type of station according to the suppliers - a previous verison did the same a couple of years back. I ran the cumulus version 1.0 during that time and the setting to stop data collection on any bad USB transfer worked for me and such an option would be great.
In relation to the rain counter. The station is set to five minute intervals but the software to 1 min. (A 1 min interval on the station would fill tothe memory too quickly in the event of a hangup). It would appear the bad data would be there for five mins and would eventually clear? Not sure how this works - is the counter reset at midnight? In any event such a large jump in counters is bad data and should also be removed on spike supression? The other supressions seem to be working fine in the latest versions. In fact you have highlighted the warning about the rain counter?
I will try to roll back the data as a work around but a dead stop on data collection would help a lot. What I don't know is if the bad data is actaully in the station memory - can I set flag to see this in a log?

Regards

George
Post Reply