Welcome to the Cumulus Support forum.

Latest Cumulus MX release v3.0.0 build 3050 - 1 May 2019.
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014

See the Wiki for downloading either version.

Very Slow Data Processing

Discussion specific to Fine Offset and similar rebadged weather stations
Post Reply
User avatar
logjam
Posts: 163
Joined: Fri 30 Nov 2012 8:01 pm
Weather Station: Maplin N96GY
Operating System: Raspbian on Raspberry Pi
Location: East Lincolnshire
Contact:

Very Slow Data Processing

Post by logjam » Mon 21 Jan 2013 7:53 pm

Hello

I've been using this software for weeks with no problems.
Today I've noticed that whenever Cumulus communicates with the console it takes about 16 seconds to successfully process each line of information and write the data in the monthly data file. While it does this Cumulus 'freezes' and I can't do anything else with the programme. The main concern is when I catch up the data after Cumulus has been off for a while. It takes 16 seconds to read 1 minute of data. When it finally finishes reading all the data, (which can take hours) it ignores everything since the catch up starts, and that data is not downloaded.

I'm using an FO N96GY with Cumulus 1.9.3 Build 1058
I have tried resetting the console, restarting the computer, rolling back the software etc, but with no change. It seems to have nothing to do with the software version. I'm not having any other problems with USB devices
You do not have the required permissions to view the files attached to this post.

User avatar
steve
Cumulus Author
Posts: 26717
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Very Slow Data Processing

Post by steve » Mon 21 Jan 2013 8:10 pm

The actual download of the data is very quick, so it's not a USB issue. The actual processing of the data is also quick. It's the stuff it does after it's processed the data that is taking 20 seconds - writing the log file, and calculating the trends etc. And it's not just the logger data where this happens, it takes 20 seconds each time it writes an entry to the log files (and does the trends etc) for live data also.

There's not enough in the diags files to see what is taking so much time. Please turn on the debug log, leave it for 5 minutes, and then attach the debug.log file. That might give some more clues.
Steve

User avatar
logjam
Posts: 163
Joined: Fri 30 Nov 2012 8:01 pm
Weather Station: Maplin N96GY
Operating System: Raspbian on Raspberry Pi
Location: East Lincolnshire
Contact:

Re: Very Slow Data Processing

Post by logjam » Mon 21 Jan 2013 8:27 pm

Many thanks Steve

Here is the debug log as requested.
You do not have the required permissions to view the files attached to this post.

User avatar
mcrossley
Posts: 5554
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Stretch Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Very Slow Data Processing

Post by mcrossley » Mon 21 Jan 2013 8:36 pm

Looks like creating the WXNow File takes 20 seconds for some reason?

User avatar
steve
Cumulus Author
Posts: 26717
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Very Slow Data Processing

Post by steve » Mon 21 Jan 2013 8:38 pm

That's narrowed it down a little, but doesn't help much:

2628.009 : 20:19:00 Creating WXNow File
2647.915 : 20:19:20 Updating WOW

There are quite a number of actions between those two, that don't get logged - adding entries to the graphs, a few 'trend' calculations, writing the entry to the log file.

Could you temporarily change the logging interval in Cumulus to 5 minutes, and then run again for a few minutes with the debug log switched on, please? That would at least tell us whether it's writing the log file entry that's causing the delay. Beyond that I'll have to add more trace points to the debug log.
Steve

User avatar
logjam
Posts: 163
Joined: Fri 30 Nov 2012 8:01 pm
Weather Station: Maplin N96GY
Operating System: Raspbian on Raspberry Pi
Location: East Lincolnshire
Contact:

Re: Very Slow Data Processing

Post by logjam » Mon 21 Jan 2013 8:52 pm

The debug log after changing the logging interval
You do not have the required permissions to view the files attached to this post.

User avatar
steve
Cumulus Author
Posts: 26717
Joined: Mon 02 Jun 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: Very Slow Data Processing

Post by steve » Mon 21 Jan 2013 8:58 pm

It's the writing of the log file (or today.ini, which it does at the same time).

1223.718 : 20:48:00 Creating WXNow File
1223.734 : 20:48:00 Updating WOW

But when it writes the log file:

1343.746 : 20:50:00 Creating WXNow File
1363.776 : Creating web graphs
1364.821 : 20:50:21 Updating WOW

I don't think the creation of the web graphs is relevant, as we saw it was taking 20 seconds previously without that.

I'm afraid I have no idea why writing the log files takes 20 seconds; perhaps you have some kind of corruption, or a bad sector on the hard drive? You should perhaps run some disk checks.
Steve

User avatar
logjam
Posts: 163
Joined: Fri 30 Nov 2012 8:01 pm
Weather Station: Maplin N96GY
Operating System: Raspbian on Raspberry Pi
Location: East Lincolnshire
Contact:

Re: Very Slow Data Processing

Post by logjam » Mon 21 Jan 2013 9:06 pm

Many thanks. That gives me something to work on. Come to think about it I did have problems starting Windows yesterday evening. . . . . . . . . :oops:

User avatar
logjam
Posts: 163
Joined: Fri 30 Nov 2012 8:01 pm
Weather Station: Maplin N96GY
Operating System: Raspbian on Raspberry Pi
Location: East Lincolnshire
Contact:

Re: Very Slow Data Processing

Post by logjam » Mon 21 Jan 2013 9:39 pm

That was the problem! I've now corrected the hard drive corruption and all is working as it was. :D

Many thanks for the help.

Post Reply