Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.11.4 (build 3133) - 25 May 2021 (please see announcement regarding releases since 3.5.0)
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

sql catch up

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
jlmr731
Posts: 187
Joined: Sat 27 Aug 2016 12:11 am
Weather Station: Davis vantage pro 2
Operating System: Debian
Location: Youngstown, Ohio
Contact:

sql catch up

Post by jlmr731 »

Did a reboot of system and after starting MX i see missing entry's for monthly sql table 11:44 to 11:52 the entrys between them are missing (reboot time)
Just going to paste MXDiag as they are not large and i see sql catch up listed

Code: Select all

2021-05-14 11:50:45.804  ========================== Cumulus MX starting ==========================
2021-05-14 11:50:45.805 Command line: /home/jeff/CumulusMX/CumulusMX.exe 
2021-05-14 11:50:45.805 Cumulus MX v.3.11.1 build 3130
2021-05-14 11:50:45.806 Platform: Unix
2021-05-14 11:50:45.806 OS version: Unix 4.19.0.16
2021-05-14 11:50:45.806 Mono version: 5.18.0.240 (Debian 5.18.0.240+dfsg-3 Wed Apr 17 16:37:36 UTC 2019)
2021-05-14 11:50:45.806 Current culture: English (United States)
2021-05-14 11:50:45.806 Directory separator=[/] Decimal separator=[.] List separator=[,]
2021-05-14 11:50:45.807 Date separator=[/] Time separator=[:]
2021-05-14 11:50:45.807 Standard time zone name:   EST
2021-05-14 11:50:45.807 Daylight saving time name: EDT
2021-05-14 11:50:45.807 Daylight saving time? True
2021-05-14 11:50:45.807 5/14/2021 11:50:45 AM
2021-05-14 11:50:45.820 Reading Cumulus.ini file
2021-05-14 11:50:45.823 AvgSpdMins=10 AvgSpdTime=00:10:00
2021-05-14 11:50:45.824 Cumulus start date: 8/20/2016
2021-05-14 11:50:45.860 No start-up PING
2021-05-14 11:50:45.860 No start-up delay - disabled
2021-05-14 11:50:45.863 Data path = data/
2021-05-14 11:50:45.880 BackupData: Creating backup folder backup/20210514115045/
2021-05-14 11:50:45.902 Created backup folder backup/20210514115045/
2021-05-14 11:50:45.902 Debug logging is disabled
2021-05-14 11:50:45.902 Data logging is disabled
2021-05-14 11:50:45.902 FTP logging is disabled
2021-05-14 11:50:45.902 Spike logging is disabled
2021-05-14 11:50:45.902 Logging interval = 1 mins
2021-05-14 11:50:45.902 Real time interval = 60 secs
2021-05-14 11:50:45.902 NoSensorCheck = 0
2021-05-14 11:50:45.916 Calculating sunrise and sunset times
2021-05-14 11:50:45.919 Sunrise: 06:05:02
2021-05-14 11:50:45.919 Sunset : 20:33:12
2021-05-14 11:50:45.919 Tomorrow sunrise: 06:04:04
2021-05-14 11:50:45.919 Tomorrow sunset : 20:34:12
2021-05-14 11:50:45.964 Station type: Davis Vantage Pro2
2021-05-14 11:50:45.965 Units.Wind=mph RainUnit=in TempUnit=°F PressureUnit=in
2021-05-14 11:50:45.965 YTDRain=0.000 Year=0
2021-05-14 11:50:45.965 RainDayThreshold=-1.000
2021-05-14 11:50:45.965 Roll over hour=0
2021-05-14 11:50:45.965 Offsets and Multipliers:
2021-05-14 11:50:45.966 PO=0.000 TO=0.000 HO=0 WDO=0 ITO=0.000 SO=0.000 UVO=0.000
2021-05-14 11:50:45.966 PM=1.000 WSM=1.000 WGM=1.000 TM=1.000 TM2=0.000 HM=1.000 HM2=0.000 RM=1.000 SM=1.000 UVM=1.000
2021-05-14 11:50:45.966 Spike removal:
2021-05-14 11:50:45.966 TD=999.000 GD=999.000 WD=999.000 HD=999.000 PD=999.000 MR=999.000 MH=999.000
2021-05-14 11:50:45.966 Limits:
2021-05-14 11:50:45.966 TH=60.0 TL=-60.0 DH=40.0 PH=1090.00 PL=870.00 GH=90.000
2021-05-14 11:50:45.966 Primary AQ Sensor = Undefined
2021-05-14 11:50:46.017 Cumulus Starting
2021-05-14 11:50:46.031 HTML root path = /home/jeff/CumulusMX/interface
2021-05-14 11:50:46.110 Opening station
2021-05-14 11:50:46.117 Last update time from today.ini: 5/14/2021 11:44:00 AM
2021-05-14 11:50:46.117 Read today file: Date = 5/14/2021, LastUpdateTime = 5/14/2021 11:44:00 AM, Month = 5
2021-05-14 11:50:46.117 ReadTodayfile: Rain day start = 5.53
2021-05-14 11:50:46.117 ReadTodayfile: set initialiseRainCounterOnFirstData false
2021-05-14 11:50:46.120 /home/jeff/CumulusMX/data/alltime.ini
2021-05-14 11:50:46.120 Alltime.ini file read
2021-05-14 11:50:46.125 MonthlyAlltime.ini file read
2021-05-14 11:50:46.127 Month.ini file read
2021-05-14 11:50:46.129 Year.ini file read
2021-05-14 11:50:46.130 Finding raintoday from logfile data/May21log.txt
2021-05-14 11:50:46.130 Expecting listsep=, decimal=.
2021-05-14 11:50:46.412 Midnight rain found in the following entry:
2021-05-14 11:50:46.412 14/05/21,00:00,48.8,57,34.2,0,1,334,0.00,0.00,30.37,5.53,71.1,40,0,48.8,48.8,0.0,0,0.000,0.000,45.6,0,0.0,334,0.00,0.00,48.8,9.3
2021-05-14 11:50:46.422 Midnight rain found, setting midnight rain count = 5.53
2021-05-14 11:50:46.422 Checking rain counter = 5.53
2021-05-14 11:50:46.422 Rain counter set to = 5.53
2021-05-14 11:50:46.423 Getting rain totals, rain season start = 1
2021-05-14 11:50:46.423 Today = 14/05/21
2021-05-14 11:50:46.468 Rainthismonth from dayfile.txt: 0.9
2021-05-14 11:50:46.468 Rainthisyear from dayfile.txt: 5.53
2021-05-14 11:50:46.469 Station type = Davis
2021-05-14 11:50:46.470 LOOP2 enabled
2021-05-14 11:50:46.470 IP address = 192.168.100.13 Port = 22222
2021-05-14 11:50:46.470 periodic disconnect = 0
2021-05-14 11:50:46.470 InitTCP: Connecting to the station
2021-05-14 11:50:46.736 This Cumulus MX instance is running the latest version
2021-05-14 11:50:47.474 InitTCP: Flushing input stream
2021-05-14 11:50:52.901 InitTCP: Connection confirmed
2021-05-14 11:50:52.902 Connected OK
2021-05-14 11:50:52.902 Reading firmware version
2021-05-14 11:50:53.403 FW version = 3.80
2021-05-14 11:50:53.404 Reading reception stats
2021-05-14 11:50:53.905 CheckLoggerInterval: Reading logger interval
2021-05-14 11:50:54.406 Last update time = 5/14/2021 11:44:00 AM
2021-05-14 11:50:54.407 Reading console time
2021-05-14 11:50:54.907 Console clock: 5/14/2021 11:50:54 AM
2021-05-14 11:50:54.907 Console clock: Accurate to +/- 30 seconds, no need to set it (diff=0s)
2021-05-14 11:50:54.908 Reading history data from log files
2021-05-14 11:50:54.908 Loading last N hour data from data logs: 5/14/2021 11:44:00 AM
2021-05-14 11:50:54.909 LoadLastHour: Attempting to load last hour entries
2021-05-14 11:50:55.222 LoadLastHour: Loaded 61 entries to last hour data list
2021-05-14 11:50:55.223 LoadLast3Hour: Attempting to load 3 hour data list
2021-05-14 11:50:55.563 LoadLast3Hour: Loaded 181 entries to last 3 hour data list
2021-05-14 11:50:55.564 LoadGraphData: Attempting to load 168 hours of entries to graph data list
2021-05-14 11:50:55.931 LoadGraphData: Loaded 10081 entries to graph data list
2021-05-14 11:50:55.932 LoadRecent: Attempting to load 7 days of entries to recent data list
2021-05-14 11:50:56.372 LoadRecent: Loaded 10081 entries to recent data list
2021-05-14 11:50:56.372 LoadDayFile: Attempting to load the day file
2021-05-14 11:50:56.471 LoadDayFile: Loaded 1720 entries to recent daily data list
2021-05-14 11:50:56.480 Reading archive data from logger
2021-05-14 11:50:56.482 Creating extra sensors
2021-05-14 11:50:56.513 834 web tags initialised
2021-05-14 11:50:56.640 Get Archive Data
2021-05-14 11:50:56.640 Rollover hour = 0
2021-05-14 11:50:56.640 GetArchiveData: Last Archive Date: 5/14/2021 11:44:00 AM
2021-05-14 11:50:56.640 GetArchiveData: Sending DMPAFT
2021-05-14 11:50:57.140 GetArchiveData: Received response to DMPAFT, sending start date and time
2021-05-14 11:50:57.641 GetArchiveData: Reading data: 2 pages , offset = 4
2021-05-14 11:50:57.641 GetArchiveData: Reading archive page 0
2021-05-14 11:50:57.641 GetArchiveData: Waiting for response
2021-05-14 11:50:58.142 GetArchiveData: Loaded archive record for Page=0 Record=4 Timestamp=5/14/2021 11:45:00 AM
2021-05-14 11:50:58.142 GetArchiveData: Processing archive record for 5/14/2021 11:45:00 AM
2021-05-14 11:50:58.155 GetArchiveData: Page=0 Record=4 Timestamp=5/14/2021 11:45:00 AM
2021-05-14 11:50:58.157 DoLogFile: Writing log entry for 5/14/2021 11:45:00 AM
2021-05-14 11:50:58.160 DoLogFile: Written log entry for 5/14/2021 11:45:00 AM
2021-05-14 11:50:58.161 Writing today.ini, LastUpdateTime = 5/14/2021 11:45:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:50:58.163 GetArchiveData: Log file entry written
2021-05-14 11:50:58.168 1 archive entries processed
2021-05-14 11:50:58.168 GetArchiveData: Reading archive page 1
2021-05-14 11:50:58.168 GetArchiveData: Waiting for response
2021-05-14 11:50:58.668 GetArchiveData: Loaded archive record for Page=1 Record=0 Timestamp=5/14/2021 11:46:00 AM
2021-05-14 11:50:58.668 GetArchiveData: Processing archive record for 5/14/2021 11:46:00 AM
2021-05-14 11:50:58.668 GetArchiveData: Page=1 Record=0 Timestamp=5/14/2021 11:46:00 AM
2021-05-14 11:50:58.668 DoLogFile: Writing log entry for 5/14/2021 11:46:00 AM
2021-05-14 11:50:58.668 DoLogFile: Written log entry for 5/14/2021 11:46:00 AM
2021-05-14 11:50:58.669 Writing today.ini, LastUpdateTime = 5/14/2021 11:46:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:50:58.669 GetArchiveData: Log file entry written
2021-05-14 11:50:58.669 2 archive entries processed
2021-05-14 11:50:58.669 GetArchiveData: Loaded archive record for Page=1 Record=1 Timestamp=5/14/2021 11:47:00 AM
2021-05-14 11:50:58.669 GetArchiveData: Processing archive record for 5/14/2021 11:47:00 AM
2021-05-14 11:50:58.670 GetArchiveData: Page=1 Record=1 Timestamp=5/14/2021 11:47:00 AM
2021-05-14 11:50:58.670 DoLogFile: Writing log entry for 5/14/2021 11:47:00 AM
2021-05-14 11:50:58.670 DoLogFile: Written log entry for 5/14/2021 11:47:00 AM
2021-05-14 11:50:58.670 Writing today.ini, LastUpdateTime = 5/14/2021 11:47:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:50:58.674 GetArchiveData: Log file entry written
2021-05-14 11:50:58.674 3 archive entries processed
2021-05-14 11:50:58.674 GetArchiveData: Loaded archive record for Page=1 Record=2 Timestamp=5/14/2021 11:48:00 AM
2021-05-14 11:50:58.674 GetArchiveData: Processing archive record for 5/14/2021 11:48:00 AM
2021-05-14 11:50:58.675 GetArchiveData: Page=1 Record=2 Timestamp=5/14/2021 11:48:00 AM
2021-05-14 11:50:58.675 DoLogFile: Writing log entry for 5/14/2021 11:48:00 AM
2021-05-14 11:50:58.675 DoLogFile: Written log entry for 5/14/2021 11:48:00 AM
2021-05-14 11:50:58.675 Writing today.ini, LastUpdateTime = 5/14/2021 11:48:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:50:58.676 GetArchiveData: Log file entry written
2021-05-14 11:50:58.676 4 archive entries processed
2021-05-14 11:50:58.676 GetArchiveData: Loaded archive record for Page=1 Record=3 Timestamp=5/14/2021 11:49:00 AM
2021-05-14 11:50:58.676 GetArchiveData: Processing archive record for 5/14/2021 11:49:00 AM
2021-05-14 11:50:58.676 GetArchiveData: Page=1 Record=3 Timestamp=5/14/2021 11:49:00 AM
2021-05-14 11:50:58.676 DoLogFile: Writing log entry for 5/14/2021 11:49:00 AM
2021-05-14 11:50:58.676 DoLogFile: Written log entry for 5/14/2021 11:49:00 AM
2021-05-14 11:50:58.677 Writing today.ini, LastUpdateTime = 5/14/2021 11:49:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:50:58.677 GetArchiveData: Log file entry written
2021-05-14 11:50:58.677 5 archive entries processed
2021-05-14 11:50:58.677 GetArchiveData: Loaded archive record for Page=1 Record=4 Timestamp=5/14/2021 11:50:00 AM
2021-05-14 11:50:58.677 GetArchiveData: Processing archive record for 5/14/2021 11:50:00 AM
2021-05-14 11:50:58.678 GetArchiveData: Page=1 Record=4 Timestamp=5/14/2021 11:50:00 AM
2021-05-14 11:50:58.678 DoLogFile: Writing log entry for 5/14/2021 11:50:00 AM
2021-05-14 11:50:58.678 DoLogFile: Written log entry for 5/14/2021 11:50:00 AM
2021-05-14 11:50:58.679 Writing today.ini, LastUpdateTime = 5/14/2021 11:50:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:50:58.679 GetArchiveData: Log file entry written
2021-05-14 11:50:58.679 6 archive entries processed
2021-05-14 11:50:58.679 Get Archive Data
2021-05-14 11:50:58.679 Rollover hour = 0
2021-05-14 11:50:58.679 GetArchiveData: Last Archive Date: 5/14/2021 11:50:00 AM
2021-05-14 11:50:58.679 GetArchiveData: Sending DMPAFT
2021-05-14 11:51:01.877 InitTCP: Connecting to the station
2021-05-14 11:51:02.878 InitTCP: Flushing input stream
2021-05-14 11:51:07.300 InitTCP: Connection confirmed
2021-05-14 11:51:07.300 InitTCP: Connecting to the station
2021-05-14 11:51:08.301 InitTCP: Flushing input stream
2021-05-14 11:51:12.723 InitTCP: Connection confirmed
2021-05-14 11:51:12.723 GetArchiveData: No Ack in response to DMPAFT
2021-05-14 11:51:12.723 GetArchiveData: Sending DMPAFT
2021-05-14 11:51:12.723 GetArchiveData: Error sending LOOP command [DMPAFT]: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
2021-05-14 11:51:12.723 InitTCP: Connecting to the station
2021-05-14 11:51:13.724 InitTCP: Flushing input stream
2021-05-14 11:51:18.145 InitTCP: Connection confirmed
2021-05-14 11:51:18.146 Logger archive reading thread completed
2021-05-14 11:51:18.148 Start Extra Sensors
2021-05-14 11:51:18.148 Start Timers
2021-05-14 11:51:18.148 Starting 1-minute timer
2021-05-14 11:51:18.148 Start normal reading loop
2021-05-14 11:51:18.149 SendLoopCommand: Starting - LPS 2 1
2021-05-14 11:51:18.149 Data logging interval = 0 (1 mins)
2021-05-14 11:51:18.149 Starting Realtime timer, interval = 60 seconds
2021-05-14 11:51:18.149 Starting MySQL catchup thread
2021-05-14 11:51:18.150 Normal running
2021-05-14 11:51:18.157 MySQL Archive: End of MySQL archive upload
2021-05-14 11:51:19.151 SendLoopCommand: Starting - LOOP 20
2021-05-14 11:51:56.120 SendLoopCommand: Starting - LPS 2 1
2021-05-14 11:51:57.121 SendLoopCommand: Starting - LOOP 20
2021-05-14 11:52:00.167 DoLogFile: Writing log entry for 5/14/2021 11:52:00 AM
2021-05-14 11:52:00.169 DoLogFile: Written log entry for 5/14/2021 11:52:00 AM
2021-05-14 11:52:00.169 Writing today.ini, LastUpdateTime = 5/14/2021 11:52:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:52:34.121 SendLoopCommand: Starting - LPS 2 1
2021-05-14 11:52:35.122 SendLoopCommand: Starting - LOOP 20
2021-05-14 11:53:00.169 DoLogFile: Writing log entry for 5/14/2021 11:53:00 AM
2021-05-14 11:53:00.169 DoLogFile: Written log entry for 5/14/2021 11:53:00 AM
2021-05-14 11:53:00.170 Writing today.ini, LastUpdateTime = 5/14/2021 11:53:00 AM raindaystart = 5.53 rain counter = 5.53
2021-05-14 11:53:12.126 SendLoopCommand: Starting - LPS 2 1
2021-05-14 11:53:13.127 SendLoopCommand: Starting - LOOP 20
So not sure what is happening

User avatar
mcrossley
Posts: 8473
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Buster Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: sql catch up

Post by mcrossley »

Erk! Yes, there is a bug that kills the MySQL catch-up before it does anything. Sorry about that, I'll get a fix out asap.

jlmr731
Posts: 187
Joined: Sat 27 Aug 2016 12:11 am
Weather Station: Davis vantage pro 2
Operating System: Debian
Location: Youngstown, Ohio
Contact:

Re: sql catch up

Post by jlmr731 »

good to hear its only a bug, hope its an easy fix. Thanking You !

User avatar
mcrossley
Posts: 8473
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Buster Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: sql catch up

Post by mcrossley »

Fixed and tested.

I'm still working on a couple of other things in this release, mainly testing them left now, I have a day out tomorrow, I hope to release this on Sunday barring any issues.

sfws
Posts: 983
Joined: Fri 27 Jul 2012 11:29 am
Weather Station: Combination Mix of FO stations
Operating System: R Pi OS

Re: sql catch up

Post by sfws »

jlmr731 wrote:
Fri 14 May 2021 4:00 pm
I see missing entry's for monthly sql table 11:44 to 11:52 the entries between them are missing (reboot time)
Mark perhaps could have mentioned that he has this routine that will insert the missing rows for you, as his next MX release won't tackle past catch-up errors.

On your Debian

Code: Select all

sudo mono ExportToMySQL.exe data/Apr21log.txt
will work through the log file, and since it uses INSERT IGNORE, only missing rows are inserted. As both CumulusMX.exe and ExportToMySQL.exe will be accessing the same log file and same database table, it is best to run this immediately after former has done a log update (or while MX stopped for an upgrade).

EDIT: As subsequent post reports, when I copied and pasted in the link to "https://github.com/cumulusmx/ExportToMySQL", my copy action missed off final letter "L". Iin correcting that manually my inexpert typing did not register shift depression at same time as "L" key pressed and so inserted a lower case "l" . Link above now has fully correct URL.
Last edited by sfws on Sun 16 May 2021 5:06 am, edited 1 time in total.

jlmr731
Posts: 187
Joined: Sat 27 Aug 2016 12:11 am
Weather Station: Davis vantage pro 2
Operating System: Debian
Location: Youngstown, Ohio
Contact:

Re: sql catch up

Post by jlmr731 »

sfws wrote:
Sat 15 May 2021 6:45 pm

Mark perhaps could have mentioned that he has this routine that will insert the missing rows for you, as his next MX release won't tackle past catch-up errors.
Thanks SFWS, I did use this and worked well (edit your link extra I in url). I did look at tables and logs after running and have 1 missing entry at 11:51 I believe it must be due to MX start-up just hit it at the right (wrong) time, but for anybody else that has this problem using this routine will work for you.

User avatar
mcrossley
Posts: 8473
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Buster Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: sql catch up

Post by mcrossley »

sfws wrote:
Sat 15 May 2021 6:45 pm
[
Mark perhaps could have mentioned that he has this routine that will insert the missing rows for you, as his next MX release won't tackle past catch-up errors.
:thumbsup:

jlmr731
Posts: 187
Joined: Sat 27 Aug 2016 12:11 am
Weather Station: Davis vantage pro 2
Operating System: Debian
Location: Youngstown, Ohio
Contact:

Re: sql catch up

Post by jlmr731 »

Mark 3131 worked perfectly thanks a billion :clap:

The it guy
Posts: 25
Joined: Fri 22 Feb 2019 12:46 am
Weather Station: Davis
Operating System: Windows 10

Re: sql catch up

Post by The it guy »

will this work for the davis Air link as well?

User avatar
mcrossley
Posts: 8473
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Buster Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: sql catch up

Post by mcrossley »

The AirLink data is not logged in the MySQL database? (as standard anyway)

Post Reply