3.20.0 question

[Mapantz]

Using my script the dayfile updated - all days since Jan 2010 - in about 20 seconds, and that is running on a Raspberry Pi.

ExportToMySQL adds missing rows, it does not add missing data to existing rows.

I'm still updating with the importcumulus file, it adds one row every 5 seconds for me.

re ExportToMySQL
I refreshed my entire dayfile so that it contained the newly added 3 columns. I deleted all data in my dayfile SQL table, then ran ExportToMySQL. It did not populate chillhours/HighRain24h/THighRain24h

ExportToMySQL is also slow - 5 seconds per row.

[sfws]

I am staggered to hear how slow progress is for you, that must imply very slow transfers to your database server. (Or very inefficient php)

re ExportToMySQL
I refreshed my entire dayfile so that it contained the newly added 3 columns. I deleted all data in my dayfile SQL table, then ran ExportToMySQL. It did not populate chillhours/HighRain24h/THighRain24h

ExportToMySQL is also slow - 5 seconds per row.

That does suggest the slow speed issue is between your PC and your database server.

I'm guessing your dayfile.txt file is well under 3000 lines from the start date of March 2016 on your web site.

I tested with a dayfile.txt holding 3696 lines.
I have MX and the database server on the same RPi 3 computer, using ExportToMySQL.exe a specially created empty table was filled with all rows and columns in much less than one minute. So similar experience to Mark, but I was not counting the seconds.

Presumably you have checked your dayfile.txt does have the required fields?
Are you absolutely sure you are using ExportToMySQL.exe release 1.6.0 modified 3 days ago, not some old utility file? The new code does populate the new columns.

[Mapantz]

I am staggered to hear how slow progress is for you, that must be very inefficient code you are using in your php.

re ExportToMySQL
I refreshed my entire dayfile so that it contained the newly added 3 columns. I deleted all data in my dayfile SQL table, then ran ExportToMySQL. It did not populate chillhours/HighRain24h/THighRain24h

ExportToMySQL is also slow - 5 seconds per row.

I'm guessing your dayfile.txt file is well under 3000 lines from the start date of March 2016 on your web site.

I tested with a dayfile.txt holding 3696 lines.
I use a RPi 3 as my computer, using ExportToMySQL.exe a specially created empty table was filled with all rows and columns in much less than one minute. So similar experience to Mark, but I was not counting the seconds.

Presumably you have checked your dayfile.txt does have the required fields?
Are you absolutely sure you are using ExportToMySQL.exe release 1.6.0 modified 3 days ago, not some old utility file?

I'm using the file from here: https://cumuluswiki.org/a/ImportCumulusFile

I did it the unorthodox way.. I did a truncate of the table, uploaded my new dayfile.txt with the new columns of data filled in for all entries, then ran ImportCumulusFile.php. It takes 5 second per row, and it will only fill in 3 months worth of data at a time, so you had to refresh the page when it finished each update. It actually doesn't report back any data either.. it only did so on the last 100 rows.

Data insert done, 100 rows inserted
Script complete: 21/08/22 - 20:27:16
Execution time: 269.7 secs

[philpugh]

Mark,

Is the field size for ChillHours correct (5.2) as I had to set it to 6.2 to import my data - it wasn't displaying 2227.1 hours correctly.

On second thought I wonder if the ChillHours should be a daily value - the CreateMissing.exe seems to put a running total for ChillHours into the dayfile.txt file?

[mcrossley]

@philpugh, no it is not correct, CMX creates it as 7,dp - where dp = number of temperature decimals.

I see that 5,2 is in SQL script, I'll fix that.

Chill hours are cumulative.

[philpugh]

Thanks Mark - I'll update to 7,2.

[sfws]

I wonder if the ChillHours should be a daily value - the CreateMissing.exe seems to put a running total for ChillHours into the dayfile.txt file?

Chill hours are cumulative.

The post at viewtopic.php?p=111362#p111362 mentions Steve Loft's proposal that MX could store cumulative chill hours. The topic at viewtopic.php?t=4736 covers the background/original request to Steve Loft for monitoring chill hours, but note that the enhancement tracker mentioned with a link was lost in a change of hosts years ago (some of its text is preserved in Wiki).

[BeaumarisWX]

Using my script the dayfile updated - all days since Jan 2010 - in about 20 seconds, and that is running on a Raspberry Pi.

ExportToMySQL adds missing rows, it does not add missing data to existing rows.

viewtopic.php?p=165767#p165767

Hi Mark,

Thanks for the tips regards your Dayfile / Excel / Sql Script, used it for mine although I only had data going back to 7th October 2019 only took a few seconds using my NUC Haydes Win10.

Now to set about displaying new data, have to get the thinking cap on, anyone done anything with it yet ?

Kind Regards,

[freddie]

This thread has been really useful regarding getting the new data from the log files into my database. My upgrade steps were:

• Add the new columns to the database tables (plus change my REST API to reflect this).
• Do the MX upgrade.
• Run CreateMissing.
• Run ExportToMsql.
• Use the new dayfile.txt to create UPDATE statements to populate the missing data in the new columns.

I wasn't looking forward to this release - but it went very smoothly in the end.

I only had one snag, and that was that CreateMissing complained about my March 2022 data file and couldn't process 2nd March to 31st March for some reason - although it did manage to extract ChillHours data somehow, just not the rainfall. I have looked at the data in the file and can't see anything wrong with it. @mcrossley I wondered if you could have a quick look to see if I've missed anything obvious? I have used Mk1 Eyeball, plus imported the data into Excel.

Attached is the log file for March 2022, plus the CreateMissing log.

Thanks.

Mar22log.zip

CreateMissing-20220828-120333.zip

[mcrossley]

@freddie, I just took a quick look on my tablet. The CreateMissing log file is outputting US format dates, I need to check, is that something it always does and I just haven't noticed, or was it run under a US locale?

I do not see anything wrong with the log file at first glance.

[freddie]

@freddie, I just took a quick look on my tablet. The CreateMissing log file is outputting US format dates, I need to check, is that something it always does and I just haven't noticed, or was it run under a US locale?

It could be either - I will check tomorrow.

[freddie]

@freddie, I just took a quick look on my tablet. The CreateMissing log file is outputting US format dates, I need to check, is that something it always does and I just haven't noticed, or was it run under a US locale?

It could be either - I will check tomorrow.

My locale is set to C.UTF-8. I hadn't heard of it before - but apparently it is the "POSIX standards-compliant default locale". It appears to be an attempt by Debian to modernise the standard C locale. It is widely available in Linux distros.

Every day is a school day

[mcrossley]

Never heard of it before either - whatever it is, it uses a US date format.