Page 1 of 1
EOD process
Posted: Thu 09 Apr 2020 9:26 am
by sfws
MX adds more functionality at End of Day than Cumulus 1 offered.
This is how I think MX works, have I got it right?
Cumulus MX End of Day process
=============================
Hour change (done every hour)
Calculating Sunrise and sunset (done every hour)
Entering Day Reset (message about current day of month)
Day Reset (message about date ending, time shown as 00:00:00 because time not defined, not because it is midnight, it might be 9am or 10am)
Run EOD standard or custom SQL (the SQl is updated every time readings are updated)
Custom Http EOD call
Create line to append to dayfile.txt
Open dayfile.txt
Update dayfile.txt
Read Rain Counter
Update today.ini with yesterday's rain
Write yesterday.ini
Create NOAA monthly
Save NOAA monthly
Create NOAA yearly
Save NOAA yearly
Run EOD External Program
Processs any Extra Files with EOD option selected
Day reset complete
Read readings from weather station to assign to start of new day
Create daily backup folder to hold files as at start of new day
Copy all files from data folder (plus Cumulus.ini) into new daily folder
Resume normal operation of reading station, processing real-time and standard time interval functions
In the first Extra Files with standard interval FTP upload, add upload of Extra Files processed at end of day
What I have not worked out is why Cumulus MX changes <#metdate> before it runs the EOD SQL. The implication is to replicate the dayfile.txt date in a custom EOD SQL you need to use <#metdateyesterday> instead! I suppose it is all part of so called EOD actions mostly being START of day actions, like the stored month.ini in the daily backup will be the empty one for the new month (not the position at end of previous month) if the month has changed at rollover.
Re: EOD process
Posted: Thu 09 Apr 2020 10:38 am
by mcrossley
CMX doesn't explicitly change the #metdate, it's just that #metdate is derived from the current date/time, so since it will be after the rollover time when the SQL code is called it is already "today", therefore your date should refer to yesterday.
Re: EOD process
Posted: Sun 19 Apr 2020 10:36 pm
by mcrossley
Well, the "so called EOD actions" have to run at the start of the next day otherwise the day wouldn't have ended! They are called end of day actions because they finalise the day and store everything away, then reset things for the next day.
The daily backup is just a snapshot of the files, so yes at the start of the month the month.ini file will be empty, but the previous months file is backed up as well.
sfws wrote: ↑Thu 09 Apr 2020 9:26 am
This is how I think MX works, have I got it right?
Cumulus MX End of Day process
=============================
Hour change (done every hour)
Calculating Sunrise and sunset (done every hour)
Entering Day Reset (message about current day of month)
Day Reset (message about date ending, time shown as 00:00:00 because time not defined, not because it is midnight, it might be 9am or 10am)
Run EOD standard or custom SQL (the SQl is updated every time readings are updated)
Custom Http EOD call
Create line to append to dayfile.txt
Open dayfile.txt
Update dayfile.txt
Read Rain Counter
Update today.ini with yesterday's rain
Write yesterday.ini
Create NOAA monthly
Save NOAA monthly
Create NOAA yearly
Save NOAA yearly
Run EOD External Program
Processs any Extra Files with EOD option selected
Day reset complete
Read readings from weather station to assign to start of new day
Create daily backup folder to hold files as at start of new day
Copy all files from data folder (plus Cumulus.ini) into new daily folder
Resume normal operation of reading station, processing real-time and standard time interval functions
In the first Extra Files with standard interval FTP upload, add upload of Extra Files processed at end of day
Not quite, reading through the DayReset() function, you have mixed up the EOD processing and the normal interval processing. So day reset does this...
Reset midnight rain
Entering Day Reset (message about current day of month)
Day Reset (message about date ending, time shown as 00:00:00 because time not defined, not because it is midnight, it might be 9am or 10am)
Run EOD custom SQL
Save dayfile entry
Write monthly & yearly file entries
Write any new records
if day = 1 then: copy month.ini to saved file, reset monthly figures
if day = 1 and month = 1 then: copy year.ini to saved file, reset yearly figures
Copy todays high/lows to yesterdays
Reset todays high/lows to current
Write today.ini & yesterday.ini
Create NOAA reports
Execute user daily external program
Process Extra EOD files
But independent of that the normal processing is also taking place, What happens depends on your settings, and if your FTP interval is synchronised with the logging interval.
So as rollover occurs on the hour, then the normal interval and hourly processes will also occur - at the same time*
*depending on the processing capability of your machine.
Re: EOD process
Posted: Mon 20 Apr 2020 5:41 am
by HansR
mcrossley wrote: ↑Sun 19 Apr 2020 10:36 pm
[...] will also occur - at the same time*
At the same time in computer terms is always tricky, so I assume there is no dependency or interference between those actions.
Re: EOD process
Posted: Mon 20 Apr 2020 7:30 am
by mcrossley
HansR wrote: ↑Mon 20 Apr 2020 5:41 am
mcrossley wrote: ↑Sun 19 Apr 2020 10:36 pm
[...] will also occur - at the same time*
At the same time in computer terms is always tricky, so I assume there is no dependency or interference between those actions.
They run independently on separate threads.
When I ran CMX on a Pi zero - v. slow and single CPU - I did discover a race condition on rollover that didn't occur on faster multi-CPU machines, but that was fixed a while back. I don't think there are any more, but that doesn't mean they aren't there.
Re: EOD process
Posted: Thu 30 Apr 2020 7:30 am
by sfws
Mark
In
viewtopic.php?f=39&t=12943 Steve Loft lists within
Known issues:
End of day reset code gets the date of day just ending wrong for 0900 rollovers
Is this a resolved issue, or is this why the <metdateyesterday> has to be used in Custom routines?
You may have seen I posted yesterday
viewtopic.php?f=43&t=18033&p=141782#p141782 a request for an updated version of the above post to be included within the modern MX versions sub-forum, because as well as
Known Issues it contains
FAQ that help people to swap from C1 to MX.
Re: EOD process
Posted: Thu 30 Apr 2020 8:59 am
by mcrossley
sfws wrote: ↑Thu 30 Apr 2020 7:30 am
End of day reset code gets the date of day just ending wrong for 0900 rollovers
Is this a resolved issue, or is this why the <metdateyesterday> has to be used in Custom routines?
I don't know what that issue was, but I haven't heard of anyone using the 9am rollover having any issues, so I assume it was fixed until I hear otherwise.
As for having to use metdayyesterday, no, as I explained before, it's because the web tags take their values from the current clock time, which is always going to be the "next" day during the rollover processing as it triggers at the rollover time and thus runs the following day. The only workaround for that I can see is to change the web tags to use a fixed date that is updated by the day reset code - there may be unintended consequences though.
Re: EOD process
Posted: Thu 30 Apr 2020 9:42 am
by freddie
mcrossley wrote: ↑Thu 30 Apr 2020 8:59 am
sfws wrote: ↑Thu 30 Apr 2020 7:30 am
End of day reset code gets the date of day just ending wrong for 0900 rollovers
Is this a resolved issue, or is this why the <metdateyesterday> has to be used in Custom routines?
I don't know what that issue was, but I haven't heard of anyone using the 9am rollover having any issues, so I assume it was fixed until I hear otherwise.
Yes, it got fixed. I think it was me that spotted it at the time (5 years ago?).
mcrossley wrote: ↑Thu 30 Apr 2020 8:59 am
As for having to use metdayyesterday, no, as I explained before, it's because the web tags take their values from the current clock time, which is always going to be the "next" day during the rollover processing as it triggers at the rollover time and thus runs the following day. The only workaround for that I can see is to change the web tags to use a fixed date that is updated by the day reset code - there may be unintended consequences though.
I do this (using the "yesterday" tags) for my 0900 tweets, as I display the max/min/rainfall for the meteorological day in each tweet, and the values at 0900 would be wrong otherwise. I have a cron task that swaps the twitter.txt template file to the "yesterday" version at 0855, and then swaps back to the "today" version at 0905. It's no real hassle. If you consider it as "start of day" processing rather than "end of day" then it all makes perfect sense
Re: EOD process
Posted: Thu 30 Apr 2020 11:47 am
by sfws
freddie wrote: ↑Thu 30 Apr 2020 9:42 am
Yes, it got fixed. I think it was me that spotted it at the time (5 years ago?).
Thanks Freddie. So that is how
https://twitter.com/DorringtonWx works. As for "Start of day"
mcrossley wrote: ↑Sun 19 Apr 2020 10:36 pm
The "so called EOD actions" have to run at the start of the next day otherwise the day wouldn't have ended!
I have a batch job that updates my database tables (Cumulus initiates it via external programme feature), I have to test whether it is the last day of a month that has ended in three places where it has to work differently to avoid update using start of new month statistics. I also in my versions of the PHP web tag templates have to do double extra file actions to properly handle today/yesterday issues.
Maybe, Freddie, you are the person who can do the update I request in
viewtopic.php?f=43&t=18033 after all if you have knowledges of fixes done by Steve Loft, but not recorded.
Re: EOD process
Posted: Thu 30 Apr 2020 11:55 am
by freddie
sfws wrote: ↑Thu 30 Apr 2020 11:47 amMaybe, Freddie, you are the person who can do the update I request in
viewtopic.php?f=43&t=18033 after all if you have knowledges of fixes done by Steve Loft, but not recorded.
Yep, I can certainly look at that at some point. I expect the fix was recorded (by way of a forum post) as I wouldn't have been aware that it was fixed. Steve probably forgot to edit the fixes list.