wrong clock after first start up

[Carbonara]

hi to everyone,
with the latest version of cmx i've noticed that at first start up cmx start to recording with a wrong clock date. How can i set the correct clock ? (the system clock is ok)
Thanks in advance!

cmx_clock.png

this happen with davis and fineoffset station

[mcrossley]

I'm guessing you have CMX set to auto-start on boot?

The raspberry pi does not have a battery backed up hardware clock. So on first start-up it uses the date/time from when it was last run until it can set the clock from an NTP source.

You need to delay CMX running until this has happened (or add a clock module - which is what I have done).

You can use the start-up delay settings, and/or the Ping feature to check the network is up before allowing CMX to download data from your station.

[Carbonara]

Thanks Mark,
ok. i'll try the latest version and then the "ping start up"

[Carbonara]

so,
i've deleted the old cmx folder and i've upload the new one.
After setting all main things for the station i restart the cmx and got this.

cmx_clock3.png

(the ping set up is setting to 10 sec delay after 8.8.8.8 pings )

[sfws]

i've noticed that at first start up cmx start to recording with a wrong clock date.

I have had this issue a few times, I did wonder whether I should put in a request for MX to detect a sudden massive change in the clock time, but was not sure what it should then do!

When Storm Arwen hit last November my power went off at a minute past 5 am and stayed off for more than 11 hours.
When the RPi rebooted (after both power and broadband network came back), Cumulus MX restarted processing at what it thought was a few minutes past 5am, so it did not do any historic catch-up, the time was actually about 5pm so around 12 hours of historic data was missing. When I had checked every electric appliance had restarted, and my fridge/freezer contents were okay, I discovered the RPi time sync did not happen until many minutes of wrong time-stamped data had been stored, (presumably the RPi does its clock check at specific clock times, not immediately after network connection). The additional problem was that my 9am rollover time was not seen! So I had measurements applied to wrong day as well as wrong time. The resolution was to reload the data folder with files from a back-up, which got as good a set of data as I could achieve including a successful rollover.

The network checking solution within MX does not work for me.
1) My settings in Raspberry Pi Configuration --> System tab --> Network at Boot is Wait for network already, so when my RPi is rebooted, as in my Storm Arwen experience, it has already ensured the network is connected, any subsequent check ping for network when MX starts is pointless (that MX ping coding is using the RPi time whether it is correct or not, and this was before I read the condition that was coded in MX).
2) It is irrelevant because the RPi clock sync does not happen as soon as the network is available as in my Storm Arwen experience.
3) In my rural location, the fibre to box internet capacity is insufficient for the area, network link frequently goes down while my RPi computer continues to run. My weather station connects via USB, so I am happy for MX to continue running, or restart after an upgrade (with the clock time already right), without the network.

Using a start-up delay has not ensured I avoid wrong time issues on every such occasion so far.
1) Possibly, I was not setting a long enough start-up delay, the issue is that I can't predict when the clock sync will occur
2) I sometimes reboot my computer as part of upgrading to a new MX release, and do not want a long start-up delay for a short downtime (with the clock time about right) (Edited part of this line, previously thought & typing did not match)

Ready for Storm Dudley last month, I added a real-time clock (RTC) module to my RPI3b+. This did not prevent me having a wrong time issue again after the next power issue. I'm not a technical expert, but maybe my RPi is not recognising my RTC. I followed the instructions for installing my RTC, but maybe for a RPi3b there is some other change to a file somewhere that is needed. I did an online search, but that yielded conflicting advice on what you should change after installing a RTC, so I decided not to risk any changes mentioned in any of those posts!

My logical mind decided it would make better sense to do an explicit check for time-sync in the systemd definition file, so on a reboot, the cumulusmx service does not start until the time sync has taken place. Once again, I found conflicting advice online, and was confused. The change I did make after this research was wrong as when I next had a power problem (the power connection fell out of the RPi!), I experienced the restarting with wrong time issue again.

Very recently there was some heavy rain halting my gardening, so I decided to research this time-synch issue again, this time in the Linux manual. It was hard reading, it presents everything in a technical way, but I knew it could be relied upon to be correct. After that I implemented a new systemd definition file solution. My MX service has successfully restarted with this new service definition.
I have not had a power issue to upset the time, and prove it waits for time-sync, but I felt confident enough to document my fix here in the Wiki. (EDIT: Link updated, my Wiki edit for 3.15.1 changed the section heading!)(Another edit - link became invalid again as I then did a substantial rewrite of the whole MX on Linux Wiki page following discussion in another forum topic!)

[mcrossley]

Further to the reply by @sfws above...

The MX start-up delay has a secondary option. The "Max system uptime to apply the start-up delay" value.

Specifying this value gets around the objection that you do not want the start-up delay to apply if you just restart MX whilst your system is running. The start-up delay will not occur if your system has been running longer than the time you specify.

Regarding the service definition file. You will find later versions have the entries...

Code: Select all

Wants=network-online.target
After=network-online.target

Earlier versions only had the After entry.

This is because the "After" network-online.target will only apply if the network online detection service has been started. If no service "Wants" the network online detection service, then it will not be started. This may explain why some people were not seeing MX wait for the network in the past.
Adding the Wants forces it to start.

[Carbonara]

so,
i've deleted the old cmx folder and i've upload the new one.
After setting all main things for the station i restart the cmx and got this.
cmx_clock3.png
(the ping set up is setting to 10 sec delay after 8.8.8.8 pings )

i've stop cumulus (systemctl stop...). then i've deleted the ping start up on cumulus.ini file, restarted, and now works with correct clock.

[sfws]

You will find later versions have the entries...

I know you have already created 3.15.3 - b3172 in your development, but there are actually only 2 public releases with that change.
You did it just 6 days ago for 3.15.1. Your release announcement for that release did not tell those using MX on Linux they could copy your new file (possibly amended for path to their MX) to /etc/systemd/system/cumulusmx.service, so I am sure I am not the only person who did not do that.
Anyway, I've updated the "MX on Linux" Wiki page based on what you have now said about the change. Like anything else I have typed into the Wiki it might not be perfect text for technical experts, but should be good enough for most readers.

The MX start-up delay has a secondary option. The "Max system uptime to apply the start-up delay" value.

Specifying this value gets around the objection that you do not want the start-up delay to apply

My fault, I typed the whole post just before going to bed on Monday, the correct words short downtime were in my head as I composed that post, but I typed the wrong words for the text repeated below, and misled you.

2) I also restart MX after I upgrade to a new release, and I don't want a start-up delay then.

I should have typed "2) I sometimes reboot my computer as part of upgrading to a new MX release, and do not want a long start-up delay for a short downtime."
I.e. I was talking about opposite to Long uptime that the additional set-up parameter covers.

Let me considerably expand the above to explain clearly what I meant.

2) I have seen advice in this support forum, that leaving a RPi running for long periods, can give rise to issues, and sometimes rebooting it is good practice. Therefore when I upgrade MX to a new release, I may EITHER (a) copy all files from release distribution over the existing files, then type "sudo systemctl restart cumulusmx", OR (b) type "sudo systemctl stop cumulusmx", then reboot my RPi, the system is not off long enough for the clock time to be out by more than the logging interval I use. In this case there has been a slight delay for the computer reboot and I do not want MX to add a start-up delay set long enough to always ensure that time-sync has been done.

I don't have the technical knowledge to understand this, but it appears that when I use (a) above, MONO crashes (and creates a dump file), instead of coping with MX restart (maybe another change is needed in service definition file to handle Mono stop?).
When I do (b) apparently everything is correctly closed, and correctly restarted, as no MONO crash is reported.

[flort]

On 14th May I had a power outage in the early hours of the morning and when the RPi restarted it had the wrong clock time. I have been away on holidays and only just arrived home so haven't been able to investigate until now.

The last update time in today.ini was 14/05/2022 3:35:00 AM but the clock time on the RPi at reboot was 03:22 AM. By the time it got to trying to download historical data it was 03:29 AM so tried to catch-up from 03:35 to 03:29 which was a time in the past and therefore failed.

I believe I have implemented everything as suggested on the Wiki and in this post with updates to my cumulusmx.service file. It reads -

[Unit]
Description=CumulusMX service
Documentation=https://cumuluswiki.org/a/Main_Page
Requires= time-sync.target local-fs.target
After=network-online.target
Wants=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/mono-service -d:/home/pi/CumulusMX CumulusMX.exe -service -lang en-AU
Type=forking
ExecStopPost=/bin/rm /tmp/CumulusMX.exe.lock

[Install]
WantedBy=multi-user.target

I have attached the MXdiags file which had debugging turned on at the time. The clock problem and catch-up starts at line 326. The RPi clock corrected itself at line 569 where the time changed from 03:30 to 04:05.

Appreciate if someone could have a look and see if I have missed anything in my configuration. It may also be that the time-sync command is not completely working as expected.

Regards,
Trevor

[sfws]

I believe I have implemented everything as suggested on the Wiki and in this post with updates to my cumulusmx.service file.
...
The RPi clock corrected itself at line 569 where the time changed from 03:30 to 04:05.
...
It may also be that the time-sync command is not completely working as expected.

Point 1: I did write in the Wiki that the advice re changing the systemd definition file was for technical people, this is because you can't do just that change in that file and expect your raspberry pi to have correct time on restart. There are other start-up processes running on your computer including a fake-hwclock package. It is this package that assigned the fake time of 03:22 for you. That fake time satisfied the time sync condition in your MX service definition file and so it is working as I expected although MX started with incorrect time. Changing the service definition file for MX may be a useful step, but it is not the magic solution for wrong time issues.

It is irrelevant because the RPi clock sync does not happen as soon as the network is available as in my Storm Arwen experience.
...
I decided to research this time-synch issue again
...
My MX service has successfully restarted with this new service definition.
I have not had a power issue to upset the time, and prove it waits for time-sync,

Many RPi users don't need an online connection, so these computers are designed to work offline, and not worry about correct time, and I find that useful. I have two Raspberry Pi computers, and I have often seen that when these are connected to the network, it can take several minutes before the time gets corrected, upto 10 minutes.

Point 2: Your power cut was 35 minutes long, your MX did not recognise those 35 minutes were missing, and your computer took around 10 minutes before it got the correct time from a NTP clock source. In my experience, too, MX has too much reliance on computer clock time being correct. This is why I am not keen on the "start up delay" and "wait for network" approach taken by MX as a way to cope with wrong time issues. It is sad that MX does not identify that there has been a problem when it processes that sudden time jump. If anyone has a suggestion for how MX could be changed to cope better with wrong time issues, I am sure lots of people would benefit.

I don't want anyone to think I have the expertise to say what the solution is. I have read a book by Eben Upton who was involved in the creation of these computers back in 2006, and his book does briefly mention that their Linux did require quite a lot of tailoring because of the hardware restrictions. So I don't know whether there are other configuration changes needed on my computer that I have not discovered that could lessen the effects in future.

You could discover online how to remove the Raspberry Pi fake-hwclock package, and edit out reference to it in various startup scripts, as I did after I installed a Uninterruptible Power Supply (copes with short power issues, and shuts MX before battery back-up ends) and Real Time Clock (runs only while powered) on my RPi that runs MX. Although I don't want another long term power cut to test it out, all I have is some hope my MX will cope better in future.

[flort]

Thanks for the detailed information regarding this issue. It's the first time I've seen this occur and there's obviously no simple fix. I'll probably just keep an eye on it and if this happens in the future I'll restore one of my backups and then allow the system to catch-up.

Regards,
Trevor

[mcrossley]

Another solution is to add a hardware RTC, then the clock will always be correct at boot time before the network comes up, I do that with all my "important" rPis, it's not very expensive. Otherwise the rPi restarts with the time that it stopped.

[flort]

I opted for adding a hardware RTC so hopefully no more issues. I did receive some information from the supplier about configuring the RPi to delay a service until timesync has occurred. I haven't tried their suggestion but thought I'd post it here in case it was of interest to anyone.

Code: Select all

Ensure systemd-timesyncd.service is being used:
sudo systemctl enable --now systemd-timesyncd.service

Configure time-sync.target to wait until time is synchronised:
sudo systemctl enable --now systemd-time-wait-sync.service

Edit your desired service settings:
sudo systemctl edit <name>.service

Add/edit:
[Unit]
After=time-sync.target
Wants=time-sync.target

Test by rebooting and then from terminal run:
sudo journalctl -b \
-u systemd-timesyncd.service \
-u systemd-time-wait-sync.service \
-u time-sync.target \
-u <name>.service

The log should show entries that confirm the service was only started after time was sync’d.

Regards,
Trevor

[mcrossley]

Hi Trevor, thanks for the service file tips, I'll add the After and Wants to mine.