4.4.3 b4806 - Ecowitt SD card data backfill

[mcrossley]

What that brief power outage did show was that the GW3000 does not have a battery backed RTC.

Our router takes a little while to reboot, during this time the GW3000 created two new log files for 202501, and added two 5-minute log entries to these files for the same time but with different data...

Code: Select all

2025-01-01 00:01,20.2,...
2025-01-01 00:01,20.1,...

So, it would appear the time does not change until the unit can contact the NTP server?

My suggestion would be the unit took the date/time from the last log entry and incremented from that point in time until it can get an accurate time from the NTP server.

If I were running a GW3000 live it looks like I would have to run off on some sort of mini-UPS - mind you that is our first power glitch for years!

[mcrossley]

I checked the April log again just now, there was another 1 minute "skip"...

Code: Select all

2025-04-11 19:35
2025-04-11 19:41

That is 3 days 2 hours after the reboot.
74 hours = 888 5-min intervals = nothing obvious to me?

[broadstairs]

I've been checking the data in my files for April and on the basis of 1440 records per 24 hours then the record for 23:59 on the 14th April should be record number 20160 but it is not, it is record number 20158. We has lost 1 record by 23:59 on the 8th April and now 2 records by the 14th April, checking the 13th April we still had only lost 1 record and up to the 7th April we had lost none!

Now as far as I can tell there have been no power outages (very unusual here) and certainly no restarts of the GW3000 there were two restarts of CMX one on the 1st April at 17:32 and one on the 10th April at 16:27 but neither restarts show missing records at or soon after these times.

Now this may not be a huge number but there are certainly lost records happening on the SD card using 1 minute intervals and I think there could be more missing by the 30th April . Also as I suspect the writing of records is slipping slightly causing missed records there are doubts in my mind as to whether or not the data being recorded is actually for the minute shown between records! Still worrying for those who are very concerned about accuracy.

Stuart

[paulcheffus]

Hi

I've noticed the timestamps can be constant for a while then have big jumps. Look at 14:36:44 and 14:38:01 or 14:51 and 14:52:51

data.jpg

When I was running with five minute intervals the timestamp didn't jump as frequently but was still moving.

Cheers

Paul

[broadstairs]

Don't understand where those time stamps are coming from as my GW3000 does not show seconds etc. and does not format the date like that.

Stuart

[paulcheffus]

Don't understand where those time stamps are coming from as my GW3000 does not show seconds etc. and does not format the date like that.

Stuart

Hi

This is from the Ecowitt custom upload direct to my website from the GW3000.

Cheers

Paul

[broadstairs]

Ah that explains it thanks.

I'm starting to think that the processor chip in the GW3000 is not powerful enough for what they are trying to do!

Stuart

[broadstairs]

I decided to test out the timings of a custom server on my GW3000 and in just a tad over 24 hours it has slipped by 11 seconds, so started yesterday at +32 seconds past the minute and now this morning it is +43 seconds so 11 seconds later on a 1 minute interval. So it seems like there is no timing against a clock for either writing to the SD card or sending data via a customer server interval.

So both SD card data and Custom Server data is not sent at exactly the same time for each interval consistently.

Out of interest I'm running the same test today on my GW1100 which is not currently in use by any program or sending data to Ecowitt to see if the same thing happens.

Stuart

[broadstairs]

Just to say that my GW1100 has not skipped one second yet today with the Custom Server timing, stayed at +08 seconds all day so far! The GW1100 is accessing the same sensors as the GW3000 so that part of the loading is identical but tomorrow I'll start CMX accessing the GW1100 so adding extra work for it and see if this causes a change in timings!

Stuart

[broadstairs]

Well my GW1100 now has CMX getting HTTP data from it but so far today no slip in the timing of the Custom Server data being sent still every minute at +08 seconds.

On the other hand my GW3000 is still adding 1 second to the Custom Server data about once per hour. Starting on 17th April at +32 seconds each minute and by 19th April had skipped forwards to +00 seconds and is now at +03 seconds.

Now they are both being used by CMX via HTTP downloads, neither is uploading to Ecowitt and both access the identical sensors. So the only loading difference for the GW3000 is the SD card.

Stuart

[paulcheffus]

Well my GW1100 now has CMX getting HTTP data from it but so far today no slip in the timing of the Custom Server data being sent still every minute at +08 seconds.

On the other hand my GW3000 is still adding 1 second to the Custom Server data about once per hour. Starting on 17th April at +32 seconds each minute and by 19th April had skipped forwards to +00 seconds and is now at +03 seconds.

Now they are both being used by CMX via HTTP downloads, neither is uploading to Ecowitt and both access the identical sensors. So the only loading difference for the GW3000 is the SD card.

Stuart

Hi

V1.0.3 of the GW3000 firmware did not show these time slips so it would appear to have been introduced with either v1.0.4 or v1.0.5.

Cheers

Paul

[broadstairs]

I had not tested that earlier f/w versions. I am starting to think that we may have one of two issues, either bad programming or an overload of work with the SD card. Since this is my test install with the GW3000 I might remove the SD card and run to see if the time slips still happen. When I have further test results I'll raise with Ecowitt.

Stuart

[broadstairs]

Well removing the SD card makes no difference to the skipped seconds it still happens.

I'll put some data together and ask Ecowitt what's going on.

Stuart

[mcrossley]

From the little programming I have done on microcontrollers, you would set a hardware interrupt timer to do this sort of thing. The top priority would be processing of incoming data, but you would expect that would take milliseconds per message. An interrupt timer is unaffected by CPU load (as the name suggests it's a hardware generated thing) which would trigger the code to create the data string, the writing to the SD card can be relatively leisurely low priority process in CPU terms.

You would synchronise the timer with the NTP time so the data snapshots occur precisely on the minute.

[broadstairs]

I emailed Ecowitt over the weekend and this morning they replied and said that they had fixed this issue before and asked for the MAC address of it. I pointed out I am not the only one seeing this issue of the shifting timing of both Custom Sever and SD card!

They may have thought they fixed it but certainly not on my device which is running the latest public f/w 1.0.5. I just wonder if they made some hardware changes to later models since mine came from Weatherspares so don't know if this was older stock sent to them prior to any hardware changes?

Stuart