Welcome to the Cumulus Support forum.
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4019) - 03 April 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4019) - 03 April 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
CreateMissing utility
Moderator: mcrossley
-
- Posts: 76
- Joined: Fri 11 Mar 2011 2:04 am
- Weather Station: Ecowitt GW1103
- Operating System: windows 11 Pro/Raspberry pi 4
- Location: Brunswick Australia
CreateMissing utility
The CreateMissing utility is great. I recently spent a couple of days using excel to fix the dayfile but this program found and corrected about 20 records out of 4000 since Feb 2010. The program did not add any entries from this month March 2021, it removed removed them. I added them back using notepad from the sav file.
Of course the next project for somebody is one that fixes the records automatically instead of by hand.
Of course the next project for somebody is one that fixes the records automatically instead of by hand.
-
- Posts: 76
- Joined: Fri 11 Mar 2011 2:04 am
- Weather Station: Ecowitt GW1103
- Operating System: windows 11 Pro/Raspberry pi 4
- Location: Brunswick Australia
Re: CreateMissing utility
The utility does not add the sunshine hours. My rollover is 9am. When the rollover occurs the sunshine is taken at midnight. This problem also was a problem add missing records in Cumulus1 dayfile.
- mcrossley
- Posts: 12766
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: CreateMissing utility
It *should* be working, for 9am rollover the utility should be taking the hours of sunshine at midnight. It uses two date/time ranges, one for the met day and one for the solar day, obviously if you use a midnight rollover these will be the same.
I'll run another test to see if I can reproduce the issue...
I'll run another test to see if I can reproduce the issue...
- mcrossley
- Posts: 12766
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: CreateMissing utility
Found it, it was extracting the sunshine data OK, but I had left a bit of code in that always wrote a "0" to the day file when using 9a rollover
Version 1.0.2 is now available for download to fix this.
Version 1.0.2 is now available for download to fix this.
- HansR
- Posts: 5965
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: CreateMissing utility
To be redone by everybody or only when using 9 rollover?
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
- mcrossley
- Posts: 12766
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: CreateMissing utility
It only affects 9am rollover, and only if you are adding missing days of course.
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: CreateMissing utility
Basically just an observation from a test to clarify things....
Just tried this on a migrated dataset that contains a dayfile dating back to 2016, but the station data on one install started June 2021.
Looking to fill in these missing records, June 25 to July 5.
https://w2.inverellit.com/daily.php
What I ended up with was new dayfile with only records 64 Records, not the some 1000's of the original.
That was not unexpected as the test directory I worked in only contained Jun, Jul & Aug21log.txt.
The actual live data for the above URL contains all month logs back to 2016, but with that missing June - July block.
Primarily I was wondering what the utility actually did.
Create NEW Missing entries in an Existing file, or create a new file from scratch.
Test proved it is the latter.
I hazard a guess that to do it as the former example would be a totally different code write.
Looks like the solution in my case is to copy the relevant blocks between month logs & either run the utility against the full data set,
or skip that part & copy required dayfile lines from one data set to another.
History of the situation is I had been running two test installs when I first got my GW1000.
Just tried this on a migrated dataset that contains a dayfile dating back to 2016, but the station data on one install started June 2021.
Looking to fill in these missing records, June 25 to July 5.
https://w2.inverellit.com/daily.php
What I ended up with was new dayfile with only records 64 Records, not the some 1000's of the original.
That was not unexpected as the test directory I worked in only contained Jun, Jul & Aug21log.txt.
The actual live data for the above URL contains all month logs back to 2016, but with that missing June - July block.
Primarily I was wondering what the utility actually did.
Create NEW Missing entries in an Existing file, or create a new file from scratch.
Test proved it is the latter.
I hazard a guess that to do it as the former example would be a totally different code write.
Looks like the solution in my case is to copy the relevant blocks between month logs & either run the utility against the full data set,
or skip that part & copy required dayfile lines from one data set to another.
History of the situation is I had been running two test installs when I first got my GW1000.
- A PC based install that started with an empty dataset and
- A pi based install that ran over a copy of my VP/2 data.
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
- mcrossley
- Posts: 12766
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: CreateMissing utility
No, the purpose is to create missing entries. It sounds like it could not read your day file for some reason and only created entries from the monthly log files it had available to it.
Lots of people have used it successfully.
Was there any clue in the output from the utility?
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: CreateMissing utility
Hi Mark,
Not sure if this is of ANY use, but I hadn't closed the command prompt from a few ways ago.
Phil.
Not sure if this is of ANY use, but I hadn't closed the command prompt from a few ways ago.
Phil.
You do not have the required permissions to view the files attached to this post.
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: CreateMissing utility
This is the dataset I've been working on.
Think.... I patched up the month logs between about June 24 & July 8???
Phil
Too big to upload here, but have placed it at w2.inverellit.com/data.7z
Think.... I patched up the month logs between about June 24 & July 8???
Phil
Too big to upload here, but have placed it at w2.inverellit.com/data.7z
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
- mcrossley
- Posts: 12766
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: CreateMissing utility
Thanks, I ran v1.1.0 against your data and got something very different to you!
Could you attach the corresponding log file from the MXdiags folder too please?
The resulting dayfile is attached. I used a made up Cumulus.ini file with my units (Celsius, mph, mm).
Looking at the console output, if failed to find the first day of every month in the log files. Odd as again when I ran it I didn't get that issue.
Your date format is different from mine and slightly unusual in that the day does not have a leading zero but the month does. However if that were a problem I'd expect it to be the same on days 1-9 each month.
Could you attach the corresponding log file from the MXdiags folder too please?
The resulting dayfile is attached. I used a made up Cumulus.ini file with my units (Celsius, mph, mm).
Looking at the console output, if failed to find the first day of every month in the log files. Odd as again when I ran it I didn't get that issue.
Your date format is different from mine and slightly unusual in that the day does not have a leading zero but the month does. However if that were a problem I'd expect it to be the same on days 1-9 each month.
You do not have the required permissions to view the files attached to this post.
-
- Posts: 884
- Joined: Sat 16 Jul 2016 11:59 pm
- Weather Station: Davis VP2+ & GW1000 (Standalone)
- Operating System: Win10 Pro / rPi Buster
- Location: Australia
Re: CreateMissing utility
I'll need to get my head back to the actual data sets I used.
I did have to merge a few month records also.
I have GW1000 data being read by both a PC & a pi & both had the odd gap due to experimenting.
In the meantime here's the logs from the runs I did.
As far as my ini goes.
Australia as the region.
9:00am roll over.
Metric unit with Wind in Km/h.
The dayfiles I currently looking at all seem post patched version & notably missing was the August 1st record, which has in the middle of the gap.
I've actually stolen that line from my VP/2 data at this point.
I did have to merge a few month records also.
I have GW1000 data being read by both a PC & a pi & both had the odd gap due to experimenting.
In the meantime here's the logs from the runs I did.
As far as my ini goes.
Australia as the region.
9:00am roll over.
Metric unit with Wind in Km/h.
The dayfiles I currently looking at all seem post patched version & notably missing was the August 1st record, which has in the middle of the gap.
I've actually stolen that line from my VP/2 data at this point.
You do not have the required permissions to view the files attached to this post.
:Now: :Today/Yesterday:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
Main Station Davis VP2+ Running Via Win10 Pro.
Secondary Stations, Ecowitt HP2551/GW1000 Via rPi 3 & 4 Running Buster GUI.
:Local Inverell Ecowitt Station: :Remote Ashford Ecowitt Station:
- mcrossley
- Posts: 12766
- Joined: Thu 07 Jan 2010 9:44 pm
- Weather Station: Davis VP2/WLL
- Operating System: Bullseye Lite rPi
- Location: Wilmslow, Cheshire, UK
- Contact:
Re: CreateMissing utility
OK, I see a bug in Create Missing when using 9am log rollover. It is something I struggled with at the time (to do with solar data being midnight to midnight)
- vpokroglo
- Posts: 101
- Joined: Thu 24 Feb 2011 7:45 pm
- Weather Station: WS-2350
- Operating System: Raspberry Pi 2
- Location: Slovenija
- Contact:
Re: CreateMissing utility
I get following error:
and also weird first dayfile record:
Code: Select all
Unhandled Exception:
System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.Collections.Generic.List`1[T].get_Item (System.Int32 index) [0x00009] in <533173d24dae460899d2b10975534bb0>:0
at CreateMissing.Program.Main () [0x004c7] in <34f79f08e1394bf2a57a8dc1e6c8dfcf>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.Collections.Generic.List`1[T].get_Item (System.Int32 index) [0x00009] in <533173d24dae460899d2b10975534bb0>:0
at CreateMissing.Program.Main () [0x004c7] in <34f79f08e1394bf2a57a8dc1e6c8dfcf>:0
Code: Select all
First dayfile record: 31. 12. 9999
Records Began Date : 1. 01. 2011
-
- Posts: 2477
- Joined: Wed 08 Jun 2011 11:19 am
- Weather Station: Davis Vantage Pro 2 + Ecowitt
- Operating System: GNU/Linux Ubuntu 22.04 LXC
- Location: Alcaston, Shropshire, UK
- Contact:
Re: CreateMissing utility
Could you zip up your dayfile.txt and attach it to a reply to this topic, please? It may be that there's something not quite right with it.