CreateMissing utility

[BrunswickWeather]

The CreateMissing utility is great. I recently spent a couple of days using excel to fix the dayfile but this program found and corrected about 20 records out of 4000 since Feb 2010. The program did not add any entries from this month March 2021, it removed removed them. I added them back using notepad from the sav file.
Of course the next project for somebody is one that fixes the records automatically instead of by hand.

[BrunswickWeather]

The utility does not add the sunshine hours. My rollover is 9am. When the rollover occurs the sunshine is taken at midnight. This problem also was a problem add missing records in Cumulus1 dayfile.

[mcrossley]

It *should* be working, for 9am rollover the utility should be taking the hours of sunshine at midnight. It uses two date/time ranges, one for the met day and one for the solar day, obviously if you use a midnight rollover these will be the same.

I'll run another test to see if I can reproduce the issue...

[mcrossley]

Found it, it was extracting the sunshine data OK, but I had left a bit of code in that always wrote a "0" to the day file when using 9a rollover

Version 1.0.2 is now available for download to fix this.

[HansR]

To be redone by everybody or only when using 9 rollover?

[mcrossley]

It only affects 9am rollover, and only if you are adding missing days of course.

[Phil23]

Basically just an observation from a test to clarify things....

Just tried this on a migrated dataset that contains a dayfile dating back to 2016, but the station data on one install started June 2021.
Looking to fill in these missing records, June 25 to July 5.

https://w2.inverellit.com/daily.php

What I ended up with was new dayfile with only records 64 Records, not the some 1000's of the original.
That was not unexpected as the test directory I worked in only contained Jun, Jul & Aug21log.txt.

The actual live data for the above URL contains all month logs back to 2016, but with that missing June - July block.

Primarily I was wondering what the utility actually did.

Create NEW Missing entries in an Existing file, or create a new file from scratch.

Test proved it is the latter.

I hazard a guess that to do it as the former example would be a totally different code write.

Looks like the solution in my case is to copy the relevant blocks between month logs & either run the utility against the full data set,
or skip that part & copy required dayfile lines from one data set to another.

History of the situation is I had been running two test installs when I first got my GW1000.

• A PC based install that started with an empty dataset and

• A pi based install that ran over a copy of my VP/2 data.

[mcrossley]

Primarily I was wondering what the utility actually did.

Create NEW Missing entries in an Existing file, or create a new file from scratch.

Test proved it is the latter.

No, the purpose is to create missing entries. It sounds like it could not read your day file for some reason and only created entries from the monthly log files it had available to it.

Lots of people have used it successfully.

Was there any clue in the output from the utility?

[Phil23]

Hi Mark,

Not sure if this is of ANY use, but I hadn't closed the command prompt from a few ways ago.

Phil.

[Phil23]

This is the dataset I've been working on.
Think.... I patched up the month logs between about June 24 & July 8???

Phil

Too big to upload here, but have placed it at w2.inverellit.com/data.7z

[mcrossley]

Thanks, I ran v1.1.0 against your data and got something very different to you!

Could you attach the corresponding log file from the MXdiags folder too please?

The resulting dayfile is attached. I used a made up Cumulus.ini file with my units (Celsius, mph, mm).

Looking at the console output, if failed to find the first day of every month in the log files. Odd as again when I ran it I didn't get that issue.

Your date format is different from mine and slightly unusual in that the day does not have a leading zero but the month does. However if that were a problem I'd expect it to be the same on days 1-9 each month.

[Phil23]

I'll need to get my head back to the actual data sets I used.
I did have to merge a few month records also.
I have GW1000 data being read by both a PC & a pi & both had the odd gap due to experimenting.

In the meantime here's the logs from the runs I did.

As far as my ini goes.
Australia as the region.
9:00am roll over.
Metric unit with Wind in Km/h.

The dayfiles I currently looking at all seem post patched version & notably missing was the August 1st record, which has in the middle of the gap.
I've actually stolen that line from my VP/2 data at this point.

[mcrossley]

OK, I see a bug in Create Missing when using 9am log rollover. It is something I struggled with at the time (to do with solar data being midnight to midnight)

[vpokroglo]

I get following error:

Code: Select all

Unhandled Exception:
System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
  at System.Collections.Generic.List`1[T].get_Item (System.Int32 index) [0x00009] in <533173d24dae460899d2b10975534bb0>:0 
  at CreateMissing.Program.Main () [0x004c7] in <34f79f08e1394bf2a57a8dc1e6c8dfcf>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
  at System.Collections.Generic.List`1[T].get_Item (System.Int32 index) [0x00009] in <533173d24dae460899d2b10975534bb0>:0 
  at CreateMissing.Program.Main () [0x004c7] in <34f79f08e1394bf2a57a8dc1e6c8dfcf>:0 

and also weird first dayfile record:

Code: Select all

First dayfile record: 31. 12. 9999
Records Began Date  : 1. 01. 2011

[freddie]

Could you zip up your dayfile.txt and attach it to a reply to this topic, please? It may be that there's something not quite right with it.