Welcome to the Cumulus Support forum.

Latest Cumulus MX release v3.0.2 build 3052 - 13 September 2019.
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014

See the Wiki for downloading either version.

WDConverter

Discussion of version 3 of Cumulus, which runs on Windows, Linux, and OS X. All Cumulus MX queries in here, please.

Moderator: mcrossley

Post Reply
92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

WDConverter

Post by 92merc » Tue 02 Jul 2019 4:52 pm

When's the last time any one has used the WDConverter?

I am running both WD and Cumulus. But I have the daytime file PHP file on my site to read that end of day file. I really like that. What I was able to do is take my old data from the VWS days and import it into WD. My goal is to get all of my data back from 2004 until 2017 when I started Cumulus.

When I'm entering in all of my US units into the converter, point to the file, and save, I get in the log file "022017lg.txt line 1: ''79.0'' is not a valid integer value".

The first couple of lines of my data file are:
01 02 2017 00 05 13.4 78.0 7.8 30.260 7.0 11.0 325 0 0.000 0.000 0.000 13.4
01 02 2017 00 10 13.3 79.0 8.0 30.260 10.0 14.0 317 0 0.000 0.000 0.000 13.3

I've tried renaming the files like I saw in a previous post, but to no avail. Hoping someone has recently done a conversion that can give me some assistance.

User avatar
beteljuice
Posts: 2714
Joined: Tue 09 Dec 2008 1:37 pm
Weather Station: None !
Operating System: ex XP SP3 - now W10 laptop grr
Location: Dudley, West Midlands, UK

Re: WDConverter

Post by beteljuice » Tue 02 Jul 2019 10:51 pm

You don't make it clear if the couple of lines from the 'data' and the 'log' file are the same thing.

However: " line 1: "79.0'' is not a valid integer value". Is a correct statement ! - because of the quotes it is a string.

I don't know about line #1 as the figure appears in line#2, or is the log just an error report as things 'cycle' ?
Image
......................Imagine, what you will KNOW tomorrow !

freddie
Posts: 467
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2
Operating System: GNU/Linux (Ubuntu 18.04)
Location: Dorrington, Shropshire
Contact:

Re: WDConverter

Post by freddie » Wed 03 Jul 2019 4:09 am

I don't think that's quite correct, beteljuice. The error message has placed quotes around the value 79.0 to highlight it. But the error message is accurate in that 79.0 isn't a valid integer value - it is a floating point value. 79 would be an integer value. So the converter software didn't expect to find a floating point value at that position.
Freddie
Image

User avatar
beteljuice
Posts: 2714
Joined: Tue 09 Dec 2008 1:37 pm
Weather Station: None !
Operating System: ex XP SP3 - now W10 laptop grr
Location: Dudley, West Midlands, UK

Re: WDConverter

Post by beteljuice » Wed 03 Jul 2019 9:39 am

... it is a floating point value
:bash: "fools rush in"

Edit: Re-think, but why is it happy with 78.0 in the previous line ??
Last edited by beteljuice on Wed 03 Jul 2019 2:11 pm, edited 1 time in total.
Image
......................Imagine, what you will KNOW tomorrow !

92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

Re: WDConverter

Post by 92merc » Wed 03 Jul 2019 1:52 pm

I'll see if I can take that one log file and open it up in Excel to change that column to a non-decimal number. If that works, I can live with that.

water01
Posts: 1549
Joined: Sat 13 Aug 2011 9:33 am
Weather Station: WH1081
Operating System: Windows 7 64bit HP SP1
Location: Somewhere in the USA
Contact:

Re: WDConverter

Post by water01 » Wed 03 Jul 2019 5:17 pm

You would be better using an Editor such as Notepad++ https://notepad-plus-plus.org/ as this preserves all ASCII notations. plus Excel adds all sorts of extraneous Microsoft stuff.

92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

Re: WDConverter

Post by 92merc » Wed 03 Jul 2019 7:01 pm

Problem with that is there is almost 20k lines for each log file. Editing all those lines will be time killing.

I did open them up in Excel. What's weird is the 79.0 opens as 79. It dropped the decimal. I'll have to play around with it to see if Excel can export it back out like that.

92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

Re: WDConverter

Post by 92merc » Wed 03 Jul 2019 7:17 pm

Actually, I think I figured out a workable solution. I did a Macro in NP++. It smoked that extra .0 out, then went to the next line. It actually converted most the data.

I didn't notice that down around lines 11979, that number mysteriously changed to not having the .0. So I'll have to visually scan the logs and see if there are weird anomalies like that. But I think that will be a workable solution.

Next thing, is what do I do with the resulting log file? I see I can plop the file into that data folder. But can those new logs be "processed" to make it into the "dayfile.txt" file?

User avatar
PaulMy
Posts: 1936
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis Vantage Pro 2+ w/Envoy USB
Operating System: Windows-8
Location: Komoka, ON Canada
Contact:

Re: WDConverter

Post by PaulMy » Wed 03 Jul 2019 7:38 pm

Glad you're making progress. I was looking for a past discussion on using Notepad++ RegEx (RegularExpression) plugin that helped me a few years to edit dayfile.txt. You may be past this but here is the link viewtopic.php?f=4&t=15949&p=122214#p122214

Enjoy,
Paul

92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

Re: WDConverter

Post by 92merc » Wed 03 Jul 2019 9:33 pm

Thanks. I'll look into RegEx.

92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

Re: WDConverter

Post by 92merc » Sun 07 Jul 2019 12:07 am

Just as an FYI, I got the REGEX figured out with some assistance. I used ^.{17}\-{0,1}\d+\.\d+\h-{0,1}\K(\d+)\.0 with a replace with /1.

So I converted the missing parts of Feb2017, and added Jan2017 files to my data folder. I shutdown Cumulus while posting those files. Is there anything I have to do to get Cumulus to look at the logs and add the results to my dayfile.txt file? Trying to get a handle on the process before I add more data for the other years.

User avatar
PaulMy
Posts: 1936
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis Vantage Pro 2+ w/Envoy USB
Operating System: Windows-8
Location: Komoka, ON Canada
Contact:

Re: WDConverter

Post by PaulMy » Sun 07 Jul 2019 1:26 am

Is there anything I have to do to get Cumulus to look at the logs and add the results to my dayfile.txt file?
Cumulus 1 has the Create Missing in the Edit Dayfile.txt but that is not available in CumulusMX. You could, if you're brave, run Cumulus 1 and have it create the missing data. I have used that edit feature in the past.

Enjoy,
Paul

92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

Re: WDConverter

Post by 92merc » Sun 07 Jul 2019 2:03 am

HA HA! Freakin' A! Thanks Paul. I did it.

https://www.bismarckweather.net/dayfile2.php

For anyone else, I threw a copy of 1.9.4 on my PC. Copied the data folder over. Plopped a couple of new log files I had converted. In Cumulus 1.94., Edit, Dayfile.Txt. Had it "Create Missing". I then took the data that was added to that 1.94. version of the dayfile.txt, Edited it with Notepad++ and added the missing data. I've filled in my missing data to round out 2017.

Now I just need to convert the 2004-20016 data and put that up. Thanks for the help.

User avatar
PaulMy
Posts: 1936
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis Vantage Pro 2+ w/Envoy USB
Operating System: Windows-8
Location: Komoka, ON Canada
Contact:

Re: WDConverter

Post by PaulMy » Sun 07 Jul 2019 2:33 am

You're very welcome.

Just a reminder,
The usual dayfile.txt has the days' actual highs and lows.
The monthly log files gives the actual data at each log interval, so may have missed a daily high or low if that occurred outside of the actual log recording. However, using the Create Missing to populate dayfile.txt from the monthly log is better than nothing.

Enjoy,
Paul

92merc
Posts: 52
Joined: Wed 15 Feb 2017 6:07 pm
Weather Station: Davis VP2
Operating System: Windows 10 Pro
Location: Bismarck, ND USA
Contact:

Re: WDConverter

Post by 92merc » Sun 07 Jul 2019 8:10 pm

I was able to get all of my data converted into the dayfile. The one thing I did notice is the windrun value is off. Just guessing, but it's almost a factor of 2 or more. But the rest of the data looks good.

I'm satisfied. I got most of what I was looking to do. There are a few days in there where WUnderground was missing data. Not much I can do about that. I did have some of the data still from my old VWS to import into WD. But not enough to fill the single day holes.

Thanks for the assistance.

Post Reply