Convert and import logfile

[TMJ]

I have using Weather Display for a couple of year now but thinking to replace it...
How and can i convert and import logfile to Cumulus so i can have history working from 2008..

[steve]

I don't know whether anyone has done that before; a few people have complained that I haven't provided a facility to do it, but I know very little about Weather Display, and I have neither the time nor inclination to do it. The format of the Cumulus log files are in the help and the wiki; you would need to create the 'monthly' log files (i.e. the ones that get written to every few minutes) and then Cumulus could create the rest of the files from those. Sorry I can't be more help.

[TMJ]

OK, thanks for the reply, i have to read the wiki more to se if i can convert the files in excel or similar editor.

[TMJ]

Have now done some test...
I load WD logfile in Excel and with some formulas i get this output :

01/10/11,00:00,8.8,94,07.9,0,1,270,0.0,0.0,1021.5,510.6,0,0,0,0,0,0,0,0,0,0,0,0,0
01/10/11,00:01,8.8,94,07.9,0,1,270,0.0,0.0,1021.6,510.6,0,0,0,0,0,0,0,0,0,0,0,0,0
01/10/11,00:02,8.8,94,07.9,0,1,270,0.0,0.0,1021.8,510.6,0,0,0,0,0,0,0,0,0,0,0,0,0
01/10/11,00:03,8.8,94,07.9,0,1,270,0.0,0.0,1021.4,510.6,0,0,0,0,0,0,0,0,0,0,0,0,0
01/10/11,00:04,8.8,94,07.9,0,1,270,0.0,0.0,1021.6,510.6,0,0,0,0,0,0,0,0,0,0,0,0,0
01/10/11,00:05,8.8,94,07.9,0,1,270,0.0,0.0,1021.6,510.6,0,0,0,0,0,0,0,0,0,0,0,0,0
01/10/11,00:06,8.8,94,07.9,0,2,273,0.0,0.0,1021.7,510.6,0,0,0,0,0,0,0,0,0,0,0,0,0

Do i need 0.0 or 0 in empty or no data field?
And my wind data is with no decimals (0) do i need to convert to 0.0?

[serowe]

Converting from WD isn't very difficult as Brian Hamilton has long made available to format for the data files. You need to process two files if you want to bring them into Cumulus - he has a detailed log file in the format DDMMYYlg.csv where DD is the day number; MM is the month number - 01 = Jan, 02=Feb etc; YY the year in 2 digit format. This file records the minute by minute or whatever period you have it being logged by WD and is a comma delimited file so easily read by Excel. It should also contain all the field headings on line 2 or 3. Typicall this file can be 203 Mb in size. Each month also has a space delimited file available in the format MYYYYlg.txt but this is only a 17 field record and doesn't contain all the data contained in the lg file.

You can either import these files into Excel, manipulate the columns to be the same as Cumulus, and then export it as a CSV file (make sure date formats etc are correct though) or, as I and a few others have done, forget about the Cumulus format (the changes between versions is enough of a headache to consider this) and store ALL of your data in a MySQL database and then generate all of your pages from it. This way you can import any data from any program into the database and do what YOU want. For example, my http://weather.ser.id.au pages all bring data in from a MySQL data base the main, detailed table now has over 2.7 miollion records and every page is calculated and displayed on the fly. If has data from Virtual Weather Station, Weather Display, FreeWx, Cumulus and now my own software as well as 18 years of paper records which have been transcribed. IOW - once I spent the time converting the data into a constant and stable database table, I don't have to worry about where the data comes from. if you import into Cumulus files you need to remember that the format of the detail file changed around v1.9.2 (think that was the version).

So comments like not having the time or inclination are IMHO a crock - ALL other programs available realise that they need to encourage people to use their software and thus have import routines available for just this purpose. (BTW, watch the sharks attack now!)

[steve]

Do i need 0.0 or 0 in empty or no data field?
And my wind data is with no decimals (0) do i need to convert to 0.0?

Either will do.

[nking]

So comments like not having the time or inclination are IMHO a crock - ALL other programs available realise that they need to encourage people to use their software and thus have import routines available for just this purpose. (BTW, watch the sharks attack now!)

I won't attack but I will make a few observations.
• Cumulus is FREE, it’s left entirely to the individual users conscience whether they make a donation.
• Steve originally wrote the software for his own personal use and then made it available for FREE. I don’t think he has ever stated that he intends to make Cumulus a commercial undertaking i.e. He already has a full time job and he gives his FREE time to developing and supporting Cumulus – nothing more and nothing less.
• People have a choice whether to accept or reject Cumulus based upon the understanding that it’s FREE and that Steve decides what time he wants to give to supporting it.
• There are some Cumulus users that are technically skilled and they too have made FREE files and add-ons but, as yet, no-one has provided a conversion routine.

So, your point that “ALL other programs available realise that they need to encourage people to use their software and thus have import routines available.” needs to be viewed with the above points in mind.

You seem to be technically skilled and there really is nothing to stop you providing a FREE add-on to fill this need.

PS - Sorry for going off topic

[TMJ]

OK now i have to test this if my own conversion work, i get back here with my results..

[serowe]

I won't attack but I will make a few observations.
• Cumulus is FREE, it’s left entirely to the individual users conscience whether they make a donation.

Very true - but I never mentioned anything about this.

• Steve originally wrote the software for his own personal use and then made it available for FREE. I don’t think he has ever stated that he intends to make Cumulus a commercial undertaking i.e. He already has a full time job and he gives his FREE time to developing and supporting Cumulus – nothing more and nothing less.

And so do Brian Hamilton and Andy Keir. But again - I never said anything about making it a commercial product - steve has made his view on this very very plain.

• People have a choice whether to accept or reject Cumulus based upon the understanding that it’s FREE and that Steve decides what time he wants to give to supporting it.

Ummm I'm beginning to see a pattern here - maybe my glasses need to be tested again (oh hang on, they just have been) - I didn't make any statement about whether people have a choice to accept or reject any software package.

• There are some Cumulus users that are technically skilled and they too have made FREE files and add-ons but, as yet, no-one has provided a conversion routine.

Ummmm sorry you are simply wrong on this one. There have been a few who have ofered conversion routines for WD and FreeWx. I should know as I have made the offer, in these forums, more than once. Guess what? Guess how many takers I had? None. I did get some abusive private messages and emails telling me to piss off and stop promoting other packages (huh????the routines were for converting FROM WD/FreeWx to Cumulus format) but why let facts get in the way of a good story??

So, your point that “ALL other programs available realise that they need to encourage people to use their software and thus have import routines available.” needs to be viewed with the above points in mind.

Bugger it - I'll say it again then in slightly different terms.

Cumulus and steve are trying to promote themselves as an alternative to the other weather software packages around. Currently it seems that if ANYONE wants to bring their data into Cumulus the only response they are going to get is 'go away, I'm not interested'. Yes I realise Cumulus IS freeware/shareware/donationware (whatever you want to call it). But even the simple minded can realise that instead of telling people to get lost and take their old data elsewhere or forget about utilising it, a one-off conversion/import routine is only going to enhance the product. And yes this is steve's call, but writing or having written for him a one off routine to import data isn't going to be a major task (continual enhancements that continually break, now that's a different story) and it isn;t something needed today or tomorrow. But if users know this WILL be done in the near future gives the potential for users of a product such as WD which keeps getting broken every time an upgrade is done; or FreeWx users because they can no longer use it on 64 bit systems.

But - don't let logical discussions get in the way of talking for someone who won't/can't answer for himself

[/soapbox]

Stephen

[TMJ]

Have some question how to put my data in right order when i convert files from WD to Cumulus.
Please check my image to se if i put it in the right order.

No 7 goest to 2 position i cumulus file wich one should i use.

[serowe]

This is another of those 'anomolies' in steve's files - two fields with, apparently, the same name but contain different information comparing some old data it appears that column 7 is the ongoig gust speed but column 15 is an 'average' (well this is a presumption btw). Try to get steve to answer this if you can.

Note also that you will need to ensure all missing columns contain valid data - by valid I mean a zero or similar otherwise you will have all the latter columns out of place. So in total you need all 25 columns even if some of them are 'blank'.

BTW don;t forget you also need to generate the 'daily' summaries for dayfile.txt.

[GraemeT]

There have been a few who have ofered conversion routines for WD and FreeWx. I should know as I have made the offer, in these forums, more than once.

Stephen, maybe you could to zip up your routines and make them available to people who need them. Perhaps via a link from your weather site?

[serowe]

Graeme - I've made them available to a number of people who have asked for them. The difference being though that I don't convert from WD (or VWS/FreeWx) into Cumulus format but store everything in a MySQL database now - so my routines started off life similar fo daj's Import Cumulus Files routines and have undergone massive changes to allow for the different formats. So a reasonable working knowledge of PHP will be a decided advantage in using them.

But - anyone that wants them, PM me and I will get them to you.

Stephen
(BTW - you should move closer to Mountain Gate - it's 0.7 warmer over this way )

[TMJ]

I have created some testfiles for 3 months and then started cumulus and its working to import missing files but i need to see if the data was in right place and how cumulus calculate record.
I should do some more test before i deside how to do this conversion.

[steve]

No 7 goest to 2 position i cumulus file wich one should i use.

Column 7 in the Cumulus table is the Cumulus-calculated peak gust over the last 10-minutes (i.e. my attempt at a 'useful' gust figure, consistent across all weather stations, which each have their own way of providing a 'gust' figure). Column 15 is simply the the last gust value read from the Fine Offset station. The two fields are labelled in the help according to what they contain.

You might be better putting 'annual rain' into column 12 (the total rain counter). It's what Cumulus uses on Davis stations, and it doesn't reset as often.