Dodgy historical data aproach

[cehbab]

Hi all,

I looked all over, and I couldn't find a simple enough solution for displaying all of my historical data easily, so I built my own way, piggy backing completely off of Cumulus. I have many years of data in backups from various locations I have used Cumulus, in subfolders throughout my Cumulus installation folder, so this solution may not work for anyone else.

This was built in csharp (dotnet3.5), so it should work everywhere the old and new Cumulus works, although I have never used MX.
Source is included, using VisualStudio2008, but it is really just one Program.cs file, so you could rebuild it under any environment.
It is also plain simple jquery and Flot for graphing, so its all free.

The exe looks for ALL *log.txt files of the MONTHYEARlog.txt format, within the exe's folder and all its subfolders.
It only grabs min/max indoor and outdoor temperatures, and rainfall totals, and if duplicate files are found it simply merges them together.
It outputs these found datasets into a history.js javascript file in the same folder as the exe.
This history.js contains two variables, months and years, both just arrays of sets of data.

The history.htm file included simply uses the included jquery and flot dependencies, and the history.js as well and renders them.

To install, its a bit clunky,
1) Copy HistoryToJavascript.exe to you Cumulus folder.
2) Copy web files, history.html and js folder to your destination web site.
3) Configure Cumulus to execute external program HistoryToJavascript.exe, I did it first as Program to generate history.js more quickly, and afterwards set it to execute Daily.
4) Configure Cumulus to copy file history.js to destination website
5) Update all template webfiles; ie web\indexT.htm to include history link, ie:
<td colspan="4" class="td_navigation_bar">:<a href="index.htm">now</a>:...... to include :<a href="history.htm">history</a>:

I use ftp as my destination website, and the first update or daily rollover the history.js file does not exist, so none is copied, but one is created after this first update. During the second update the history.js file exists and is sent to the remote web server.
After that I set the exe to execute only Daily, so the whole of my 50Mb backup *log.txt data is only processed once a day.

Yes its plain and buggy and chunky and dodgy, but it works enough for me.
Zoom with mouse wheel (can't on my laptop!) and pan with click.

If anyone has any suggestions or could point me in a better direction to achieve this then tell me so

Cheers,
cehbab

[gw3atz]

Just for information..........I run MX on raspberry pi. HistoryToJavascript.exe runs (sudo mono History.....) and looks at all the log files. It does NOT however make history.js Have run the program several times. As I say, just for info. I don't know enough programming etc to be able to see were any problem may be.

Cheers............Geoff

[cehbab]

If you place it in the same folder as your Cumulus, and run it from the shell, what info does it spit out ?
Do you have any MMMYYlog.txt files in that folder or below it in subfolders ?

For me, it prints out CWD and then all the matching log files that match MMMYYlog.txt

It might be that the path separator is not working under linux, or some other pathing issue.

Does it print anything out at all, and if so could you post that output here ?

Cheers.

[cehbab]

There was one bad path thing in the code, I looked up mono porting guidelines, and it told me to never use a \ in a path, so I removed it, using their recommended way of doing things.

Re-uploaded for you to try,

Only 2 small changes,
1) change path concatenation to remove hard-coded \, use Path.Combine(...) instead
2) print out that its writing history.js

Cheers.

New file attached.

[gw3atz]

Thanks for the update. It now works and makes the file history.js I now need to make sure that MX runs the .exe file. Only other thing I want to look at is the width of the page. My cumulus site has images down each side which get in the way. The graphs look fine. Not put history.htm in my main menu yet, but you can still look at it. Its the /wx/ site.

Thanks for your time with this nice add on.

Cheers............Geoff

[cehbab]

Awesome, thanks for testing

The history.htm is very plain, hence I called it dodgy .

If you make something a bit more stylish, then give a shout out and upload it back here for me to use

Also, if you want more/different data in that history.js like humidity or whatever other column is available in the logs then tell me so and its easy to add. I only needed those indoor/outdoor temps and rainfall for now, and it keeps the .js file small.

My graph only had a hard-coded 80% width and a left of 10% set in the style in the .htm, quite dodgy

Also if anyone else wants to use this and the dates can't be read because of locale or something, tell me and I'll update it.
I was lazy and parsed the logs for my default date syntax only...

Cheers.

[cehbab]

Geoff,

Because your dataset is so small, only 2 months, I could spot errors in it very easily.
My dataset is over a 5 year span and I didn't notice that it was showing the data incorrectly.

Oh well..

I updated 2 lines in the htm file, to use the correct month offsets for displaying and bounding the x axis scrolling.
I also updated the csharp source to actually use maximum's instead of minimums for the months view. silly bug
The days data is now being displayed in the correct month, and you can scroll back to the beginning, instead of not showing or scrolling to that first month

Looking at your website, using MX which I had not seen before, it is using HighCharts.
If that is already being included by default in Cumulus MX then it should be quite easy to use that graph methodology to show all history quite easily.

My interface is crap, its the history.js that I wanted like yourself.

[cehbab]

Apologies again for another upload,

I restricted the axis to not be able to scroll past unavailable data to the left and right.
I would have updated the last entry over version _c, but it has been downloaded already...
Only history.htm was changed which is quite irrelevant.

Have a happy new years

[gw3atz]

Hi cehbab...................Thanks for the updates. All working ok as far as I can tell. Now need to sort MX to handle the .exe file each day.

Best wishes for new year..

Geoff

[jlmr731]

Seems to work well with my linux system very nice job.
I only have a few months of data so process went quickly, I dont know if you can change to just grab files from data dir and ignore the others, might help with speed for people with years of log files.

Thanks

[cehbab]

Thanks for the feedback,
I've updated it to support args from the command line
Any argument you add it will search in that folder, multiple args, multiple folders.
It still looks for all files MMMYYlog.txt in all subfolders within those given folders.

If you pass no args it does default and searches from the folder context Cumulus is running in.

Also, speed is not that big of an issue if you only run it daily.
Old Cumulus has that option in its configuration screen, I don't know about the new Cumulus MX though.


Cheers.

[kocher]

Cheers Cehbab
The HistoryToJavascript.exe program stops with an error message when reading the first data file: "abr12log.txt"

I suspect that the cause is because the file has:

Decimal Sign: ,
Separator: ;

[cehbab]

Hi Kocher,

Sorry for the delay, could you attach your data file here so I can get it to parse for you ?

Thanks..

cehbab

[kocher]

Thank you very much for your reply.

This is the address of the file "abr12log.txt":

Http://kocher.es/cumulusMX/data/abr12log.txt