Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Any bright ideas?

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
water01
Posts: 3215
Joined: Sat 13 Aug 2011 9:33 am
Weather Station: Ecowitt HP2551
Operating System: Windows 10 64bit
Location: Burnham-on-Sea
Contact:

Any bright ideas?

Post by water01 »

Since late May my humidity sensor on my old WH1080 started giving occasional weird readings and gradually they and other readings got worse, so in early September I replaced it with an Ecowitt HP2551. As my luck goes this worked until early December when it too started giving me wild humidity fluctuations (1% to 99% and then back to 1% within 2 minutes).

This was obviously wrong so I contacted Ecowitt and they replaced the sensor by sending me a brand new HP2551 minus the console (very impressed, great customer service). That is now up and we are back to normal!

But if you look back at my history since last May (2021) there are wild fluctuations and obviously Dewpoint reading that CumulusMX has calculated that are nonsensical, plus the records that go with them (-50C Dewpoint yikes!!).

I was thinking that as I have all my monthly logs saved to MySQL server tables I could come up with some bright SQL to go through the table and fix this, and then extract it in CSV comma delimited format and rebuild the monthly logs with more correct data, and then use the editors to fix the records etc.

Anybody else had this problem and how did you solve it and generally can anyone come up with a better solution?
David
Image
LindaFNM
Posts: 151
Joined: Wed 06 Jan 2021 3:01 pm
Weather Station: Ecowitt
Operating System: Raspberry Pi

Re: Any bright ideas?

Post by LindaFNM »

I’m afraid I don’t have any bright ideas at all!! It’s a long, painful hard slog. You could make sure you have the data spike alarm set on CumulusMX and data parameters set. This has saved my bacon quite a bit recently. I would love it if someone else chips in with something worth trying.

I have almost 12 years worth of data of varying questionable accuracy. I started with WSWin (TFANexus) then WeatherLink (Davis VantageVue), then looked at Cumulus and couldn’t figure it at all, added Weather Display all logged and downloaded at intervals from the Vue. I then added to the Ecowitt WS80, WH40 and consoles which meant I entered the 21st century. I then arrived back at CumulusMX.

It all got very interesting as none of the weather programs have standard data, eg wswin holds wind in KPH, Weather Display in knots, and they often calculate derived values slightly differently. Having decided I’d make a go of CumulusMX I started trying to convert data, and added a Raspberry Pi. My first months with the Pi weren’t exactly spectacular as it dropped out with boring regularity (I should have re-imaged it a lot earlier) leading to more missing bits of data. I’d just upgraded the software to Bullseye and got things really running when my WS80 went bad (also replaced by Ecowitt). This, like you had led to massive chunks of missing or silly data. I’m lucky in that the Davis is still running happily and being manually logged.

Now, the crux of what is a boring story. I spent all last winter trying to sort out the data, I made a first pass just to get data in. Went cross eyed on numerous occasions, then decided to try to build an Access database for normalised data. If I have to be honest, I’ve got totally fed up with it all. I think you need to ask yourself how much you care, how much work you want, and how much granularity you need. Finding missing data is not easy, removing silly data is easier. Swapping in and out of .txt and .csv files is a pain. You don’t have a huge amount of data if it’s only from last May so it may well be worth the effort, either just using the log files or via a database. CumulusMX does have the Create Missing utility which can be very useful to calculate derived data from things like temperatures etc.

I’d settled for just getting the daily records accurate and not bothering with the logs, this is coming back to bite me via Cumulusutils which reads the daily logs so not sure what I’ll do from here. Ask yourself if you really need accurate dewpoints and other derived values. I decided that what really matters to me personally is high/low temperatures, rainfall, wind gusts and sunshine. I am lucky in that the Davis logs can be used to compare values but I can’t swop the files in as the Davis logs every 5 minutes, the Ecowitt every minute. Latterly I’ve been using my NOAA reports to cross check, you can also get details from your local METAR station.

Sorry this is so long, and probably not particularly helpful! I hope that others may be able to offer more advice.
User avatar
mcrossley
Posts: 12685
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Any bright ideas?

Post by mcrossley »

David, I have done something similar but different with my database, in that I added data that was missing from earlier versions of Cumulus - like Feels Like.

That was slightly simpler in that the data just wasn't there and I added it where the columns were null.

I first created a MySQL function to calculate the Feels Like value from the temperature, humidity, and wind speed inputs. Then just ran an update statement against all rows that had a null Feels Like value. You could do a similar thing with date/time ranges. The hardest part is creating the functions to do the calculations - the source for CMX will give you the code, it has to be translated to MySQL syntax of course.

Another, possibly simpler option may be to delete the bad values from copies of the log files and see if the CreateMissing utility will recreate them for you?
Once you had the logfiles updated you could re-import them into your database.
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: Any bright ideas?

Post by HansR »

I guess it depends what language you are comfortable with. In the end it all comes down to reading the data (for a certain log time), calculate your target value for that log time and write it back.

You can do it in SQL (I would NEVER do that) or C# (native CMX language) but for the oldies I guess FORTRAN would do very good as well being a line oriented language for scientific calculations. I would seriously consider that if it would still be easily available. The issue is more that you must be clear on what you want to do than how you want to do it.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
Post Reply