Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

It would be nice to save the lightning data to a file

Please DO NOT use this to publish your entire wish. This Forum is for specific suggestions to enhance the usability of Cumulus MX for all users, NOT your personal requirements. Please check this forum and the rejected forum to make sure you are NOT posting a DUPLICATE suggestion. It will be heavily monitored by Admin and Mark Crossley to determine the feasibility and the difficulty of the suggestion. Those Topics that are deemed inadmissible will moved to the rejected Forum. The remaining Topics will be the Accepted list of future developments, and when our voluntary development group adds it to a build, the build number will be added to the Topic title.
wicher
Posts: 2
Joined: Thu 12 Aug 2021 8:00 pm
Weather Station: GW1000 433/868MHz
Operating System: Ubuntu 20.04
Location: Poland

It would be nice to save the lightning data to a file

Post by wicher »

All data collected by the weather station is saved by CMX on the local computer, except lightning data. The last lightning data is displayed only on the Extra sensors → Lightning screen.

They propose to add to CMX a record of lightning data in a separate file containing 3 columns:
1 - Number od strikes (1 2 3 4 ..)
2 - Date and Time of strikes e.g 05.05.22 17:15:36
3 - Distance of last strike .

A new file e.g. Ligth202205.txt would be created every month.
I hope Mark C will consider this suggestion
Mapantz
Posts: 1774
Joined: Sat 17 Dec 2011 11:55 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 11 x64
Location: Dorset - UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by Mapantz »

wicher wrote: Thu 05 May 2022 8:37 pm All data collected by the weather station is saved by CMX on the local computer, except lightning data. The last lightning data is displayed only on the Extra sensors → Lightning screen.

They propose to add to CMX a record of lightning data in a separate file containing 3 columns:
1 - Number od strikes (1 2 3 4 ..)
2 - Date and Time of strikes e.g 05.05.22 17:15:36
3 - Distance of last strike .

A new file e.g. Ligth202205.txt would be created every month.
I hope Mark C will consider this suggestion
The problem I see there, is that if you don't record a strike for 280 days (as I did) then you'll have a text file full of lines saying the same date, time and distance.
Image
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: It would be nice to save the lightning data to a file

Post by HansR »

Mapantz wrote: Thu 05 May 2022 9:46 pm
wicher wrote: Thu 05 May 2022 8:37 pm All data collected by the weather station is saved by CMX on the local computer, except lightning data. The last lightning data is displayed only on the Extra sensors → Lightning screen.

They propose to add to CMX a record of lightning data in a separate file containing 3 columns:
1 - Number od strikes (1 2 3 4 ..)
2 - Date and Time of strikes e.g 05.05.22 17:15:36
3 - Distance of last strike .

A new file e.g. Ligth202205.txt would be created every month.
I hope Mark C will consider this suggestion
The problem I see there, is that if you don't record a strike for 280 days (as I did) then you'll have a text file full of lines saying the same date, time and distance.
@wicher: second that, I have requested this before.
@mapantz: no, that is not necessary. If you only log recorded strikes, distance and number of, when you read, you can fill in the empty ones if you need them. Not recorded means empty. Saves a lot of space indeed.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
mcrossley
Posts: 12685
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by mcrossley »

Unlikely to happen in the near future, but you can always create your own log file using bash.

Just create a web tag file file with the data you want in it, then set the interval program in CMX to do something like....

Code: Select all

cat webtagfile >> lightning.log
You could even get more creative and put the monthly file name in another web tag file and use that as the destination, something like...

Code: Select all

while IFS= read -r file; do
  cat datawebtagfile >> "$file"
done < "filenamewebtagfile"
wicher
Posts: 2
Joined: Thu 12 Aug 2021 8:00 pm
Weather Station: GW1000 433/868MHz
Operating System: Ubuntu 20.04
Location: Poland

Re: It would be nice to save the lightning data to a file

Post by wicher »

mcrossley wrote: Fri 06 May 2022 8:02 am Unlikely to happen in the near future, but you can always create your own log file using bash.
I'm a mathematician and I deal with the processing of meteorological data. I do not send data to any website. I need local data.
If you do not have time now, maybe in the future an idea about it:
If (new lightning data appears - > save it to a file)
mcrossley wrote: Fri 06 May 2022 8:02 am Just create a web tag file file with the data you want in it, then set the interval program in CMX to do something like....
....
You could even get more creative and put the monthly file name in another web tag file and use that as the destination, something like...
......
My guess is you are suggesting some modifications to the source program. Unfortunately, my knowledge in this field is small.
User avatar
mcrossley
Posts: 12685
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by mcrossley »

wicher wrote: Fri 06 May 2022 9:19 pm My guess is you are suggesting some modifications to the source program.
No, this is standard stuff you can do with CMX as it is. When I have time I can flesh it out a bit more for you, though I think Hans should be able to make sense of it?

It will create a log file with lots of entries the same though - like the other log files.
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: It would be nice to save the lightning data to a file

Post by HansR »

mcrossley wrote: Fri 06 May 2022 10:11 pm ...though I think Hans should be able to make sense of it?
Actually I was kind of puzzled to what you meant by it, but as I am currently occupied with some personal matters and will be away soon for some three weeks, there is not much to be expected from me on the short term. Sorry. May look at it again when I am back.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
philpugh
Posts: 380
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by philpugh »

I do this in a slightly different (and more complex?) manner.

I use a MariaDB for CumulusMX data storage as this allows more flexibility in generating reports.

I have created a table to log all ExtraSensors (I have Ecowitt Internal temp sensors, WH45 indoor Air Quality and CO2, and WH57 Lightning detector).

Using the CumulusMX console I create a regular MySQL update in MySQL Settings > Custom Upload Minutes Interval which has a five minute update setting.

The MySQL command is :-
INSERT INTO ExtraSensors (`ET1`, `EH1`, `ET2`, `LightStrikes`, `A25`, `A25A`, `A10`, `A10A`, `AT`, `AH`, `CO2`, `CO2A`) VALUES ('<#ExtraTemp1>', '<#ExtraHum1>', '<#ExtraTemp2>', '<#LightningStrikesToday>', '<#CO2-pm2p5>', '<#CO2-pm2p5-24h>', '<#CO2-pm10>', '<#CO2-pm10-24h>', '<#CO2-temp>', '<#CO2-hum>', '<#CO2>', '<#CO2-24h>');

I also have a piece of Python code running on my Raspberry Pi, just after midnight, that gets the total lightning for the previous day and writes that to a second table (ESDaily) along with Max and Min values for the other extra sensors. Finally the Python code creates a .json file with the total daily lightning strikes for the last 31 days which can be used to create a graph of daily strikes. I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse! I can post the Python code if anyone is really desperate!
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
dazza1223
Posts: 860
Joined: Sun 25 Jan 2015 8:41 pm
Weather Station: Davis Vantage Pro 2 plus
Operating System: Raspberry pi 4 (4gb)
Location: Worthing
Contact:

Re: It would be nice to save the lightning data to a file

Post by dazza1223 »

philpugh wrote: Wed 25 May 2022 12:10 pm I do this in a slightly different (and more complex?) manner.

I use a MariaDB for CumulusMX data storage as this allows more flexibility in generating reports.

I have created a table to log all ExtraSensors (I have Ecowitt Internal temp sensors, WH45 indoor Air Quality and CO2, and WH57 Lightning detector).

Using the CumulusMX console I create a regular MySQL update in MySQL Settings > Custom Upload Minutes Interval which has a five minute update setting.

The MySQL command is :-
INSERT INTO ExtraSensors (`ET1`, `EH1`, `ET2`, `LightStrikes`, `A25`, `A25A`, `A10`, `A10A`, `AT`, `AH`, `CO2`, `CO2A`) VALUES ('<#ExtraTemp1>', '<#ExtraHum1>', '<#ExtraTemp2>', '<#LightningStrikesToday>', '<#CO2-pm2p5>', '<#CO2-pm2p5-24h>', '<#CO2-pm10>', '<#CO2-pm10-24h>', '<#CO2-temp>', '<#CO2-hum>', '<#CO2>', '<#CO2-24h>');

I also have a piece of Python code running on my Raspberry Pi, just after midnight, that gets the total lightning for the previous day and writes that to a second table (ESDaily) along with Max and Min values for the other extra sensors. Finally the Python code creates a .json file with the total daily lightning strikes for the last 31 days which can be used to create a graph of daily strikes. I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse! I can post the Python code if anyone is really desperate!
hey yes i wouldn't mind using your Python code it wood come in handy thanks
Have fun and keep learning

dazza :D

https://www.davisworthing.co.uk

Image
User avatar
philpugh
Posts: 380
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by philpugh »

Attached is the python I use - it requires Python 3 and is not the best example! I found that Python has some limitations in handling creation of json files hence a funny bit of code towards the end.

I run this code via a CRONTAB entry

30 2 * * * /usr/bin/python3 /home/pi/esdaily.py >> /home/pi/esdaily.log 2>&1

which creates a log file if any issues arise - there is very little in the way of error condition handling in the code.

Enjoy and use at your own risk.
You do not have the required permissions to view the files attached to this post.
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
mcrossley
Posts: 12685
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by mcrossley »

philpugh wrote: Wed 25 May 2022 12:10 pm I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse!
Sounds like you needed a stored procedure in your MySQL db?
I use them for updating tables like sunshine by hour of day, or similar for AQI but that is by day of week.
The stored procedures are triggered by the post insert event on the dayfile table.
freddie
Posts: 2433
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by freddie »

philpugh wrote: Wed 25 May 2022 12:10 pm I also have a piece of Python code running on my Raspberry Pi, just after midnight, that gets the total lightning for the previous day and writes that to a second table (ESDaily) along with Max and Min values for the other extra sensors. Finally the Python code creates a .json file with the total daily lightning strikes for the last 31 days which can be used to create a graph of daily strikes. I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse! I can post the Python code if anyone is really desperate!
You could probably do the ESdaily update using SQL. I have an identical setup to yours and do end-of-day max/min/sum stuff using the daily mysql update.
I've written a spring-based API to my database and get the equivalent of your JSON file via that route.
Freddie
Image
User avatar
philpugh
Posts: 380
Joined: Tue 24 May 2011 8:34 am
Weather Station: See Signature
Operating System: Debian 11 (RPi4)
Location: Antrobus, Cheshire, UK
Contact:

Re: It would be nice to save the lightning data to a file

Post by philpugh »

Yes I should be able to do both table updates in MySQL via CumulusMX but I would probably still need a bit of python to produce the JSON files?

Cheers
Phil Pugh
GW1000 + WH65/WH57/WH31; HP2551 + WS68/WH40A; GW2001 WittBoy
cumulusutils generating website
http://goosegate.uk/
User avatar
dazza1223
Posts: 860
Joined: Sun 25 Jan 2015 8:41 pm
Weather Station: Davis Vantage Pro 2 plus
Operating System: Raspberry pi 4 (4gb)
Location: Worthing
Contact:

Re: It would be nice to save the lightning data to a file

Post by dazza1223 »

philpugh wrote: Thu 26 May 2022 9:45 am Yes I should be able to do both table updates in MySQL via CumulusMX but I would probably still need a bit of python to produce the JSON files?

Cheers
hey did u get the lighting data in Mysql and the triggers
Have fun and keep learning

dazza :D

https://www.davisworthing.co.uk

Image
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: It would be nice to save the lightning data to a file

Post by HansR »

Using Custom Logging might be the easiest way.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
Post Reply