Page 1 of 2
It would be nice to save the lightning data to a file
Posted: Thu 05 May 2022 8:37 pm
by wicher
All data collected by the weather station is saved by CMX on the local computer, except lightning data. The last lightning data is displayed only on the Extra sensors → Lightning screen.
They propose to add to CMX a record of lightning data in a separate file containing 3 columns:
1 - Number od strikes (1 2 3 4 ..)
2 - Date and Time of strikes e.g 05.05.22 17:15:36
3 - Distance of last strike .
A new file e.g. Ligth202205.txt would be created every month.
I hope Mark C will consider this suggestion
Re: It would be nice to save the lightning data to a file
Posted: Thu 05 May 2022 9:46 pm
by Mapantz
wicher wrote: ↑Thu 05 May 2022 8:37 pm
All data collected by the weather station is saved by CMX on the local computer, except lightning data. The last lightning data is displayed only on the Extra sensors → Lightning screen.
They propose to add to CMX a record of lightning data in a separate file containing 3 columns:
1 - Number od strikes (1 2 3 4 ..)
2 - Date and Time of strikes e.g 05.05.22 17:15:36
3 - Distance of last strike .
A new file e.g. Ligth202205.txt would be created every month.
I hope Mark C will consider this suggestion
The problem I see there, is that if you don't record a strike for 280 days (as I did) then you'll have a text file full of lines saying the same date, time and distance.
Re: It would be nice to save the lightning data to a file
Posted: Fri 06 May 2022 4:48 am
by HansR
Mapantz wrote: ↑Thu 05 May 2022 9:46 pm
wicher wrote: ↑Thu 05 May 2022 8:37 pm
All data collected by the weather station is saved by CMX on the local computer, except lightning data. The last lightning data is displayed only on the Extra sensors → Lightning screen.
They propose to add to CMX a record of lightning data in a separate file containing 3 columns:
1 - Number od strikes (1 2 3 4 ..)
2 - Date and Time of strikes e.g 05.05.22 17:15:36
3 - Distance of last strike .
A new file e.g. Ligth202205.txt would be created every month.
I hope Mark C will consider this suggestion
The problem I see there, is that if you don't record a strike for 280 days (as I did) then you'll have a text file full of lines saying the same date, time and distance.
@wicher: second that, I have requested this before.
@mapantz: no, that is not necessary. If you only log recorded strikes, distance and number of, when you read, you can fill in the empty ones if you need them. Not recorded means empty. Saves a lot of space indeed.
Re: It would be nice to save the lightning data to a file
Posted: Fri 06 May 2022 8:02 am
by mcrossley
Unlikely to happen in the near future, but you can always create your own log file using bash.
Just create a web tag file file with the data you want in it, then set the interval program in CMX to do something like....
You could even get more creative and put the monthly file name in another web tag file and use that as the destination, something like...
Code: Select all
while IFS= read -r file; do
cat datawebtagfile >> "$file"
done < "filenamewebtagfile"
Re: It would be nice to save the lightning data to a file
Posted: Fri 06 May 2022 9:19 pm
by wicher
mcrossley wrote: ↑Fri 06 May 2022 8:02 am
Unlikely to happen in the near future, but you can always create your own log file using bash.
I'm a mathematician and I deal with the processing of meteorological data. I do not send data to any website. I need local data.
If you do not have time now, maybe in the future an idea about it:
If (new lightning data appears - > save it to a file)
mcrossley wrote: ↑Fri 06 May 2022 8:02 am
Just create a web tag file file with the data you want in it, then set the interval program in CMX to do something like....
....
You could even get more creative and put the monthly file name in another web tag file and use that as the destination, something like...
......
My guess is you are suggesting some modifications to the source program. Unfortunately, my knowledge in this field is small.
Re: It would be nice to save the lightning data to a file
Posted: Fri 06 May 2022 10:11 pm
by mcrossley
wicher wrote: ↑Fri 06 May 2022 9:19 pm
My guess is you are suggesting some modifications to the source program.
No, this is standard stuff you can do with CMX as it is. When I have time I can flesh it out a bit more for you, though I think Hans should be able to make sense of it?
It will create a log file with lots of entries the same though - like the other log files.
Re: It would be nice to save the lightning data to a file
Posted: Sat 07 May 2022 7:18 am
by HansR
mcrossley wrote: ↑Fri 06 May 2022 10:11 pm
...though I think Hans should be able to make sense of it?
Actually I was kind of puzzled to what you meant by it, but as I am currently occupied with some personal matters and will be away soon for some three weeks, there is not much to be expected from me on the short term. Sorry. May look at it again when I am back.
Re: It would be nice to save the lightning data to a file
Posted: Wed 25 May 2022 12:10 pm
by philpugh
I do this in a slightly different (and more complex?) manner.
I use a MariaDB for CumulusMX data storage as this allows more flexibility in generating reports.
I have created a table to log all ExtraSensors (I have Ecowitt Internal temp sensors, WH45 indoor Air Quality and CO2, and WH57 Lightning detector).
Using the CumulusMX console I create a regular MySQL update in MySQL Settings > Custom Upload Minutes Interval which has a five minute update setting.
The MySQL command is :-
INSERT INTO ExtraSensors (`ET1`, `EH1`, `ET2`, `LightStrikes`, `A25`, `A25A`, `A10`, `A10A`, `AT`, `AH`, `CO2`, `CO2A`) VALUES ('<#ExtraTemp1>', '<#ExtraHum1>', '<#ExtraTemp2>', '<#LightningStrikesToday>', '<#CO2-pm2p5>', '<#CO2-pm2p5-24h>', '<#CO2-pm10>', '<#CO2-pm10-24h>', '<#CO2-temp>', '<#CO2-hum>', '<#CO2>', '<#CO2-24h>');
I also have a piece of Python code running on my Raspberry Pi, just after midnight, that gets the total lightning for the previous day and writes that to a second table (ESDaily) along with Max and Min values for the other extra sensors. Finally the Python code creates a .json file with the total daily lightning strikes for the last 31 days which can be used to create a graph of daily strikes. I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse! I can post the Python code if anyone is really desperate!
Re: It would be nice to save the lightning data to a file
Posted: Wed 25 May 2022 12:41 pm
by dazza1223
philpugh wrote: ↑Wed 25 May 2022 12:10 pm
I do this in a slightly different (and more complex?) manner.
I use a MariaDB for CumulusMX data storage as this allows more flexibility in generating reports.
I have created a table to log all ExtraSensors (I have Ecowitt Internal temp sensors, WH45 indoor Air Quality and CO2, and WH57 Lightning detector).
Using the CumulusMX console I create a regular MySQL update in MySQL Settings > Custom Upload Minutes Interval which has a five minute update setting.
The MySQL command is :-
INSERT INTO ExtraSensors (`ET1`, `EH1`, `ET2`, `LightStrikes`, `A25`, `A25A`, `A10`, `A10A`, `AT`, `AH`, `CO2`, `CO2A`) VALUES ('<#ExtraTemp1>', '<#ExtraHum1>', '<#ExtraTemp2>', '<#LightningStrikesToday>', '<#CO2-pm2p5>', '<#CO2-pm2p5-24h>', '<#CO2-pm10>', '<#CO2-pm10-24h>', '<#CO2-temp>', '<#CO2-hum>', '<#CO2>', '<#CO2-24h>');
I also have a piece of Python code running on my Raspberry Pi, just after midnight, that gets the total lightning for the previous day and writes that to a second table (ESDaily) along with Max and Min values for the other extra sensors. Finally the Python code creates a .json file with the total daily lightning strikes for the last 31 days which can be used to create a graph of daily strikes. I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse! I can post the Python code if anyone is really desperate!
hey yes i wouldn't mind using your Python code it wood come in handy thanks
Re: It would be nice to save the lightning data to a file
Posted: Wed 25 May 2022 2:47 pm
by philpugh
Attached is the python I use - it requires Python 3 and is not the best example! I found that Python has some limitations in handling creation of json files hence a funny bit of code towards the end.
I run this code via a CRONTAB entry
30 2 * * * /usr/bin/python3 /home/pi/esdaily.py >> /home/pi/esdaily.log 2>&1
which creates a log file if any issues arise - there is very little in the way of error condition handling in the code.
Enjoy and use at your own risk.
Re: It would be nice to save the lightning data to a file
Posted: Wed 25 May 2022 2:53 pm
by mcrossley
philpugh wrote: ↑Wed 25 May 2022 12:10 pm
I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse!
Sounds like you needed a stored procedure in your MySQL db?
I use them for updating tables like sunshine by hour of day, or similar for AQI but that is by day of week.
The stored procedures are triggered by the post insert event on the dayfile table.
Re: It would be nice to save the lightning data to a file
Posted: Wed 25 May 2022 3:47 pm
by freddie
philpugh wrote: ↑Wed 25 May 2022 12:10 pm
I also have a piece of Python code running on my Raspberry Pi, just after midnight, that gets the total lightning for the previous day and writes that to a second table (ESDaily) along with Max and Min values for the other extra sensors. Finally the Python code creates a .json file with the total daily lightning strikes for the last 31 days which can be used to create a graph of daily strikes. I did this in Python as the MySQL would have been too complex to include in the CumulusMX MySQL roll-over section - at least that is my excuse! I can post the Python code if anyone is really desperate!
You could probably do the ESdaily update using SQL. I have an identical setup to yours and do end-of-day max/min/sum stuff using the daily mysql update.
I've written a spring-based API to my database and get the equivalent of your JSON file via that route.
Re: It would be nice to save the lightning data to a file
Posted: Thu 26 May 2022 9:45 am
by philpugh
Yes I should be able to do both table updates in MySQL via CumulusMX but I would probably still need a bit of python to produce the JSON files?
Cheers
Re: It would be nice to save the lightning data to a file
Posted: Mon 30 Jan 2023 3:40 pm
by dazza1223
philpugh wrote: ↑Thu 26 May 2022 9:45 am
Yes I should be able to do both table updates in MySQL via CumulusMX but I would probably still need a bit of python to produce the JSON files?
Cheers
hey did u get the lighting data in Mysql and the triggers
Re: It would be nice to save the lightning data to a file
Posted: Mon 30 Jan 2023 4:20 pm
by HansR
Using Custom Logging might be the easiest way.