Adding Lightning data - hybrid Cumulusmx setup

[watsonm]

I have a hybrid Cumulusmx setup with several transmitters and Raspberry Pi code to receive the data directly (via RTL_433 code).
This all get mangled together by my Python code to produce an Easyweather.dat file.
This all works fine but now my latest update is to try and add a WH57 lightning detector.

The easiest scenario is to use my Python code to send the Lightning data to my mobile phone but if possible I would like to add the data
to CMX so it can be seen in the extra sensors table. Have scanned the forum and looked at the Wiki but cannot see what file or database
to add to or even if this is possible.

My current thoughts are to create my own sql database file and add in each WH57 output to that with a view to somehow displaying it in
a chart in CumulusUtils.

Thanks for any pointers...

[mcrossley]

If you want to put everything in CMX I would change your approach completely!

Gather all the data as you are now from the sensors, then send it to CMX using the Ecowitt HTP protocol. This is a simple text based protocol, your code would just send the data periodically in a http request to CMX which sits there listening for it.

As you know the Easyweather file is restricted in the data you can add to it, and the CMX update rate is fixed. Using the http approach you can send data was often as you like.

[watsonm]

Mark,

Many thanks...
Managed to get that up and running OK on a test system.
Currently running on Windows 10 side by side with Easyweather.dat setup running on the Raspberry Pi.

[iandrews]

Hi watsonm,

Has this been working ok. As I want to try and do something similar.

As well as my Davis weather station (connected to CMX) I have 7 WS-07 Temperature / humidity sensors being picked up by RTL433 running on a Pi. The data from the WS-07 sensors is sent to a influxdb and then graphed with grafana that I can view via a web page. This generally works well, (but I do get the occasional spikes in temp or hum readings).

What I would like to do is get the data into Cumulus, and that seems possible now with the Extra sensors and the Ecowitt HTTP protocol.

Do you have any code examples (Github), and do you still use HiveMQ Broker as a go between, or have you got the output from RTL433 going direct to Cumulus.

Thanks.

[watsonm]

Ian,

Yes this is working OK and has been since August 2022 I think. My only issue is I think that the broker may be gobbling up my memory. I only bought a 2gig Pi and think a 4 gig would have been better. However I may well implement an automatic re boot at midnight (+- 30 mins) while I chase the problem. I could be pumping out too much data as I send to most of the CMX third party sites as well as Ecowitt, Airlink, Luftdaten, Sensor community, calculating AQI directly from my hand built code. (123Kb worth of python!)

My main code is all hand crafted so other than RTL433 and the broker nothing of mine is on github.

Basically the code to send to CMX is to build up a buffer with the data that has been processed from the broker and then send it to CMX using a request.post call.

Code: Select all

             CMX_url = "http://192.168.1.2:8998/station/ecowitt"
             now = datetime.now()                                                                
             EW_time = now.strftime("%Y-%m-%d %H:%M:%S ")
             weather_data={'dateutc' : EW_time}
             weather_data['tempf']          =  tempfahr  
            .
            .
            .   rest of data points  
            .
             weather_data['rainratein']     =  rainratein 
             weather_data[ 'totalrainin']   =  rain_inch

              try:
                  res = requests.post( CMX_url , weather_data , timeout = 5)
                  if res.reason != "OK" :
                     log_err("Ecowitt http output  failed ############# : ")
                     print(res.status_code, res.reason)
                  pass
              except requests.exceptions.Timeout:
                  log_err("Ecowitt http output  timeout exception ############# : ")
              except requests.exceptions.TooManyRedirects:
                  log_err("Ecowitt http output  too may redirects ############# : ")
              except requests.exceptions.RequestException as e:
                  log_err("Ecowitt http output  failed ############# : ")
                  pass

An example of a buffer stream :

Code: Select all

   
CumulusMX Data :{'dateutc': '2023-01-19 16:56:21 ', 'tempf': 36.5, 'tempinf': 68.18, 'solarradiation': 0, 'humidity': '67', 'baromabsin': 29.457858, 'baromrelin': 29.837909, 'windspeedmph': '0.9127', 'windgustmph': '1.8253', 'winddir': 251, 'humidityin': 44, 'rainratein': 0.0, 'totalrainin': 30.862222}

Soil data is sent to : x = requests.post( CMX_url+"extra" , data, timeout = 5)
as is PM2.5/10 data.

Code: Select all

   if float(pmg2p5) != -1:
         upd = {'pm25_co2'  :  pmg2p5 }
         soil_data.update(upd)

    if float(pmg25twenty) != -1:
         upd = {'pm25_24h_co2'  :  pmg25twenty }
         soil_data.update(upd)

    if float(pmg10) != -1:
         upd = {'pm10_co2'  :  pmg10 }
         soil_data.update(upd)

    if float(pmg10twenty) != -1:
         upd = {'pm10_24h_co2'  :  pmg10twenty }
         soil_data.update(upd)
         
          try:
        x = requests.post( CMX_url+"extra" , soil_data, timeout = 5)
        y = x.text.find('success')

        if y != 1 :
             log_err("Soil1 http output  failed ############# : ")

    except :
             log_err("Soil1 http output  exception ############# : ")
 

Example data stream:

Code: Select all

Soil data: {'SoilMoisture1': 57, 'SoilMoisture2': 75, 'pm25_co2': 3.0, 'pm25_24h_co2': 2.43, 'pm10_co2': 3.0, 'pm10_24h_co2': 2.41}

[iandrews]

Wow, thanks, I'm not really a Linux / python guy (Windows support was my thing), but it sort of makes sense, so I will look into it more, and see what I can do.

[watsonm]

I'm was a Windows support guy too (retired 15years) and I originally wrote my code to run on both os's. (I was teaching myself Linux and python). I doubt it does now because of some of the imports I've had to include.

[iandrews]

So, I have made some progress.

It seems (though doesn't seem to be that well documented) that RTL_433 (since version 20.11 I think) has an inbuilt HTTP server. If you use -F http (it can be used with other -F values, so I still have data going into my influxdb as well) it allows you to either bring up a web page which displays sensor info, or you can go to /events and the sensor info gets sent as JSON.

There are some example scrips here:

https://github.com/merbanan/rtl_433/com ... b2b87c4cd6

I'm not sure exactly what the difference is between the events, stream and ws methods, but I chose the events example and have modified it to receive the sensor data, and then forward on to a test instance of Cumulus MX that I installed and that I enabled the Extra Sensors on (as HTTP Ecowitt). And I now have the data displaying on the Extra sensors table, and in charts.

Still playing around with it, and got some tidying up to do (such as remove the print lines that won't be needed when it runs in the background, but the code (just the bit that reads the data and uploads it) is below. Struggled a bit with creating the data string to upload, but working off your example I seem to have got it.

Code: Select all

        data = json.loads(line)

        #
        # Change for your custom handling below, this is a simple example
        #
        # E.g. match `model`.
        if data["model"] == "Ambientweather-F007TH":    #Is this a temp sensor
            sensor = str(data["channel"])                           #Get the channel number
            if "temperature_C" in data and "humidity" in data:            #Does the data have both temp and hum
                print(sensor,'Temperature',data["temperature_C"],'Humidity',data["humidity"])
                tempf = round(data["temperature_C"]*1.8+32,2)         #converts temp from c to f
                tempsensor = 'temp'+sensor+'f'                                    #create temp variable containing sensor number eg temp1f 
                humsensor = 'humidity'+sensor                                    #create hum variable containing sensor number eg numidityf 
                print(tempsensor, humsensor)
                temp_data={}                                                             #create the data to upload
                temp_data[tempsensor] = tempf
                temp_data[humsensor] = data["humidity"]
                print(temp_data)
                try:
                    x = requests.post(CMX_url , temp_data, timeout = 5)     #upload data
                    print(x)
                except:
                    print("Error with upload, continue")

Thanks.

[watsonm]

Ian,
Well done...

I am using the HIVEMQ broker so to get the rtl433 data I use the following in my rtl433.service file.

Code: Select all

ExecStart=/usr/local/bin/rtl_433     -d :00000100 -p 15 -C si -F "mqtt://192.168.1.15:1883,user=xxxxxn,pass=xxxxxxxx, retain=0"  

-d is the device id code ( had to change it as I us another rtl dongle to pull in aircraft data!! ... yes still a nerd at 74!!)
-p 15 Correct rtl-sdr tuner frequency offset error
-C native si Convert units in decoded output.
-F mqtt convert output for Broker.

I am using the broker as it allows me to easily send and receive data from two other Pi units. One tracking airquality down by my garage and another doing most of the aircraft work but also working a second lightning unit (as3935) in the loft.

[iandrews]

Mike,

I was going to try and use a mqtt broker, but in doing research I spotted the http option so decided to try that.

My command line is currently:

rtl_433 -R 20 -R 12 -C si -F "influx://localhost:8086/write?db=rtl433&p=pass&u=user" -F http

The 2 -R's specify the devices I am scanning for, -C like you converts the units (though have to convert the temp from C back into F for the Ecowitt HTTP protocol). Then using -F output is sent to a influxdb and now also the http server.

I also run a pi (well 2) with SDR dongles for decoding and displaying aircraft.