Is Cumulus for me? Developing driver to support station that outputs JSON observations

[mikejz]

I have a Raddy L7 weather station. I attempted to use WeeWx with the interceptor but I am giving up on it due to lack of support for Python 3.

Goal of the effort is to transmit APRS observations and store historic data.

In looking at Cumulus, it seems that is just a matter of writing the current observations to realtime.txt, right?

I found that my station's local IP address will output a JSON of the observations. It is as easy as taking this, formating and converting, and putting it in realtime.txt?

The JSON URL is http://ip/client?command=record

The update period appears to vary between 5-30 seconds.

Example output
Sensor Parameter Value
-------------------------------------------------------
Indoor Temperature 69.4
Indoor Humidity 43
Outdoor Temperature 44.7
Outdoor Humidity 70
Pressure Absolute 26.74
Pressure Relative 29.86
Wind Speed Max Daily Gust 3.8
Wind Speed Wind 0.0
Wind Speed Gust 0.0
Wind Speed Direction 213
Wind Speed Wind Average 2 Minute0.0
Wind Speed Direction Average 2 Minute204
Wind Speed Wind Average 10 Minute0.2
Wind Speed Direction Average 10 Minute300
Rainfall Rate 0.0
Rainfall Hour 0.0
Rainfall Day 0.0
Rainfall Week 0.73
Rainfall Month 0.73
Rainfall Year 6.45
Rainfall Total 11.39
Solar Light 0.3
Solar UVI 0.0
JSON Message:
{"sensor":[ {"title":"Indoor","list":[["Temperature","71.6","°F"],["Humidity","41","%"]]},{"title":"Outdoor","list":[["Temperature","59.2","°F"],["Humidity","44","%"]]},{"title":"Pressure","list":[["Absolute","26.77","inhg"],["Relative","29.89","inhg"]]},{"title":"Wind Speed","list":[["Max Daily Gust","8.9","mph"],["Wind","3.6","mph"],["Gust","3.8","mph"],["Direction","264","°"],["Wind Average 2 Minute","4.3","mph"],["Direction Average 2 Minute","246","°"],["Wind Average 10 Minute","3.1","mph"],["Direction Average 10 Minute","241","°"]]},{"title":"Rainfall","list":[["Rate","0.0","inch/hr"],["Hour","0.0","inch","43"],["Day","0.0","inch","44"],["Week","0.73","inch","45"],["Month","0.73","inch","46"],["Year","6.45","inch","47"],["Total","11.39","inch","48"]],"range":"Range: 0inch to 393.7inch."},{"title":"Solar","list":[["Light","153.59","w/m²"],["UVI","0.8",""]]}],"battery":{"title":"Battery","list":["All battery are ok"]}}

[mcrossley]

Realtime.txt is an output file. But MX does support a station type of EasyWeather that supports input from a text file in a specific format - it just reads the first line of the file. See https://www.jim-easterbrook.me.uk/weather/ew/ and https://www.cumuluswiki.org/a/EasyWeath ... est_record:

I do have an item on the ToDo list to add a JSON input station type that will support all sensor types - but it's still on the list!

[mcrossley]

Another alternative (better in many ways), is to use the "HTTP Station (Ecowitt)" station, which receives data via a HTTP POST. You can send this as often as you like - so every time the data updates, and it supports all the extra sensors that the EW file does not.

The format of the POST message body is URL encoded (why not just send it as GET then? Who knows)

A complex example:

Code: Select all

PASSKEY=<redacted>&stationtype=GW1000A_V1.6.8&dateutc=2021-07-23+17:13:34&tempinf=80.6&humidityin=50&baromrelin=29.940&baromabsin=29.081&tempf=81.3&humidity=43&winddir=296&windspeedmph=2.46&windgustmph=4.25&maxdailygust=14.09&solarradiation=226.28&uv=1&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.000&monthlyrainin=4.118&yearlyrainin=29.055&totalrainin=29.055&temp1f=83.48&humidity1=39&temp2f=87.98&humidity2=40&temp3f=82.04&humidity3=40&temp4f=93.56&humidity4=34&temp5f=-11.38&temp6f=87.26&humidity6=38&temp7f=45.50&humidity7=40&soilmoisture1=51&soilmoisture2=65&soilmoisture3=72&soilmoisture4=36&soilmoisture5=48&pm25_ch1=11.0&pm25_avg_24h_ch1=10.8&pm25_ch2=13.0&pm25_avg_24h_ch2=15.0&tf_co2=80.8&humi_co2=48&pm25_co2=4.8&pm25_24h_co2=6.1&pm10_co2=4.9&pm10_24h_co2=6.5&co2=493&co2_24h=454&lightning_time=1627039348&lightning_num=3&lightning=24&wh65batt=0&wh80batt=3.06&batt1=0&batt2=0&batt3=0&batt4=0&batt5=0&batt6=0&batt7=0&soilbatt1=1.5&soilbatt2=1.4&soilbatt3=1.5&soilbatt4=1.5&soilbatt5=1.6&pm25batt1=4&pm25batt2=4&wh57batt=4&co2_batt=6&freq=868M&model=GW1000_Pro

Just send the data fields you have, plus the initial fields and the last field. The PASSKEY is irrelevant, MX does not use it, so "123" is fine.

[mikejz]

Another alternative (better in many ways), is to use the "HTTP Station (Ecowitt)" station, which receives data via a HTTP POST. You can send this as often as you like - so every time the data updates, and it supports all the extra sensors that the EW file does not.

The format of the POST message body is URL encoded (why not just send it as GET then? Who knows)

A complex example:

Code: Select all

PASSKEY=<redacted>&stationtype=GW1000A_V1.6.8&dateutc=2021-07-23+17:13:34&tempinf=80.6&humidityin=50&baromrelin=29.940&baromabsin=29.081&tempf=81.3&humidity=43&winddir=296&windspeedmph=2.46&windgustmph=4.25&maxdailygust=14.09&solarradiation=226.28&uv=1&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.000&monthlyrainin=4.118&yearlyrainin=29.055&totalrainin=29.055&temp1f=83.48&humidity1=39&temp2f=87.98&humidity2=40&temp3f=82.04&humidity3=40&temp4f=93.56&humidity4=34&temp5f=-11.38&temp6f=87.26&humidity6=38&temp7f=45.50&humidity7=40&soilmoisture1=51&soilmoisture2=65&soilmoisture3=72&soilmoisture4=36&soilmoisture5=48&pm25_ch1=11.0&pm25_avg_24h_ch1=10.8&pm25_ch2=13.0&pm25_avg_24h_ch2=15.0&tf_co2=80.8&humi_co2=48&pm25_co2=4.8&pm25_24h_co2=6.1&pm10_co2=4.9&pm10_24h_co2=6.5&co2=493&co2_24h=454&lightning_time=1627039348&lightning_num=3&lightning=24&wh65batt=0&wh80batt=3.06&batt1=0&batt2=0&batt3=0&batt4=0&batt5=0&batt6=0&batt7=0&soilbatt1=1.5&soilbatt2=1.4&soilbatt3=1.5&soilbatt4=1.5&soilbatt5=1.6&pm25batt1=4&pm25batt2=4&wh57batt=4&co2_batt=6&freq=868M&model=GW1000_Pro

Just send the data fields you have, plus the initial fields and the last field. The PASSKEY is irrelevant, MX does not use it, so "123" is fine.

Thanks!

Anyone have an existing script I can borrow from?

[mikejz]

Another alternative (better in many ways), is to use the "HTTP Station (Ecowitt)" station, which receives data via a HTTP POST. You can send this as often as you like - so every time the data updates, and it supports all the extra sensors that the EW file does not.

The format of the POST message body is URL encoded (why not just send it as GET then? Who knows)

A complex example:

Code: Select all

PASSKEY=<redacted>&stationtype=GW1000A_V1.6.8&dateutc=2021-07-23+17:13:34&tempinf=80.6&humidityin=50&baromrelin=29.940&baromabsin=29.081&tempf=81.3&humidity=43&winddir=296&windspeedmph=2.46&windgustmph=4.25&maxdailygust=14.09&solarradiation=226.28&uv=1&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.000&monthlyrainin=4.118&yearlyrainin=29.055&totalrainin=29.055&temp1f=83.48&humidity1=39&temp2f=87.98&humidity2=40&temp3f=82.04&humidity3=40&temp4f=93.56&humidity4=34&temp5f=-11.38&temp6f=87.26&humidity6=38&temp7f=45.50&humidity7=40&soilmoisture1=51&soilmoisture2=65&soilmoisture3=72&soilmoisture4=36&soilmoisture5=48&pm25_ch1=11.0&pm25_avg_24h_ch1=10.8&pm25_ch2=13.0&pm25_avg_24h_ch2=15.0&tf_co2=80.8&humi_co2=48&pm25_co2=4.8&pm25_24h_co2=6.1&pm10_co2=4.9&pm10_24h_co2=6.5&co2=493&co2_24h=454&lightning_time=1627039348&lightning_num=3&lightning=24&wh65batt=0&wh80batt=3.06&batt1=0&batt2=0&batt3=0&batt4=0&batt5=0&batt6=0&batt7=0&soilbatt1=1.5&soilbatt2=1.4&soilbatt3=1.5&soilbatt4=1.5&soilbatt5=1.6&pm25batt1=4&pm25batt2=4&wh57batt=4&co2_batt=6&freq=868M&model=GW1000_Pro

Just send the data fields you have, plus the initial fields and the last field. The PASSKEY is irrelevant, MX does not use it, so "123" is fine.

I am trying the following code to send the data but getting a failure to connect error " Failed to establish a new connection: [Errno 111] Connection refused')"

Code: Select all

# Define the URL to send the POST request to
post_url = "http://localhost:8989/station/echwitt"

# Send the POST request
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
post_response = requests.post(post_url, data=formatted_response, headers=headers)
print (formatted_response)
# Print the response from the POST request
print(post_response.status_code)
print(post_response.text)

[freddie]

Is CumulusMX running at the time you ran your script? Does the hostname and port in your script match the hostname and port MX is running on? Any firewall between the script host and the MX host?

[HansR]

Code: Select all

# Define the URL to send the POST request to
post_url = "http://localhost:8989/station/echwitt"

[...]

Try: http://localhost:8989/station/ecowitt (spot the difference!)

[freddie]

Good spot @hansr, I missed that!

Edit: thinking about it, though, a typo like that should give you a 404 or perhaps 500 http response. So I'm staying with my original theory.

[mikejz]

Looking good. Only thing that seems off is something with the units and the rainfall.

MX.JPG

Code: Select all

import requests
import json
from datetime import datetime
import time

def fetch_and_send_data():
    # Fetch the JSON data
    json_url = "http://192.168.1.9/client?command=record"
    response = requests.get(json_url)
    data = response.json()

    # Mapping function
    def map_data(data):
        mapped_data = {
            "tempinf": None,
            "humidityin": None,
            "baromrelin": None,
            "baromabsin": None,
            "tempf": None,
            "humidity": None,
            "winddir": None,
            "windspeedmph": None,
            "windgustmph": None,
            "maxdailygust": None,
            "solarradiation": None,
            "uv": None,
            "rainratein": None,
            "eventrainin": None,
            "hourlyrainin": None,
            "dailyrainin": None,
            "weeklyrainin": None,
            "monthlyrainin": None,
            "yearlyrainin": None,
            "totalrainin": None
        }

        for sensor in data['sensor']:
            title = sensor['title']
            for parameter in sensor['list']:
                param_name = parameter[0]
                value = parameter[1]

                if title == "Indoor":
                    if param_name == "Temperature":
                        mapped_data["tempinf"] = value
                    elif param_name == "Humidity":
                        mapped_data["humidityin"] = value

                elif title == "Outdoor":
                    if param_name == "Temperature":
                        mapped_data["tempf"] = value
                    elif param_name == "Humidity":
                        mapped_data["humidity"] = value

                elif title == "Pressure":
                    if param_name == "Absolute":
                        mapped_data["baromabsin"] = value
                    elif param_name == "Relative":
                        mapped_data["baromrelin"] = value

                elif title == "Wind Speed":
                    if param_name == "Direction":
                        mapped_data["winddir"] = value
                    elif param_name == "Wind":
                        mapped_data["windspeedmph"] = value
                    elif param_name == "Gust":
                        mapped_data["windgustmph"] = value
                    elif param_name == "Max Daily Gust":
                        mapped_data["maxdailygust"] = value

                elif title == "Solar":
                    if param_name == "Light":
                        mapped_data["solarradiation"] = value
                    elif param_name == "UVI":
                        mapped_data["uv"] = value

                elif title == "Rainfall":
                    if param_name == "Rate":
                        mapped_data["rainratein"] = value
                    elif param_name == "Hour":
                        mapped_data["hourlyrainin"] = value
                    elif param_name == "Day":
                        mapped_data["dailyrainin"] = value
                    elif param_name == "Week":
                        mapped_data["weeklyrainin"] = value
                    elif param_name == "Month":
                        mapped_data["monthlyrainin"] = value
                    elif param_name == "Year":
                        mapped_data["yearlyrainin"] = value
                    elif param_name == "Total":
                        mapped_data["totalrainin"] = value

        return mapped_data

    mapped_data = map_data(data)

    # Create the formatted response
    dateutc = datetime.utcnow().strftime('%Y-%m-%d+%H:%M:%S')
    PASSKEY = "<redacted>"
    stationtype = "GW1000A_V1.6.8"
    formatted_response = f"PASSKEY={PASSKEY}&stationtype={stationtype}&dateutc={dateutc}"

    for key, value in mapped_data.items():
        if value is not None:
            formatted_response += f"&{key}={value}"

    # Define the URL to send the POST request to
    post_url = "http://localhost:8998/station/ecowitt"

    # Send the POST request
    headers = {'Content-Type': 'application/x-www-form-urlencoded'}
    post_response = requests.post(post_url, data=formatted_response, headers=headers)
    print(formatted_response)
    # Print the response from the POST request
    print(post_response.status_code)
    print(post_response.text)

# Main loop to repeat the function every x seconds
x = 60  # Set the interval (in seconds)
while True:
    fetch_and_send_data()
    time.sleep(x)

Value Sent:
PASSKEY=<redacted>&stationtype=GW1000A_V1.6.8&dateutc=2024-05-24+00:07:12&tempinf=75.7&humidityin=35&baromrelin=29.77&baromabsin=26.65&tempf=91.6&humidity=6&winddir=288&windspeedmph=8.5&windgustmph=10.3&maxdailygust=19.9&solarradiation=144.14&uv=1.5&rainratein=0.0&hourlyrainin=0.0&dailyrainin=0.0&weeklyrainin=0.09&monthlyrainin=0.28&yearlyrainin=11.09&totalrainin=16.03
200
success

Results:

[HansR]

Looking good. Only thing that seems off is something with the units and the rainfall. MX.JPG

[...]

Value Sent:
PASSKEY=<redacted>&stationtype=GW1000A_V1.6.8&dateutc=2024-05-24+00:07:12&tempinf=75.7&humidityin=35&baromrelin=29.77&baromabsin=26.65&tempf=91.6&humidity=6&winddir=288&windspeedmph=8.5&windgustmph=10.3&maxdailygust=19.9&solarradiation=144.14&uv=1.5&rainratein=0.0&hourlyrainin=0.0&dailyrainin=0.0&weeklyrainin=0.09&monthlyrainin=0.28&yearlyrainin=11.09&totalrainin=16.03
200
success

Results:

Mmm... obviously CMX is converting the values to non-imperial units while your settings are imperial.

Wrt the rain: the daily rain is 0 and I am not sure if CMX does use the other values. I assume it will use its internal counters and ignore those weekly, monthly and yearly values of the station.

Maybe @Mark should have a say here.

[EDIT:] And it would be nice to know if @freddie was right or me, in other words was it the typo or host configuration

[mcrossley]

As you have a total rainfall counter MX will use that and ignore all the others.

MX only uses an incrementing counter, so it will register rainfall from the point it it first started. If you want to enter this year's rainfall to the point when it was started you have to enter that value manually in the rainfall editor.

[mikejz]

Looking good. Only thing that seems off is something with the units and the rainfall. MX.JPG

[...]

Value Sent:
PASSKEY=<redacted>&stationtype=GW1000A_V1.6.8&dateutc=2024-05-24+00:07:12&tempinf=75.7&humidityin=35&baromrelin=29.77&baromabsin=26.65&tempf=91.6&humidity=6&winddir=288&windspeedmph=8.5&windgustmph=10.3&maxdailygust=19.9&solarradiation=144.14&uv=1.5&rainratein=0.0&hourlyrainin=0.0&dailyrainin=0.0&weeklyrainin=0.09&monthlyrainin=0.28&yearlyrainin=11.09&totalrainin=16.03
200
success

Results:

Mmm... obviously CMX is converting the values to non-imperial units while your settings are imperial.

Wrt the rain: the daily rain is 0 and I am not sure if CMX does use the other values. I assume it will use its internal counters and ignore those weekly, monthly and yearly values of the station.

Maybe @Mark should have a say here.

[EDIT:] And it would be nice to know if @freddie was right or me, in other words was it the typo or host configuration

That's the odd thing. The station is sending imperial and CMX is set to imperial, but somehow it's doing a conversion.

[mcrossley]

I'm not sure what is wrong with the units?

In the screen shot above MX is showing 33.1C for the outdoor temp, you are sending 93.1F, so that is correct?

Oops! I zoomed in and see MX is shows 33.1F!

I don't understand this as it is working for real Ecowitt stations.

[mcrossley]

Please switch on debug and data logging in Cumulus. Restart MX, then post the latest MXdiags log file after say 10 minutes.

Thanks.

[mikejz]

Please switch on debug and data logging in Cumulus. Restart MX, then post the latest MXdiags log file after say 10 minutes.

Thanks.

I did a reinstall due to a failed upgrade and outdoor temp/pressure were not updating as opposed to odd values.

Looks like there are old values that are tripping limits

2024-05-26 16:29:18 - Spike removal: NewVal=94.8 OldVal=999.0 SpikeTempDiff=999.0 HighLimit=60.0 LowLimit=-60.0
2024-05-26 16:29:18 - Spike removal: Temp difference greater than specified; reading ignored
2024-05-26 16:29:18 - Spike removal: NewVal=29.9 OldVal=9999.0 SpikePressDiff=999.0 HighLimit=1090.0 LowLimit=870.0
2024-05-26 16:29:18 - Spike removal: Pressure difference greater than specified; reading ignored
2024-05-26 16:29:07 - Spike removal: NewVal=94.8 OldVal=999.0 SpikeTempDiff=999.0 HighLimit=60.0 LowLimit=-60.0
2024-05-26 16:29:07 - Spike removal: Temp difference greater than specified; reading ignored
2024-05-26 16:29:07 - Spike removal: NewVal=29.9 OldVal=9999.0 SpikePressDiff=999.0 HighLimit=1090.0 LowLimit=870.0
2024-05-26 16:29:07 - Spike removal: Pressure difference greater than specified; reading ignored
2024-05-26 16:29:00 - Minimum data set of pressure, temperature, and wind is not available and NoSensorCheck is not enabled. Skip processing
2024-05-26 16:28:57 - Spike removal: NewVal=94.8 OldVal=999.0 SpikeTempDiff=999.0 HighLimit=60.0 LowLimit=-60.0
2024-05-26 16:28:57 - Spike removal: Temp difference greater than specified; reading ignored
2024-05-26 16:28:57 - Spike removal: NewVal=29.9 OldVal=9999.0 SpikePressDiff=999.0 HighLimit=1090.0 LowLimit=870.0
2024-05-26 16:28:57 - Spike removal: Pressure difference greater than specified; reading ignored
2024-05-26 16:28:47 - Spike removal: NewVal=94.7 OldVal=999.0 SpikeTempDiff=999.0 HighLimit=60.0 LowLimit=-60.0
2024-05-26 16:28:47 - Spike removal: Temp difference greater than specified; reading ignored
2024-05-26 16:28:47 - Spike removal: NewVal=29.9 OldVal=9999.0 SpikePressDiff=999.0 HighLimit=1090.0 LowLimit=870.0
2024-05-26 16:28:47 - Spike removal: Pressure difference greater than specified; reading ignored
2024-05-26 16:28:37 - Spike removal: NewVal=94.7 OldVal=999.0 SpikeTempDiff=999.0 HighLimit=60.0 LowLimit=-60.0
2024-05-26 16:28:37 - Spike removal: Temp difference greater than specified; reading ignored
2024-05-26 16:28:37 - Spike removal: NewVal=29.9 OldVal=9999.0 SpikePressDiff=999.0 HighLimit=1090.0 LowLimit=870.0
2024-05-26 16:28:37 - Spike removal: Pressure difference greater than specified; reading ignored
2024-05-26 16:28:29 - You are not running the latest version of Cumulus MX, build 4023 is available.
2024-05-26 16:28:28 - API.GetHistoricData: Missing Ecowitt API data in the configuration, aborting!
2024-05-26 16:28:28 - LoadDayFile: No Dayfile found - No entries added to recent daily data list

Code: Select all

PASSKEY=ASDFASDF1234ASDF1234&stationtype=RADDY_L7&dateutc=2024-05-26+23:30:08&tempinf=75.0&humidityin=37&baromrelin=29.86&baromabsin=26.73&tempf=94.9&humidity=10&winddir=301&windspeedmph=2.7&windgustmph=2.9&maxdailygust=14.1&solarradiation=695.0&uv=2.3&rainratein=0.0&hourlyrainin=0.0&dailyrainin=0.0&weeklyrainin=0.0&monthlyrainin=0.3&yearlyrainin=11.1&totalrainin=16.04
200
success