NOAA Reports - Day/Month Log File [Solved]

[Cambium]

EDIT: See Other Thread which contained more info/help with solving it.
viewtopic.php?f=40&t=19964&hilit=NOAA

Does Cumulus only log data when it's physically opened? I ask this because I noticed the 7th and 10th is missing from the log file and I believe I didnt open the program on the 7th.

But then on the NOAA report the 7th and 10th is there but missing the 9th which has data in the log file.

[mcrossley]

Your station does not have a logger, so yes Cumulus will only capture data when it is running. You have to run MX 24x7 if you want complete records, which is why many people now run it on a Raspberry Pi.

Other stations that do have loggers** do not require Cumulus to be running 24x7 as it can download the missed data from the station next time you start it.

**One day Ecowitt will may finally release their web API, which will Cumulus to catch-up using the web based data. But until that day happens - no catch-up for these stations.

[water01]

**One day Ecowitt will may finally release their web API, which will Cumulus to catch-up using the web based data. But until that day happens - no catch-up for these stations.

Interesting Mark, anything we Ecowitt users can do to put pressure on Ecowittt to do so?

[Cambium]

Oh wow. Interesting. Is there a way to copy and paste the data from WeatherUnderground or Ecowitt into the file so Cumulus can read it?

[Gyvate]

Oh wow. Interesting. Is there a way to copy and paste the data from WeatherUnderground or Ecowitt into the file so Cumulus can read it?

In principle there is - not too simple but not too complicated either.
Also depends on whether you use extra sensors (what CMX considers to be an extra sensor - as they have a separate log file) or not.
You can look up the structure of the csv log files (the sequence and meaning of the comma separated entries (or semicolon separated depending on your locale))
- you find the headers in the CumulusMX directory. e.g. dayfileheader.txt, monthlyfileheader.txt, extrafileheader.txt

Find below how I managed to add missing values from ecowitt.net recordings:
Download the data e.g. from Ecowitt.net, load them into MS EXCEL, get the data fields into the proper sequence and save them as csv type of file.
Then you can open the actual mmmYYYlog.txt file (e.g. Dec21log.txt) and (if applicable) ExtraLogYYYYMM.txt (e.g. ExtraLog202112.txt) and paste the missing data into the gaps. Save it under a different name (e.g. Dec21log1.txt) and complete your work until you have copied all your downloaded entries. Then open the original Dec21log.txt again, copy the meanwhile accumulated new entries into your Dec21log1.txt file. Save. Delete (or rename) Dec21log.txt and rename your work file Dec21log1.txt to Dec21log.txt. Make sure these last steps happen in the time window between two writings by CMX (before the next full minute if you have a one minute logging interval, before the next 5th minute if you have a 5 minute logging interval etc.).
If you have extra sensors, you have to do the same with the extra log file.
If the the added entries contain records (maximum values of the day, month, year, alltime), you will also have to change the corresponding .ini files

[Cambium]

Oh wow. Interesting. Is there a way to copy and paste the data from WeatherUnderground or Ecowitt into the file so Cumulus can read it?

In principle there is - not too simple but not too complicated either.
Also depends on whether you use extra sensors (what CMX considers to be an extra sensor - as they have a separate log file) or not.
You can look up the structure of the csv log files (the sequence and meaning of the comma separated entries (or semicolon separated depending on your locale))
- you find the headers in the CumulusMX directory. e.g. dayfileheader.txt, monthlyfileheader.txt, extrafileheader.txt

Find below how I managed to add missing values from ecowitt.net recordings:
Download the data e.g. from Ecowitt.net, load them into MS EXCEL, get the data fields into the proper sequence and save them as csv type of file.
Then you can open the actual mmmYYYlog.txt file (e.g. Dec21log.txt) and (if applicable) ExtraLogYYYYMM.txt (e.g. ExtraLog202112.txt) and paste the missing data into the gaps. Save it under a different name (e.g. Dec21log1.txt) and complete your work until you have copied all your downloaded entries. Then open the original Dec21log.txt again, copy the meanwhile accumulated new entries into your Dec21log1.txt file. Save. Delete (or rename) Dec21log.txt and rename your work file Dec21log1.txt to Dec21log.txt. Make sure these last steps happen in the time window between two writings by CMX (before the next full minute if you have a one minute logging interval, before the next 5th minute if you have a 5 minute logging interval etc.).
If you have extra sensors, you have to do the same with the extra log file.
If the the added entries contain records (maximum values of the day, month, year, alltime), you will also have to change the corresponding .ini files

Thank You for this! I will try it when I have time. I remember going into the old Cumulus files fixing errors and stuff so I'm familiar with the headers, format, ect. But didnt know how to download from Ecowitt. Thanks!

[Gyvate]

At Ecowitt, you have to click on the "Export" button between the tiles and the graphs.
At the left hand side you choose the graph and download time frame.
To consider:
Choosing daily will give a data records for every 5 minutes, weekly will be in 30 minute portions, monthly in 4 hour portions and yearly in daily portions.

[water01]

Does the TP3000 come with an SD slot in the Console like the HP2551? If it does you can ask it to backup all the data to the SD card and use that to update the CumulusMX files.

[Gyvate]

The ProWeather TP3000WC is a sort of Ecowitt WS2320E with a black/greenish-grey display of the console.
However the console, which is also battery backed-up, can store up to 3550 records which can be retrieved under Windows with the WeatherSmartIP software. Once 3550 records are stored, the oldest will be overwritten.
So Mark was only partially right - it (the console) stores data, but via the http connection/custom server they cannot be retrieved as this interfaces only uses what the console pushes and is not pulling data from the console API. Now, depending on your chosen storage interval, it covers a longer time period: 1 minute interval = 2 days, 5 minute interval = 12 days etc. (3550/interval). The software will automatically retrieve records it hasn't archived yet in its (by the way MS Access) database. So the computer doesn't have to run 24/7.
The data archived on the PC can be exported as CSV files, regrouped and copied into the CMX log files. For a TP3000 it will be the mmmYYlog.txt only as the console only has basic sensors (i.e. no ExtraSensor log file needed). Also the retrieval and (re-)integration into CMX will be independent of an internet connection.
@Mark:
by the way, the API of the WS2320E/WH4000SE/.... is documented and e.g. Meteobridge can directly connect to the console via the API.
It retrieves the records via the API like the WeatherSmart IP software does. Only under openWRT and not under WIndows.
The advantage here is that the custom server function remains available for other purposes.

[Cambium]

At Ecowitt, you have to click on the "Export" button between the tiles and the graphs.
At the left hand side you choose the graph and download time frame.
To consider:
Choosing daily will give a data records for every 5 minutes, weekly will be in 30 minute portions, monthly in 4 hour portions and yearly in daily portions.

Yup.. I quickly tried this morning and it's a big task rearranging the data to enter it into the cumulus log file. Will try again later

Does the TP3000 come with an SD slot in the Console like the HP2551? If it does you can ask it to backup all the data to the SD card and use that to update the CumulusMX files.

Nope. It's as cheap as you can get. lol It was a feature I was looking for but just gave up after spending too much time looking into a station that would work with Cumulus. I just stuck with this one.

The ProWeather TP3000WC is a sort of Ecowitt WS2320E with a black/greenish-grey display of the console.
However the console, which is also battery backed-up, can store up to 3550 records which can be retrieved under Windows with the WeatherSmartIP software. Once 3550 records are stored, the oldest will be overwritten.
So Mark was only partially right - it (the console) stores data, but via the http connection/custom server they cannot be retrieved as this interfaces only uses what the console pushes and is not pulling data from the console API. Now, depending on your chosen storage interval, it covers a longer time period: 1 minute interval = 2 days, 5 minute interval = 12 days etc. (3550/interval). The software will automatically retrieve records it hasn't archived yet in its (by the way MS Access) database. So the computer doesn't have to run 24/7.
The data archived on the PC can be exported as CSV files, regrouped and copied into the CMX log files. For a TP3000 it will be the mmmYYlog.txt only as the console only has basic sensors (i.e. no ExtraSensor log file needed). Also the retrieval and (re-)integration into CMX will be independent of an internet connection.

So much to learn. Thats right! Good reminder the software will download the data!!

[Gyvate]

you may want to (re-?)read the feature description at https://www.tyconsystems.com/tp3000wc

[Cambium]

So Im in the process of pasting data into the MXDec21 Log file and noticed a couple things...

The WeatherSmart software HAS THIS that Cumulus doesnt have in the monthly log file: Which is fine I think
Relative (inHg), Absolute (inHg), Weekly Rainfall(in, Monthly Rainfall(in), Annual Rainfall(in), Total Rainfall(in)

BUT... The WeatherSmart software DOESNT have this which is needed in the Cumulus Monthly log file:
Rainfall rate, Sea level pressure, Rainfall counter, Current gust, Heat Index, UV Index, Solar Radiation, Evapotranspiration, Annual Evapotranspiration, Apparent temperature Max Solar radiation, Hours of sunshine, Wind bearing , RG-11 Rain , Rain Since Midnight , Feels Like Humidex

Should I put 0's for all those columns?

[Gyvate]

WeatherSmartIP provides the following observations
dateTime = Time #Date+Time - format depending on your locale - the same with units, they depend on what you choose in the WSIP settings
they should match what CMX usually uses for you
Relative Pressure(hpa), hPa #sealevel pressure
Absolute Pressure(hpa), hPa
Indoor Temperature(C), degree_C
Indoor Humidity(%), percent
Outdoor Temperature(C), degree_C
Outdoor Humidity(%), percent
Wind Speed(km/h), km_per_hour
Wind Direction, degree_compass #wind bearing
Gust(km/h), km_per_hour #current gust
Day Rainfall(mm), mm #rainfall counter (per day), the total counter (year) you have to calculate yourself
DewPoint(C), degree_C
WindChill(C), degree_C
Light(W/M2), watt_per_meter_squared #solar radiation
UVI, uv_index
sunshine hours are only there if they are provided by the console, same with ET etc. - what you don't have you will have to leave empty ,, (nothing between two subsequent separators, here ",") or calculate yourself if you think this data important for you

[Cambium]

WeatherSmartIP provides the following observations
dateTime = Time #Date+Time - format depending on your locale - the same with units, they depend on what you choose in the WSIP settings
they should match what CMX usually uses for you
Relative Pressure(hpa), hPa #sealevel pressure
Absolute Pressure(hpa), hPa
Indoor Temperature(C), degree_C
Indoor Humidity(%), percent
Outdoor Temperature(C), degree_C
Outdoor Humidity(%), percent
Wind Speed(km/h), km_per_hour
Wind Direction, degree_compass #wind bearing
Gust(km/h), km_per_hour #current gust
Day Rainfall(mm), mm #rainfall counter (per day), the total counter (year) you have to calculate yourself
DewPoint(C), degree_C
WindChill(C), degree_C
Light(W/M2), watt_per_meter_squared #solar radiation
UVI, uv_index
sunshine hours are only there if they are provided by the console, same with ET etc. - what you don't have you will have to leave empty ,, (nothing between two subsequent separators, here ",") or calculate yourself if you think this data important for you

Yeah, I was using the dayfile header, not the monthly file header. Whoops. So I got everything in and used the empty ,, without anything between that. Tried a few different times including just using 21 as the year instead of 2021... Cant get it to come up. When I refresh the NOAA monthly report or the monthly logs its empty. I revert back to the original so I still have that availble.

Not sure what I'm doing wrong yet. I'm copying and pasting the text right into the Dec21 file in the backup and the data folders.

New file looks like this.

[Gyvate]

what does the "old" file look like ?
is the date-time appropriate ?
shouldn't it be

Code: Select all

01/12/21,17:09,7.0,96,6.4,4.6,13.7,82,0.0,8.4,998.0,1057.3,21.5,49,6.5,7.0,7.0,0.0,0,0.16,27.67,5.3,0,0.0,85,0.0,8.4,7.0,7.0

=>
1. date notation should be 01/12/21 and not 12/01/21
2. time notation should be 17:00 and not 05:00 PM (if it has to be PM, then also AM for the times between midnight and noon)
3. not existing values should be marked "0.0" (I have to correct my earlier statement here after looking into my log files in detail)

Doesn't your normal ("old") Dec21log.txt record from CMX via http interface look like the example from mine ?
You have to use the same notation as CMX usually has - the console export may have a different format and needs possibly to be converted beforehand
(your notation might or might not be different from mine in the above example)