Missing data

[access-mdb]

I've just started running Cumulus MX on an Ecowitt HP2551. I upload the data from the console to ecowitt.net. All working fine now thanks to lots of help from Freddie. So as it was all working fine, I changed to starting CMX as a service. Last night I shut the PC down assuming that when I restarted the PC this morning it would just continue on from there. However, it didn't and all the data from yesterday evening to this morning is missing. I think I'm right in saying that it should have copied the data down. The PC CMX is running on has wifi connection to the internet and very occasionally it doesn't start correctly and this morning was just such a case. It's possible that this has caused the problem.
So the question is, how do I redo from last night to get the missing data. Do I just copy the backup file from yesterday and restart CMX (I've stopped it pro tem)?
It's possible that the information is in the wiki, but I couldn't find any.

[LindaFNM]

I hope someone more knowledgeable will reply to you, but my understanding is this:

I have an Ecowitt 2551 and am running CumulusMX on a permanently running Raspberry Pi, I also upload to Ecowitt.net.

The problem with the Ecowitt console is that it isn’t a data logger. It’s very confusing as it records data to an SD card if inserted, but this isn’t actually a logger in the true sense of the word that a program like CumulusMX can access. I really wish that Ecowitt would implement some sort of logger for their consoles which would be very useful in the event of problems. This is why I eventually went the Raspberry Pi route and have a GW1100.

To get your weather data into Cumulus you’ll need to use the .csv file either from the SD card on the console, or export a .csv from Ecowitt.net, or think about having a computer of some sort permanently running to capture the ‘live’ data.

I hope this helps a little.

[access-mdb]

Thanks Linda, but what you describe is very clunky in my view. If Cumulus 1 was shut down, when it was restarted it collected data from when it shut down. Does what you're saying mean the CMX just loses the data? I normally leave my PC running but thought I would shut down, because any computer is likely to be shut down at any time - power cut, hardware problems, maintenance etc.
I tried using the backup to replace the missing data, but there was still no data. Perhaps I didn't do it properly. I'll look again at copying the data from either the Ecowit export or the console backup.

[mcrossley]

If Cumulus 1 was shut down, when it was restarted it collected data from when it shut down.

Cumulus 1 did not support the HP2551, if it did it wold be exactly the same.

As explained, that station does not have a logger function, so no program can recover data from when it was not running. At present** if you want that feature then you will have to buy another brand of station that does support logging historic data. In my view it is a big disadvantage of the Ecowitt stations.

** Ecowitt have long promised to provide an API to their cloud storage to access historic data. That would only work if you allow your station to upload to the Ecowitt cloud of course.*** But this API has been promised for a year now with no release date announced so far - every query has produced a response of "in a month or two / next month" etc.

*** This is how the historic catch-up works with the Davis WeatherLink Live device.

[water01]

I have a HP2551 as well but as already stated although it has internal memory and an SD card it does not act as a logger until and when they update the API as Mark says. I enquired about this and they hoped to have it available by the end of the year, but don't hold your breath.

Until then you either run your PC 24/7 or download the data from Ecowitt.net or the backup data on your SD card and update the records manually.

[access-mdb]

OK, thanks. I will have to update the data manually. Is there something that says what the dayfile headers are as well as what the headers are for the 10 minute data (Dec21log.txt)? There was for C1, but I haven't found such for CMX.
Perhaps I will have to dive into a Raspberry Pi, but which one is best for this?

[Cambium]

Last night I shut the PC down assuming that when I restarted the PC this morning it would just continue on from there. However, it didn't and all the data from yesterday evening to this morning is missing.

Yup, I'm having a similar issue. I found out Cumulus has to be running 24/7 with our stations to properly read the data otherwise you have to manually enter the data and that became a time consuming thing. I decided to maually enter the missing data after every month.

Is yours picking up the minimum temp from the time you start it to stop? If you look at the day file or NOAA report does it have the minimum temp?

[Cambium]

OK, thanks. I will have to update the data manually. Is there something that says what the dayfile headers are as well as what the headers are for the 10 minute data (Dec21log.txt)? There was for C1, but I haven't found such for CMX.

Its in the CMX folder.

[SamiS]

Perhaps I will have to dive into a Raspberry Pi, but which one is best for this?

Some prefer low cost, some other small size or low power need, some want the most powerful version to run all kinds of server applications on the same machine with CumulusMX. Therefore it is quite impossible to name the ”best” solution.

My personal choice is Raspberry 4B 4GB version with Argon One M.2 case and m.2 sata ssd powered by genuine Rpi power supply. This combination allows booting the system straight from ssd and getting rid of the microsd cards, which tend to fail too often without any warnings. The case also collects all the interfaces on one side of the case and provides a full-sized hdmi without any adapter cords.

PS. Merry Christmas for everyone!

[access-mdb]

Thanks cambium, it's so obvious once you see them! I will beat this!

[access-mdb]

Some prefer low cost, some other small size or low power need, some want the most powerful version to run all kinds of server applications on the same machine with CumulusMX. Therefore it is quite impossible to name the ”best” solution.

PS. Merry Christmas for everyone!

Thanks SamiS, I appreciate it's a difficult question to answer. My needs would be for it to be just for CMX - I don't really want to start learning too many skills at my advanced age!

[LindaFNM]

I’m running a Pi 3B+ also with an SSD drive after spending too much time worrying about my SD card!!! I chose this Pi as it also boots straight from the SSD, not sure some of the others do. I started with a ‘beginner’ kit with NOOBS included from Amazon, not sure where you’re located but it was about £60 in the UK.

I’m not young either You can either use the pre loaded system provided by Mark as found on the website or go it alone. There is an excellent set of instructions in the wiki for this, and flashing the OS from windows is surprisingly easy, also good instructions from the Raspberry Pi website. I like the fact that if you do ‘mess up’ it’s easy to start over, just make sure to take regular backups, there is a disk copy facility within the Pi OS.

Importing data from other .csv’s isn’t the best way to go as you have to get the formats right before you even start.

My next step is a web server, but I’m still prevaricating! I need to just do it. Please can I hijack here, can you run your web server and CumulusMX from ‘local host’ or do they clash? Thank you.

[access-mdb]

Well thanks again to all who've contributed. What have I learnt? Don't blame CMX for Ecowitt's failings! However, there will always be occasions of missing data and it's clunky to manually fit the data in (especially as the columns are different). I use VBA quite a lot, used it at work before I retired some years ago and have continued to use it in my own Access databases. I've started to use it in Excel (something I've never really used in anger). So I plan on creating a VBA script which reads the relevant CSVs and allocates the different values to the appropriate columns. I can then save the result as a CSV and squirt it into the monthly file or dayfile in the correct place - very easy (I hope).
However, the naming of the various fields isn't consistent, so I'm not sure of all the transformations. I'm attaching a simple spreadsheet with the monthly file attempt. Column A is all the monthly headers, Column B is just the column letters of these. Column C is the column are where the Ecowitt export columns are matched, and column D the console BU. My intention is to use the Ecowitt data as much as possible only using the console BU if necessary.I would appreciate it if someone can look at this and tell me where I can find the missing entries or where there are errors. Thanks in advance.

data_preparation.xlsx

p.s. If it works, then I would be happy to share this with others, but maybe I'm getting ahead of myself here!

[PaulMy]

When you have data in the monthly log files there is a CMX utility CreateMissing that you can run to add the missing days in dayfile.txt https://cumuluswiki.org/a/Software#Create_Missing

Enjoy,
Paul

[Gyvate]

Well thanks again to all who've contributed. What have I learnt? Don't blame CMX for Ecowitt's failings! However, there will always be occasions of missing data and it's clunky to manually fit the data in (especially as the columns are different). I use VBA quite a lot, used it at work before I retired some years ago and have continued to use it in my own Access databases. I've started to use it in Excel (something I've never really used in anger). So I plan on creating a VBA script which reads the relevant CSVs and allocates the different values to the appropriate columns. I can then save the result as a CSV and squirt it into the monthly file or dayfile in the correct place - very easy (I hope).
However, the naming of the various fields isn't consistent, so I'm not sure of all the transformations. I'm attaching a simple spreadsheet with the monthly file attempt. Column A is all the monthly headers, Column B is just the column letters of these. Column C is the column are where the Ecowitt export columns are matched, and column D the console BU. My intention is to use the Ecowitt data as much as possible only using the console BU if necessary.I would appreciate it if someone can look at this and tell me where I can find the missing entries or where there are errors. Thanks in advance.
data_preparation.xlsx
p.s. If it works, then I would be happy to share this with others, but maybe I'm getting ahead of myself here!

Unfortunately I don't know what you mean by "BU".
If you enlighten me here, I might be able to support.

When your console posts to Ecowitt.net, you have two sources for collecting your past data
a) download an Excel file from Ecowitt.net (beware that the period [day, week, month ...] selected defines also the interval [5 min, 15 min, 30 min ...]
b) take the CSV files from the HP2551 console
b1) when a SD card is inserted, the console writes the basic sensor data (Temp/Hum out/in, wind, rain, solar, UV, winddir and calculated values like dewpoint) on the SD card AND into the internal memory. Also, when a SD card is inserted, a separate CSV file is written for the extra sensors (if you have one or more). New files are also created periodically - the names of the CSV files will tell.
b2) when there is no SD card inserted the basic sensor data is written to the internal memory. In order to retrieve it from there, you have to insert a SD card (FAT32, SDHC) and make a backup from the settings menu. Then all data in memory is written as a CSV file to the SD card.
-
But, when you are familiar with VBA, it should be easy for you to write one or more VBA modules which do the job as you described - read the CSV files, rearrange the columns, do some reformatting e.g. for date-time and create one (or two if you have extra sensors) CSV files (per month) in the CMX format.
Then it's just a matter of copy and paste into the mmmYYlog.txt file(s) - for the actual month file you will have to choose the time between two CMX writes (on the minute). The dayfile can be created/updated from the month file(s) with the CreateMissing.exe utility.

If you create such an utility to create csv files for CMX from the HP2551 console csv files, you would find a lot a people who'd like to use your solution.

A few more remarks regarding the FOSHK (clone) consoles - the only console which does a real logging is the WS2320E/WH4000SE console. However, the reading of this console's memory (3552 records - 1 record per chosen logging interval [from 16 seconds to one month]) can until now not be read by CMX. CMX so far only uses the custom server http posts from the console. And if the data logger (CMX) is not running, the posts will be missed. The WS2320E has an API which could be queried by CMX if that functionality were implemented. The WS2320E console comes with a dashboard program (WeatherSmartIP) which saves the data into a local MS Access database. If the program is shutdown and later on reconnected, it reads all the missing records (unless the blackout period was too long, see 3552 records) and saves them into the MS Access database.

Other consoles (like the HP350x, HP2551) "only" write to SD cards. These SD cards however - unfortunately - cannot be accessed by a computer program - manual user interaction is needed (take out SD card, replace it with another, then copy the CSV files from the SD card for further processing).

The main target group of FOSHK/Ecowitt are "normal" users and not the comparatively few enthusiasts like us who would like to access the logged data in the console memory and log data in software like CMX beyond the cloud solutions like WU or ecowitt.net which come with the console.
To speak here of "Ecowitt's failings" is maybe not the adequate choice of words.

However, Ecowitt is listening a lot to their customers, also to their more demanding customers.
An API to access the ecowitt.net web site (cloud) and download one's own actual and historical data from there is in the beta testing phase.
This won't keep you from having to transform/reformat your downloaded data to the CMX format, though, but it could be automatized.

The best way to solve the existing situation is to implement some data logger software like CMX (or Meteobridge or weewx or ....), but in any case this logger (software/hardware) has to run 24/7. The cheapest and still easy to implement solution is using a RaspberryPi computer, from RPi3+ upwards, i.e. also the latest versions like RPi4B-2GB, -4GB or -8GB. The safest way is (as @Linda already pointed out) to either use a SSD instead of a SD card, or use an industrial SLC SD card. 16 GB are enough for the operating system (OS) and CMX. Costs about 27 EUR/33 USD - but these days difficult to get due to COVID-19 side effects (www.arrow.com, www.mouser.com).

The installation is not a big deal - some RPi kits come with pre-installed OS (NOOBS) or one downloads the RaspberryPi Imager under Windows and can easily write the Raspbian/Debian image on the SD card. When one creates an empty file by the name "ssh" (without "") on the boot partition, one can access the RPi via SSH (PuTTY) and start the raspi-config program, activates VNC and can then access the RPi from a WIndows PC after installing a VNC client (TightVNC, RealVNC, ...), so no extra mouse, keyboard, monitor needed. Then the CMX program can be simply unpacked/unzipped into the /home/pi directory where /home/pi/CumulusMX will be created with all what is needed.
One more thing: under Debian/Raspian ["Linux"] the mono library is needed.
sudo apt-get install mono-complete
Is the command to execute in the command line windows. It's described in the WiKi.

Alternatively one can use a CMX-Raspi image (see Wiki) which also be installed on the SD via the Raspberry Pi Imager.

3rd option, if you don't want to enter the Linux world, is to install Windows 11 on a RPi4B-4GB and then CMX on it - one of my test installation runs on Win11 - perfect - astonishingly smooth. I'm talking here about 24/7 solutions using a Raspberry Pi.

I have consolidated my RPi4 "fleet" into a RPi4-8GB on which run three weewx instances and a CumulusMX instance.
It boots from a 480 GB SSD with a Geekworm setup
(https://www.amazon.com/Geekworm-NASPi-L ... 217&sr=8-3
or
https://www.amazon.co.uk/Geekworm-NASPi ... 198&sr=8-1)
There are comprehensive (and working !) instructions available on the internet how to make a RPi4B boot from a SSD instead from a SD card (SD card only needed initially for the first setup).

This has become a rather lengthy post - but it may also answer a few open questions.