Cumulus Support

Running CumulusMX on a Raspberry Pi, and (probably) like most others, I'm running it off a "surveillance" standard SD card.

One of the things on my to-do list is moving the OS and MX to some USB flash memory or SSD.

How long are people finding their SD cards are lasting? I know it's a theoretical risk, but are people actually seeing their SD cards go a bit dud after a while?

Cheers

Whitling2k

whitling2k wrote: ↑Fri 11 Mar 2022 12:40 pm Running CumulusMX on a Raspberry Pi, and (probably) like most others, I'm running it off a "surveillance" standard SD card.

One of the things on my to-do list is moving the OS and MX to some USB flash memory or SSD.

How long are people finding their SD cards are lasting? I know it's a theoretical risk, but are people actually seeing their SD cards go a bit dud after a while?

Cheers

Whitling2k

My cards work somewhere between 1 and two years. I make CMX backups to my NAS, daily/weekly depending on the interval and catch-up posssibilities, to prevent data loss in case of a true crash. I just moved to USB flash memory (16 Gb) on my RPi4 (too slow on the RPi3 if I understand it well).

whitling2k wrote: ↑Fri 11 Mar 2022 12:40 pm How long are people finding their SD cards are lasting?

My standard high capacity micro SD card remains ok, 19 months after purchase of my Raspberry Pi 3 B+ which runs continuously (except Storm Arwen 12 hour power cut last November), this card continues in use for boot/system files, but at least 6 months ago I stopped it being used for temporary system files.

I actually have 3 of these cards and use the provided "SD Card Copier" to periodically back up the card that is in use, alternately to one or other of spare cards kept in different rooms. Thus, should a crash happen, I would quickly replace the inserted card, keeping operating system and what I have installed using apt, with loss of only recent data.

Note: To reduce SD card wear, I edited the fstab file, adding the line tmpfs /var/tmp tmpfs nodev,nosuid,size=1M 0 0, obtaining access to RAM for storing folders/files. I generated symbolic links to redirect the existing tmp folder used by system, and to redirect each temporary file created by MX, so RAM has been used for temporary files for several months now.

whitling2k wrote: ↑Fri 11 Mar 2022 12:40 pm One of the things on my to-do list is moving the OS and MX to some USB flash memory or SSD.

I originally tried installing MX on USB flash memory to save wear on SD card, but abandoned that after experience with two flash memory devices in turn. In each case, they lasted a few months, then failed, worn out presumably just by MX writing temporary files.
Notes: 1) My RPi was left operating unattended when it had the flash memory inserted. 2) I tried the second flash memory as the electronics shortage in pandemic meant I could not buy a suitable SSD and connecting lead for several months. 3) MX was subsequently installed on the micro-SD card, and I started using RAM for all temporary files. 4) When SSDs and connecting leads were back in stock, 7 months ago, I achieved my desired switch to SSD.

My 7 month old SSD is used for MX folders/files, and for all user home folders/files.
I expect the SSD to last longer than what is left of my lifetime! Initially, I found my SSD occasionally disconnected, I can only guess passing lorries somehow induced vibrations. I seem to have cured that by buying an enclosure sold as being designed to hold the RPi board, a SSD, and a HAT board, although it was a long struggle to work out how to fit the boards/SSD, and assemble the various parts of that enclosure, as instructions were not provided.
Notes: 1) The SSD does not get used for temporary files, 2) I have not yet moved the boot/system folders/files onto SSD, 3) I have many files in home folders as since installing SSD, I use my RPi as my main computer.

I have two rapsberry pi's (one pi3 one pi 4) and both are running off SSDs. Both run 24/7 . The Pi 3 has been running for at least three years probably 4 tracking aircraft and
supplying to FR24 and others and the PI 4 has been running for two or three years with cumulusmx passing info to windy.com, weather underground and Met office WOW.

It is now used as my Linux test system as well. Both are copied to my windows box on a regular basis using win32disk image. Both SSDs come from old expired laptops whose batteries died and could not be re sourced!!! and they both had four or five years work before that. As a retired software engineer who fortunately loved his job the laptops were well used!!!

YMMV

I don't have a real problem with SD card longevity (many years). But you should note that:

1. SD cards do wear out if used for too many write cycles.
2. BUT SO DO USB Flash Drives.
3. AND SO DO SSDs [but after a longer time]

4. Mark Crossley has kindly provided the pre-built Raspberry Pi image. This is MUCH PREFERRED for use on RPis because it has been amended to minimise SD card /SSD writes by making use of a RAM drive instead.

Is there a list of the symbolic links required to minimise the sd/ssd writes?

My pi 4 system has several applications on it that I would prefer not to have to re install if I use the pre built Raspberry pi image.
I have also upgraded it to Bullseye.
As an old Windows programmer trying to get up to speed on Linux I am afraid this area is one I am currently struggling with.

I have created a RAM disk in /etc/fstab and also redirected my log files with the following:

tmpfs /var/tmpcmx tmpfs nodev,nosuid,size=3M 0 0 ##ram drive for cumulusmx stuff
tmpfs /var/tmp tmpfs defaults,noatime,mode=1777 0 0 ##make tmp files go to ram
tmpfs /var/log tmpfs defaults,noatime,mode=0755 0 0 ##make log files go to ram.

but now have no clue what command to run on what files ...!

As has been pointed out, the CMX Raspberry Pi image already has the CMX tmp files written to a memory based filesystem.

SSD drives generally have a much higher endurance than SD cards to begin with, and also because they tend to have a higher capacity, it means they have more memory locations to "spread the load" over. But capacity also applies to SD cards, using a bigger card than you have to means less wear on each memory cell as the cards wear levelling algorithm should even out the writes across the whole card.

Contents of post deleted given incorrect response in next post

sfws wrote: ↑Sun 13 Mar 2022 6:23 pm ... he did not read that you don't want to use the pre built image because that will lose the other packages you have loaded.

The OP has not written anything like that !

Cortmalaw wrote: ↑Sun 13 Mar 2022 10:08 pm

sfws wrote: ↑Sun 13 Mar 2022 6:23 pm ... he did not read that you don't want to use the pre built image because that will lose the other packages you have loaded.

The OP has not written anything like that !

Also, even if the OP had, what's to stop the OP from just installing the packages needed on top of the image (like I've done)? The pre-made image was a godsend for me coming from a way old version of CMX to this. I had a little challenges with the data files but once I had that sorted out it went incredibly smooth.

I really haven't added much to mine (things like nano instead of vi for the editor). It's been very easy to update, aside from some of the interface customizations I've made, to include things like making the wind chill block in the interface dynamic so it will change to heat index when the temp goes over 80F.

Contents of post deleted, given the response it got

sfws wrote: ↑Mon 14 Mar 2022 4:43 am 1) Read the words that follow "I would prefer", either in the quote in my post at viewtopic.php?p=162124#p162124, or in the post I quoted from at viewtopic.php?p=162115#p162115.

2) That was what I responded to, Cortmalaw. It is you who can't read the posts, not me.

3) The post at viewtopic.php?p=162121#p162121, could be clearer if it is actually arguing the greater capacity of the SSD makes it better than RAM for storing the temporary files, although that conflicts with post at https://www.wxforum.net/index.php?topic ... #msg404603

4) No wonder I rarely devote my precious time to reading, or contributing to, this forum.

I wish we were all as briljant as you dear @sfws.
Unfortunately we are lesser gods and need to deal with that in this hard problematic environment where we all try to trick each other into more problems than we originally posted. So forgive us because we do not know what we are doing.

And with respect to your point 4: if it really sucks here, just stay away. Better for yourself and for us.

NOTE: I suggest to the moderator to remove posts from @sfws which does not explain but which are only to point to others they make errors, are wrong and stupid. If you do please remove this post as well.

Folks,
My apologies for creating an argument but I have now resolved the issue.

As a retired (73) assembler and Windows programmer I have using this project to try and keep my brain active and learn more about Linux

My setup is non standard and I am sure that is not unique!! I have an old Maplin N96GY system and have split it up with the wind
sensors on the roof of the garage and the temp and rain sensor on top of the fence. As a result I bought two transmitters (the original
one was eating batteries anyway) and removed the console. The transmitters data is picked up by RTL433 package on the Pi, decoded and then sent
on to a hiveMQ broker on the Pi. I also have a BMP388 temp and pressure sensor by the Pi which (after debugging the code) sends its data to the
broker as well. I have written some python code to subscribe to the broker and process the messages and create the easyweather.dat file.

I have also upgraded to Bullseye and run Cumulusutils and that is the reason I would prefer to change as little as possible .

So yesterday I taught myself something about hard/soft links, linux ramdisk and tmpfs. I then put the CumulusMX Raspberry pi
image onto a spare SSD. After inspection I implemented the links on my own setup and it all seems to work. Because I
have implemented it to suit my configuration I will not add anything to the wi-ki but will note here that the files requiring
links to tmpfs were:

realtime.txt
wxnow.txt
realtimegaugesT.txttmp
websitedataT.jsontmp

Hopefully I haven't missed any but it all seems to work.

Hope this helps someone else and apologies again for creating an argument.


@sfws Thanks for the pointer to Web sub folder. I'll go have another read of the whole page!!

Thanks for the great conversation everyone - It's an interesting topic with lots of viewpoints.

It looks like my 32G 'surveillance' rated SD card will keep me going for a while whilst I get my head in order. Having said that, my worry is silent errors, mis-recorded parameters that slip through my monthly QA.

Ultimately, I am considering getting a second 8GB Pi 4 and running the entire CumulusMX (and separately, web-server & IP cam streams) on a ram drive; and copying the image back to an SSD nightly. Rather than constant micro-writes... Again... Time... or lack thereof.

Thinking out aloud - would that even make a difference? It's the same volume of data being written.

I do have a spare SSD, but it's quite high performance (gaming desktop, pre-M2 upgrade) and it seems a waste killing it slowly with such small writes on a Pi.

I switched to a "cheap" 128 GB SSD in April 2017, that has been running non-stop since, with my live CMX install, a test CMX install, my main MariaDB (with 5 second real-time data being stored), my Pi-hole DNS, and two public web sites running PHP - which also create quite a few temp files.

I don't know when it will fail - as it will one day - but it is holding up fine so far. Most SSD support the viewing of "wear statistics" and bad cell counts which give an indication of likely upcoming failure, last time I checked mine it was still fine.