Cumulus Support

Create an official docker image - preferably with options for cumulus utils?

I personally think it would be a great idea and bring more people to the project that would have passed it over before.


thanks
Ian.

Actually, I think you are probably wrong.

My impression is that for large number of users this is the first time they have even used Linux and/or Raspberry Pis, and the number who would know what docker is, let alone have the ability to use a docker image would be tiny.

For me there isn't enough time to keep up with the work needed on Cumulus and the rPi image let alone take on creating/maintaining/releasing docker images. And Cumulus Utils is an independent project, I know Hans keeps it aligned to CMX, but that would add another layer of complexity and cooperation.

For those of us who are tech savvy it is sometimes difficult to see the larger number who just want to get their own weather station online. Indeed I have seen posts regarding the decline in users for software like CMX, WD etc, because the PWS manufacturers are including the ability to upload to cloud solutions in their products.

There is already one docker image out there, maybe a small group of people could collaborate on that, or create their own and keep it maintained, one man efforts tend to die a death eventually.

Second Marks remark.

So I'm going to try hard and be really nice....

Your native installation process is a dirty <insert explicitive> mess.

Seriously - you have either a one stop shop of installing an image or a 40 step process of installing the software. Have you never heard of scripting?

Oh you want to install pihole?

curl -sSL https://install.pi-hole.net | bash

done - oh you have a nice eink LCD and you want to install that?

curl https://get.pimoroni.com/inky | bash

Oh you have an eviro+ and want to measure air, light, gas and other things?

git clone https://github.com/pimoroni/enviroplus-python
cd enviroplus-python
sudo ./install.sh

Oh - you want to install cumulus on a computer that has been used more than once?

viewtopic.php?t=17895


WHAT? You think I'm wrong - which is fair enough, but most people are now realizing that when they send their data to weather underground they don't get it back. It's not 'their's' and fair, ecowiit allow you to export your own data but you still have to do something with it.


I was actually considering a native install because I've managed to bugger up my docker one by installing cutils - it worked perfectly until I tried to install that, which I think I made a mess of - and now even when I go back to the default docker one (which I had to compile because it was never pushed to the hub) I can no longer get it to update the local site to publish data.


I installed docker
I compiled the executable (which would not be required with an image)
I created a docker compose file
I added this
and bang - it was up and running. Now as I said, with docker hub images it would have been much faster - the contributor never did this, but you would need armhf arm64 and amd64 compiles. However it turns a 40 step (slight exaggeration) process into something that's really, really simple.



So as it stands, I have a lot of learning to do - but if I need to hack the living poop out of optoisolated attempt I will. And if I need to learn how to push to dockerhub I'll be doing that too - but you really could be making this MUCH more accessible to normies just by doing this - yet you refuse and yet this idiot here has a 40 container stack running on a raspberry pi (8GB) with little problem.


I'm seriously considering going back to optoisolated attempt and cutting the cruft back to a very basic install (without nginx) and then compiling and publishing to docker hub - then producing the docker-compose files so that it can be as simple, or as complex as it needs to be.

I'm also 40+ I'm slightly disappointed that the people here appear to have succumbed to complexity when they could have their image pushed by https://www.linuxserver.io/ or some other organization to breathe back new life into this project - to be frank the only reason I came here was that my poop bresser wouldn't work with weewx and I bought an ecowitt and I was looking to find easier options to produce fantastic data with minimal effort.

I still have MQTT publishing the info - I'm still struggle bussing trying to get telegraf to pick it up and put it into influxdB - and I've not even tried to get grafana to make it all nice and prutty.

But for goodness sake - is it really that hard to compile 3 or 4 dockerhub images? Or do you want to venture further down the path of oblivion?

That is the constructive kind of post we all need.
Ever considered just picking up the code and compile it yourself into what you believe is the best way to go?
As it is very easy as you say, you might just have to push the button for new releases.

You happy, maybe some others happy and thus the big step for mankind has been taken.
CMX is public domain so create your project with the code and publish back here on the forum. You never know, it might catch on.

How is that for an idea?

NB: I am only making CumulusUtils and I really don't have a clue about Docker but if it's the Holy Grail, please show me.

So I'm in the process of setting things up, I've compiled the binary currently for arm64,

From optoisolated code, I've removed nginx to give people the choice - but I might add it back depending on thoughts. I've added lshw to make it compatible with cutils and currently there is NO USB support. I would really like to have a build that does both but admittedly this is a current limitation as I don't want to build more than I need right now.

I have a few options as to how to install docker-compose and set it up correctly, but I've not configured a vanilla environment yet to know if it will work.
For those that already use Docker and Docker compose, just put the above code into your docker-compose.yml and start the stack. I'll probably change it to create a folder in the users home directory and store the data there so that the data is persistent, but this currently keeps everything contained.

I've forked optoisolated git project and will update everything there once it's all finalised.

I'm likely going to use a script to install docker etc in the meantime if you would like to have a poke about in the weeds - I recommend this program to get the basics installed.

https://sensorsiot.github.io/IOTstack/Basic_setup/
As it currently stands, you then need to find the docker-compose.yml in the IOTstack folder and add the code further above but in the future, I might see if I can get cumulusMX added as a package. In the beginning I'll be using this script set to test but it's also a good set of tools.

I've added lazydocker as a nicer way to view docker running - however something like portainer might be a more powerful tool for those interested and I've added bpytop for monitor things in an enjoyable way (although a container such as glances could be an option)

I am puzzled as to what the benefit is of using a docker. Running CMX on Windows is a no brainer, just load it up and run it since 99% of Windows installs have .net already. Running it on Linux for me was equally a no brainer, install mono using my usual package manager and load and run CMX with no issues at all. I can see no reason at all why Mark should be involved, fine if people want to create a docker image but to me it seems more complex than running CMX natively in Linux or Windows plus updating CMX in such an environment means I'd have to wait for the people who build the image to update it, test it and release it. In my case I can simply load the new release in Linux or Windows immediately and run it. Also if I'm invited to beta test then it is easy to run a test version - not possible with a docker image.

Now as I said I see no reason why someone should not create and support such an image if they wish to. I just do not see any benefit.

Stuart

After my half drunken birthday rant (above) I've put my foot in it.

Disclaimer aside, I'll address your points. I can understand why you feel that way - I will however ignore your comment on windows. I just have a problem taking Windows seriously anymore.

Running it on Linux for me was equally a no brainer, install mono using my usual package manager and load and run CMX with no issues at all.

I agree. and then shoe in anything else you have running on port 80 And then you have security, is mono fully secure? what about the CMX exe? Is it not better to keep those items contained from your main system, especially if you are hosting a public website? What if there is a bug and it brings down your entire system? And convenience? I appreciate that you might only have CMX running on a system but with my data on a different drive/location, regardless of what happens - if I have my docker-compose file, I can bring my 40+ containers back online in less than half an hour and that's anything from asking Siri what airplanes are flying overhead to code-server, plex or home assistant.

I can see no reason at all why Mark should be involved, fine if people want to create a docker image but to me it seems more complex than running CMX natively in Linux or Windows

I agree. And while I hoped the benefits of such a system would be self-explanatory and easy for the 'kids' that already have something like this setup to add to their ever expanding 'internet of things' I appreciate that if that's not his wheelhouse or interest - well that's up to him (sorry about speaking about you in the third person). Once you have docker and docker-compose installed - it's not that complicated, I intend to show that, however admittedly IOTstack is throwing a spanner into the works, and I need to make things less complex. I do totally admit that if you want to install it in windows and just let it go - then go you. I will also add here that originally, I thought potentially taking and using the code without permission may have been some form of violation, so knowing that it's freeware and all the other tools I'm using are GNU then that does ease my frustration.

plus updating CMX in such an environment means I'd have to wait for the people who build the image to update it, test it and release it.

I have a big mix of geekery and nerdery in me that if I get this up and running the way I wish I will look to compile anything that Mark puts out in as speedily a time as possible. There is nothing worse than watching perfectly good projects go to waste and not be updated months if not years later. That's a compromise that I'm willing to make because if I want the latest things installed then I must contribute to the community and put the work in.

Also if I'm invited to beta test then it is easy to run a test version - not possible with a docker image

I agree. However, by saying that you are putting yourself in the same place I am by being a small percentage of the type of people that will use the software.

Now as I said I see no reason why someone should not create and support such an image if they wish to. I just do not see any benefit.

And then you have me - someone that had a Youshiko /bresser unit that gained a fault in the humidity sensor and a frustration in being unable to export the data at the best of times - to an ecowitt unit and the intention of getting some nice lightning readings charted into grafana. I'm no further closer into getting cutils installed - however it might be easier with docker, but apparently, I'm maintaining three architectures of software I didn't write - yay me.

I did learn how to compile and push to dockerhub today; however, I need to understand tags more for architecture. I guess if I can give back a little for all the people that build images for me and save me hundreds of hours of my time over the years - then why not?

Thank you broadstairs, for the feedback, Although I do feel the UI of cumulus is a little long in the tooth, I really do think the project has a place in the future and I intend to help keep it there. Now - I wonder, do I need to compile something on target architecture? I feel I should be able to cross-compile but I think that's research for another day.

Apparently it wasn't research for another day ...
Hardware agnostic against amd64, arm32v7, arm64v8 - if anyone is willing to check any of these untested architectures I'd really appreciate it.

I'm afraid your reply to my post Nossie does not convince me in the slightest. My weather has always been run on a separate PC not used for anything else and actually it is the ONLY PC I have running Windows, and that is because another piece of Weather software I use only runs under windows (although I'm trying to get the author to change that. In running that PC since 2005 I've had zero issues with hackers or viruses. The only issue I've ever had with a virus was to open a file from someone I though I could trust I've been in the computer world as a professional from 1966 yes way before PCs and I worked with one of the very first IBM PCs prior to its release to the public. Docker to me is an unnecessary complication and frankly I think most new users of CMX are likely to go for an RPi with the supplied image and run nothing else on that RPi, if they are really worried about security they will run that isolated from their home lan. As for the install process I think it is very slick and the default interface is basic yes but it works, also the basic web site works and you know what they say "if its not broken don't fix it". Most real enthusiasts will end up either using an existing template or build their own website. Also in your docker container how are you going to cope with required extras like MySQL or the like or the need to run other software like WXSim perhaps which is Windows only but requires access to the data files?

I think most people who start with an AWS and are not PC savvy will either go the route to use the cloud like with Davis or Ecowitt and perhaps use an Android tablet as a console for Ecowitt. They may move on to a PC solution later if required and I suspect probably 75% or more will use Windows because try buying a PC without it! Some will undoubtedly use an RPi with a ready built mage.

Stuart

The only issue I've ever had with a virus was to open a file from someone I though I could trust

The above is the best reason to stop this project. Trust. I obviously don't have any and I don't deserve any. Outside of reliability and that new builds would be pushed in a timely manner. There is nothing to suggest I couldn't sneak in a cryptominer or some other nefarious code into the build. If you build your own, it might take more time, but you eradicate most of that possibility. That of course would be a non-issue if the developer created the builds ..... but as that request has already been declined \o/

I've been in the computer world as a professional from 1966 yes way before PCs and I worked with one of the very first IBM PCs prior to its release to the public. Docker to me is an unnecessary complication

That's quite an impressive tenure - I think computing from then until about the late 90s was the most interesting - prior to the complete commoditization of the industry. I dream of an alternative world where Commodore didn't become incompetent, and BeOS won out against Windows. However, to paraphrase what you have said "Docker is an unsecesary complication" and "if it's not broke don't fix it" Back in your day a computer 'professional' had to be someone that could understand assembly and trace faults on TTL chips, these days if you can cobble some crap together in visual basic and connect two switches together you are a 'professional'. I'm far from dismissing your expertise, I'm quite envious of the lore, insight and experience you must have built up in that time and you have likely forgotten more in your life than the majority of 'IT pros' today learn in their lifetime. But then you say this:

Also in your docker container how are you going to cope with required extras like MySQL or the like or the need to run other software like WXSim perhaps which is Windows only but requires access to the data files?

The majority of that compose file above is mapping out folders that are fully accessible on the local system. In my own case the data is stored in a completely different location than the container and nightly it's cloned over to a backup drive and then moved offsite. Even in my current setup I could easily share that data across my network and import it into WXSim natively on a windows workstation. You are very quick to dismiss docker and yet that data is persistent and independent and easily accessible if you wish by fundamental design. In my configuration the binaries are contained to run completely independently, the data files are shared and currently I have setup to push in cutils when I get around to it.



With the greatest respect, from my perspective, most of the people that have responded here seem quite happy to trundle along with their pdp11 and VAX machines punching out tape to feed into their statistics collectors for prosperity because it's 'not broke'. While the 'young kids' are exporting their data from their weather stations, importing it into home assistant and controlling their thermostats lights and irrigation systems based on data collected from their local environment. Do I believe in change for the sake of change? No but when I use this software and browse the forum - I feel like I'm stuck in a time bubble from the 90 to early 2000s. Is that because the codebase is very well featured but elderly, is that also because it's the weather and most of the consumers are over the age of retirement, with a lot of time on their hands? Maybe it's a bit of both - but I've still not got over the fact it's a UK forum with a lot of UK people, all talking about the weather. Stereotyping much?

Anyway, I produced my proof of concept and going forward I'll create my own stuff for personal use only. I came to the realization that I originally thought this project was on life support - but it's not and even just a few months ago new features were added. So I would have the added complexity of outputting multiple builds and trying to debug features and devices that I don't have access to. Because an ample collection of people still uses usb connected devices, the next part of my project would have been to get that working and configured using a switch in the configuration. To meet the expectations of the forum, I think that would have been required - even though I don't use it personally.

So with that in mind and hopefully not insulting the community too much - I will withdraw public support

this project was on life support - but it's not and even just a few months ago new features were added.

Not sure how you got this impression Mark is releasing so many builds it hard to keep up (not that I am complaining as an Ecowitt user where most of that weather stations functions have been added in the last year!!).

Only today there was a new release with a bucket load of new features and as an Alpha/Beta tester I also know there is a major upgrade under test by some of us as we speak.

I have been in the computer industry almost as long as broadstairs (he beats me by 2 years), but the last thing I would discourage is any individual making code available that makes our lives more interesting.

We do indeed have people I am sure using CumulusMX that would like a docker image, it is an easy way for instance someone with a Synology NAS to get started!!

If you are willing to keep up with Mark and support people who use your image please go ahead and create the docker images, it improves the visibility of CumulusMX, which benefits us all in the long run.

I'm a little late to the party but I'll put my piece in.

I'd love a docker container, it's much easier to manage with things like port mapping controlled by a fairly simple yaml file. I run nothing locally these days which makes it easier for me to spin something up, test it out, and destroy it if I don't want to run it long term without ending up with a system filled with endless packages required for random things I tried once.

If the author isn't interested in building one that's ok, I appreciate the hard work they're putting in on the application and if I had a choice I'd certainly favour improvements to that over a container.

There is nothing to suggest I couldn't sneak in a cryptominer or some other nefarious code into the build. If you build your own, it might take more time, but you eradicate most of that possibility. That of course would be a non-issue if the developer created the builds ..... but as that request has already been declined

I tend to disagree here. I suspect it would be far easier for a developer to bury something in the code than it would to build something into a container. Containers can be inspected in quite a bit of detail by anyone that knows basic Linux if they want, an exe is much harder to dig through.

The mitigation is open sourcing so people can review everything but even though the source of Cumulus MX is open, how many people actually review that code for risks? That the code is open suggests that there's nothing nefarious going on but it isn't a guarantee. You can solve the container build problem in the same way.

Anyway, that's all a bit of a opining. What I wanted to say was that like you @nossie I used the work of optoisolated to get started. I very quickly decided I'd like a published container that anyone can use so I went hunting around docker hub to find one. Unfortunately both of those were months out of date (including yours) so they were quite a few versions behind.

I decided to build my own instead and I didn't want to run into the usual some person sets up a build, pushes it, then disappears problem that plagues open source so I set up an automated build pipeline so that every time a new version is released a build is kicked off and tagged with that version number within a day.

Naturally if I'd waited a couple of days your newest build would have been there and I'd never have gone down that rabbit hole :/

If you're interested I'd be happy to collaborate with you on your container (looks like you're doing a lot more work on testing than I am!) if you'd like to set an automated build up.

I would even go as far as to say if the Cumulus MX team were happy to have contributors to an official docker container repo under the Cumulus MX banner then I'd be happy to put my hand up to contribute to that.

Hi hucknz

I'm busy at the moment so wont be able to respond properly until later - when you said my image was out of date, do you mean the pi image or another architecture? The build I push to docker hub should be the most UTD version - atleast for ARM64. I did start building multi arch, but then the above thread and I realized I didn't have the time to test every single build, so I put that on the back burner.

- the Arm64 build runs on my pi, and pushes to my website as www.stewartonweather.com

As a side note, one of the things that made the builds more difficult is I'm having issues forcing the image to create files and folders other than root. But admittedly I didn't go back to that bit.

Anyway, quick note to point out I am still doing this (if only quietly hah) I might push another multi arch image and see what happens \o/

So far I've been managing to keep up with the official builds (for arm64 anyway) within a couple of days of the dev releasing, I'm a little worried about your observation, but it might be because you haven't been using ARM64?

.... to be continued in a number of hours

Hey nossie, I was doing this a couple of weeks ago so the latest version at the time was (if I recall correctly) ~2 months old. I think all the builds were the same date but I am running it on x64 so you could be right about the architecture.

I wanted a build with the latest version so I set the automatic build up, probably at about the same time as you were releasing your latest build of course!

I've got far simpler needs for the container so I haven't had to do any complex scripting to set up extra stuff as part of the build process. My main goal was something that I could just setup and leave alone (more little people joining the family soon so time will be short for a while) that will keep my system up-to-date without me doing anything. I'm still learning about the other bits and pieces you can add to Cumulus so who knows, it may get more complicated down the line.

I just took a quick look at your latest builds and it does look like there might be something going wrong with them. There are a bunch of variants tagged (amd64, arm v7, etc.) but docker hub is identifying them all as linux/arm64/v8 builds and 4/5 are exactly the same size. Might be worth a look.