Welcome to the Cumulus Support forum.
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4019) - 03 April 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024
Cumulus MX V4 beta test release 4.0.0 (build 4019) - 03 April 2024
Legacy Cumulus 1 release 1.9.4 (build 1099) - 28 November 2014
(a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)
Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki
Beginner errors
Moderator: HansR
-
- Posts: 159
- Joined: Wed 06 Jan 2021 3:01 pm
- Weather Station: Ecowitt
- Operating System: Raspberry Pi
Beginner errors
I've made an attempt to get cumulusutils going, without much success. Please can someone point me in the right direction? I know I have some incorrect data, but I'm hoping that is being ignored. Is this the case please, or is this my problem? Thanks for any help.
The website is local on my Raspberry Pi and will not be published publicly.
Enclosed is the latest logfile.
The website is local on my Raspberry Pi and will not be published publicly.
Enclosed is the latest logfile.
You do not have the required permissions to view the files attached to this post.
- HansR
- Posts: 5964
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: Beginner errors
Hi Linda, thanks for trying CumulusUtils.
You are running a RPi 3 and you have almost 2.5 million records (1 minute log frequency). That is quite a series but afaik this should fit onto the memory of the RPi3 although it takes some time to read (3.5 min so far in your case for a full website run; note this performance will be dramatically improved with version 7 coming up somewhere this year with the new CMX developments).
However, the real problem is that the log stops while or after creating the Wind Rose without any additional information so it does not seem an exception and I don't know what happened.
That is an unseen phenomenon and the only thing I can come up with without any additional information is that there is a memory error or you stopped it yourself because you thought it took too long? Might it be that it started to use the swap device? If that is the case you would be running against the limit of the RPi but that would surprise me because I know some have run it with more data. But yes, a five minute interval would be nicer
Having said all that, without more information I cannot say much more. I suspect the memory size so I would advice to monitor that aspect (open a second terminal and activate htop to check the memory while running CUtils). You could 'speed up' testing that by only running the graphs command as that is where the month files are read so if the problem is there you localise it (NOTE that it is not a real speed up because reading your monthly logfiles takes two minutes on your machine).
If that does not give decisive information and the log remains the same, I would invite you send me the whole data directory so that I can examine the situation, first on windows, then on my RPi3. It would be worth the trouble if I can improve CUtils!
Btw: you mention the website will not be public. You are not the only one who works like this, perfectly fine although I prefer public sites for CumulusUtils because:
Yes those errors are ignored. If there is more than 10 errors in a file the whole file is ignored but apparently you have set the parameter IgnoreDataErrors to true so it will always continue.
You are running a RPi 3 and you have almost 2.5 million records (1 minute log frequency). That is quite a series but afaik this should fit onto the memory of the RPi3 although it takes some time to read (3.5 min so far in your case for a full website run; note this performance will be dramatically improved with version 7 coming up somewhere this year with the new CMX developments).
However, the real problem is that the log stops while or after creating the Wind Rose without any additional information so it does not seem an exception and I don't know what happened.
That is an unseen phenomenon and the only thing I can come up with without any additional information is that there is a memory error or you stopped it yourself because you thought it took too long? Might it be that it started to use the swap device? If that is the case you would be running against the limit of the RPi but that would surprise me because I know some have run it with more data. But yes, a five minute interval would be nicer
Having said all that, without more information I cannot say much more. I suspect the memory size so I would advice to monitor that aspect (open a second terminal and activate htop to check the memory while running CUtils). You could 'speed up' testing that by only running the graphs command as that is where the month files are read so if the problem is there you localise it (NOTE that it is not a real speed up because reading your monthly logfiles takes two minutes on your machine).
If that does not give decisive information and the log remains the same, I would invite you send me the whole data directory so that I can examine the situation, first on windows, then on my RPi3. It would be worth the trouble if I can improve CUtils!
Btw: you mention the website will not be public. You are not the only one who works like this, perfectly fine although I prefer public sites for CumulusUtils because:
- That's what CumulusUtils was meant for
- It makes support so much easier (being able to see the console and the debugger)
- it shares inspiration for other users
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
-
- Posts: 159
- Joined: Wed 06 Jan 2021 3:01 pm
- Weather Station: Ecowitt
- Operating System: Raspberry Pi
Re: Beginner errors
Hans,
Thank you very much for replying to me.
The terminal just says ‘Killed’ when it stops running. I was going cross eyed when I gave up yesterday, I’ll have another try later on today. Do you think there would be any advantage to try temporarily removing some of the data files to cut down the amount of processing? I have almost 12 years worth of data that I transferred to Cumulus (it took me ages, and I’m still trying to correct it!!). I was going to reinstate the error login later when I’d got things going. It’s better to do things a bit at a time, then I understand better . I’ll also try running with just the graphs command to see what happens.
The other thing I could try is setting Cumulus up on my windows machine and trying the process there, do you think that would help? Presumably, if I could get it actually running I could copy the whole lot back to the Pi. I do have a non running CumulusMX on the PC as a test area for all of the data files. I learned to get the data as right as possible BEFORE putting it into the live area.
In the event that I still can’t get it running, do you want me to pm you with the whole data directory? There’s obviously quite a bit of it. Sorry about this, but also a bit relieved I’m sort of getting the hang of it. Thanks once again.
Linda
Thank you very much for replying to me.
The terminal just says ‘Killed’ when it stops running. I was going cross eyed when I gave up yesterday, I’ll have another try later on today. Do you think there would be any advantage to try temporarily removing some of the data files to cut down the amount of processing? I have almost 12 years worth of data that I transferred to Cumulus (it took me ages, and I’m still trying to correct it!!). I was going to reinstate the error login later when I’d got things going. It’s better to do things a bit at a time, then I understand better . I’ll also try running with just the graphs command to see what happens.
The other thing I could try is setting Cumulus up on my windows machine and trying the process there, do you think that would help? Presumably, if I could get it actually running I could copy the whole lot back to the Pi. I do have a non running CumulusMX on the PC as a test area for all of the data files. I learned to get the data as right as possible BEFORE putting it into the live area.
In the event that I still can’t get it running, do you want me to pm you with the whole data directory? There’s obviously quite a bit of it. Sorry about this, but also a bit relieved I’m sort of getting the hang of it. Thanks once again.
Linda
-
- Posts: 717
- Joined: Thu 20 Aug 2020 8:35 am
- Weather Station: Davis VP2 Plus
- Operating System: RPiv4 64bit Linux 12 Bookworm
- Location: Jersey Channel Islands
- Contact:
Re: Beginner errors
Hi Hans
I just looked at your reply noting that the program stopped at the wind rose creation for Linda's site.
This has been something I have noticed when running the website update and watching the generation in the terminal session. Whilst it never stopped for more than a couple of seconds the generation definitely slowed down as the background process worked. Presumable this is a resource hungry process at this point.
I just looked at your reply noting that the program stopped at the wind rose creation for Linda's site.
This has been something I have noticed when running the website update and watching the generation in the terminal session. Whilst it never stopped for more than a couple of seconds the generation definitely slowed down as the background process worked. Presumable this is a resource hungry process at this point.
- HansR
- Posts: 5964
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: Beginner errors
Linda,
To get things faster you could start with 2021 and 2022 and see where that goes. So yes from the POV of speed that could be a good idea.
It could be a data error somewhere but that would require some deeper research.
It will run
To be honest, that should not make a difference, as you can see it came up to 2022 so it did read all files. Then it starts calculating the WindRose which indeed is a time consuming process but there are others with about the same data and it works.LindaFNM wrote: ↑Sun 09 Jan 2022 10:03 am Do you think there would be any advantage to try temporarily removing some of the data files to cut down the amount of processing? I have almost 12 years worth of data that I transferred to Cumulus (it took me ages, and I’m still trying to correct it!!). I was going to reinstate the error login later when I’d got things going.
To get things faster you could start with 2021 and 2022 and see where that goes. So yes from the POV of speed that could be a good idea.
It could be a data error somewhere but that would require some deeper research.
Trying on windows if that is not your machine which you would use operational might speed up things but in the end you would have to get back to the RPi. So yes, it could help but it will not give a solution I think unless it is the memory. While looking for a data issue Windows might help.
Yes, zip it up and PM it.
Getting something started with CUtils/CMX apparently always has a steep learning curve.
It will run
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
- HansR
- Posts: 5964
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: Beginner errors
Hi Andy,
To improve overall performance (have these extreme resource hungry actions not executed so often) the Thrifty qualifier is introduced to e.g. do this only once a week (or once a year for that matter ) depending on how urgent you think it is.
But that is only useful after you have built it at least once
So having said all the above: it looks as if it stops but it never really does, so what Linda experiences is not normal.
Yes, the WindRose is kind of headache process wrt performance. Because I build it from the standard log data (which btw are also used for the solar energy charts but that goes much faster) it always requires the file reading which takes a lot of time and then the calculation itself also is resource hungry as the list of data has to be split in months and then in wind direction and then for every entry the speed for that direction is counted for that month. It is the repeated loops which takes the time. Already several improvements have been made, the latest improvement - which among others integrates the reading and calculation process) has not been released yet.AndyKF650 wrote: ↑Sun 09 Jan 2022 2:48 pm I just looked at your reply noting that the program stopped at the wind rose creation for Linda's site.
This has been something I have noticed when running the website update and watching the generation in the terminal session. Whilst it never stopped for more than a couple of seconds the generation definitely slowed down as the background process worked. Presumable this is a resource hungry process at this point.
To improve overall performance (have these extreme resource hungry actions not executed so often) the Thrifty qualifier is introduced to e.g. do this only once a week (or once a year for that matter ) depending on how urgent you think it is.
But that is only useful after you have built it at least once
So having said all the above: it looks as if it stops but it never really does, so what Linda experiences is not normal.
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
-
- Posts: 159
- Joined: Wed 06 Jan 2021 3:01 pm
- Weather Station: Ecowitt
- Operating System: Raspberry Pi
Re: Beginner errors
OK, I've been trying again. I ran it, got the same result, then ran as CheckOnly. Log attached.
I'm wondering whether to delete Utils, and start again? The other alternative is to sort all of the bad data.
I tried running htop but I'm so new to this I didn't understand it, and didn't realise it stopped when you Ctrl C.
Oops, tried running website again after a reboot, and I thought the pi had hung as everything froze. I think it may have been working, how long does the process take
Log attached
I'm wondering whether to delete Utils, and start again? The other alternative is to sort all of the bad data.
I tried running htop but I'm so new to this I didn't understand it, and didn't realise it stopped when you Ctrl C.
Oops, tried running website again after a reboot, and I thought the pi had hung as everything froze. I think it may have been working, how long does the process take
Log attached
You do not have the required permissions to view the files attached to this post.
- HansR
- Posts: 5964
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: Beginner errors
The log is the website run and stops after 2012.
Deleting Utils is not useful and start again won't change the data.
From you first log I would say there was just one more minute to go so it would have taken in total 4m30 for a complete run.
I suggest you zip up the data directory and send it.
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
-
- Posts: 159
- Joined: Wed 06 Jan 2021 3:01 pm
- Weather Station: Ecowitt
- Operating System: Raspberry Pi
Re: Beginner errors
Hans,
I have just read your conversation with Andy properly. I have some doubts seeing what your utility does that my data is in a fit state for this. It has been obtained from old records from WLINK and WSWIN and the quality of, especially the wind data is suspect. I would hate for you to waste valuable time trying to solve large amounts of dubious data. I’ve ended up getting less and less concerned about accurate records as I’ve gone on with this!! Also, the early records are every 5 minutes, then changed to every minute when I got the Ecowitt station. There are quite a few gaps too, as in the early days my Pi wasn’t very reliable, I suspect I’d got a setup problem somewhere. It’s only run properly since I re imaged it with Bullseye.
If you really want to progress with this ( and I’m not sure that it’s worth it?), I’ll sort the data folder out tomorrow. Afraid I’ve no idea how to zip from a Raspberry Pi at the moment!
Thank you very much for trying to help.
Linda
I have just read your conversation with Andy properly. I have some doubts seeing what your utility does that my data is in a fit state for this. It has been obtained from old records from WLINK and WSWIN and the quality of, especially the wind data is suspect. I would hate for you to waste valuable time trying to solve large amounts of dubious data. I’ve ended up getting less and less concerned about accurate records as I’ve gone on with this!! Also, the early records are every 5 minutes, then changed to every minute when I got the Ecowitt station. There are quite a few gaps too, as in the early days my Pi wasn’t very reliable, I suspect I’d got a setup problem somewhere. It’s only run properly since I re imaged it with Bullseye.
If you really want to progress with this ( and I’m not sure that it’s worth it?), I’ll sort the data folder out tomorrow. Afraid I’ve no idea how to zip from a Raspberry Pi at the moment!
Thank you very much for trying to help.
Linda
- HansR
- Posts: 5964
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: Beginner errors
Linda,
CumulusUtils can handle all day series wrt changes in intervals and gaps. If there are issues they will probably be in format and completeness per line.
So just zip up the datafolder (if necessary and easier copy to windows first) and send it.
Talking/writing takes time as well.
And to prevent too high expectations: I will not program to fix data. I will look at the data and see what needs to be done, where the problems are and how you can fix things. My contribution will be limited. Your description of what you did with the data I take as input.
CumulusUtils can handle all day series wrt changes in intervals and gaps. If there are issues they will probably be in format and completeness per line.
So just zip up the datafolder (if necessary and easier copy to windows first) and send it.
Talking/writing takes time as well.
And to prevent too high expectations: I will not program to fix data. I will look at the data and see what needs to be done, where the problems are and how you can fix things. My contribution will be limited. Your description of what you did with the data I take as input.
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
-
- Posts: 159
- Joined: Wed 06 Jan 2021 3:01 pm
- Weather Station: Ecowitt
- Operating System: Raspberry Pi
Re: Beginner errors
Hans,
Thank you, I will look at it. Also, my records since 26 December are not good as my WS80 broke. CumulusMX has been doing a brilliant job at keeping the data sane but I’m having to get daily records from my Davis station to input to the dayfile………
Linda
Thank you, I will look at it. Also, my records since 26 December are not good as my WS80 broke. CumulusMX has been doing a brilliant job at keeping the data sane but I’m having to get daily records from my Davis station to input to the dayfile………
Linda
-
- Posts: 159
- Joined: Wed 06 Jan 2021 3:01 pm
- Weather Station: Ecowitt
- Operating System: Raspberry Pi
Re: Beginner errors
Hans,
My sincere apologies for probably bombarding you with PMs. I had no idea how the system worked and had assume that showing a message in my outbox was like a ‘stuck’ email. I contacted the Mods yesterday who explained how the system works, now I understand. So sorry once again.
Linda
My sincere apologies for probably bombarding you with PMs. I had no idea how the system worked and had assume that showing a message in my outbox was like a ‘stuck’ email. I contacted the Mods yesterday who explained how the system works, now I understand. So sorry once again.
Linda
- HansR
- Posts: 5964
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: Beginner errors
Hi Linda,
That's OK, no bombardment seen, not even one message. So if you did send, something definitely has gone wrong
That's OK, no bombardment seen, not even one message. So if you did send, something definitely has gone wrong
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
-
- Posts: 159
- Joined: Wed 06 Jan 2021 3:01 pm
- Weather Station: Ecowitt
- Operating System: Raspberry Pi
Re: Beginner errors
I give up ! Next stupid question, and the reason for all of the phantom PMs, I have approx 186 files in the data folder and despite zipping them in 4 sections, the file uploader says they’re too big…… Do you want/mind dozens of files, or do you have any objections to me using your email address on your website please? Thank you.
Linda
Linda
- HansR
- Posts: 5964
- Joined: Sat 20 Oct 2012 6:53 am
- Weather Station: GW1100 (WS80/WH40)
- Operating System: Raspberry OS/Bookworm
- Location: Wagenborgen (NL)
- Contact:
Re: Beginner errors
Sent you a PMLindaFNM wrote: ↑Wed 12 Jan 2022 12:24 pm I give up ! Next stupid question, and the reason for all of the phantom PMs, I have approx 186 files in the data folder and despite zipping them in 4 sections, the file uploader says they’re too big…… Do you want/mind dozens of files, or do you have any objections to me using your email address on your website please? Thank you.
Linda
Hans
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3