Beginner errors

[LindaFNM]

I've made an attempt to get cumulusutils going, without much success. Please can someone point me in the right direction? I know I have some incorrect data, but I'm hoping that is being ignored. Is this the case please, or is this my problem? Thanks for any help.

The website is local on my Raspberry Pi and will not be published publicly.

Enclosed is the latest logfile.

[HansR]

Hi Linda, thanks for trying CumulusUtils.

I know I have some incorrect data, but I'm hoping that is being ignored.

Yes those errors are ignored. If there is more than 10 errors in a file the whole file is ignored but apparently you have set the parameter IgnoreDataErrors to true so it will always continue.

You are running a RPi 3 and you have almost 2.5 million records (1 minute log frequency). That is quite a series but afaik this should fit onto the memory of the RPi3 although it takes some time to read (3.5 min so far in your case for a full website run; note this performance will be dramatically improved with version 7 coming up somewhere this year with the new CMX developments).

However, the real problem is that the log stops while or after creating the Wind Rose without any additional information so it does not seem an exception and I don't know what happened.
That is an unseen phenomenon and the only thing I can come up with without any additional information is that there is a memory error or you stopped it yourself because you thought it took too long? Might it be that it started to use the swap device? If that is the case you would be running against the limit of the RPi but that would surprise me because I know some have run it with more data. But yes, a five minute interval would be nicer

Having said all that, without more information I cannot say much more. I suspect the memory size so I would advice to monitor that aspect (open a second terminal and activate htop to check the memory while running CUtils). You could 'speed up' testing that by only running the graphs command as that is where the month files are read so if the problem is there you localise it (NOTE that it is not a real speed up because reading your monthly logfiles takes two minutes on your machine).

If that does not give decisive information and the log remains the same, I would invite you send me the whole data directory so that I can examine the situation, first on windows, then on my RPi3. It would be worth the trouble if I can improve CUtils!

Btw: you mention the website will not be public. You are not the only one who works like this, perfectly fine although I prefer public sites for CumulusUtils because:

1. That's what CumulusUtils was meant for
2. It makes support so much easier (being able to see the console and the debugger)
3. it shares inspiration for other users

[LindaFNM]

Hans,

Thank you very much for replying to me.

The terminal just says ‘Killed’ when it stops running. I was going cross eyed when I gave up yesterday, I’ll have another try later on today. Do you think there would be any advantage to try temporarily removing some of the data files to cut down the amount of processing? I have almost 12 years worth of data that I transferred to Cumulus (it took me ages, and I’m still trying to correct it!!). I was going to reinstate the error login later when I’d got things going. It’s better to do things a bit at a time, then I understand better . I’ll also try running with just the graphs command to see what happens.

The other thing I could try is setting Cumulus up on my windows machine and trying the process there, do you think that would help? Presumably, if I could get it actually running I could copy the whole lot back to the Pi. I do have a non running CumulusMX on the PC as a test area for all of the data files. I learned to get the data as right as possible BEFORE putting it into the live area.

In the event that I still can’t get it running, do you want me to pm you with the whole data directory? There’s obviously quite a bit of it. Sorry about this, but also a bit relieved I’m sort of getting the hang of it. Thanks once again.

Linda

[AndyKF650]

Hi Hans
I just looked at your reply noting that the program stopped at the wind rose creation for Linda's site.

This has been something I have noticed when running the website update and watching the generation in the terminal session. Whilst it never stopped for more than a couple of seconds the generation definitely slowed down as the background process worked. Presumable this is a resource hungry process at this point.

[HansR]

Linda,

Do you think there would be any advantage to try temporarily removing some of the data files to cut down the amount of processing? I have almost 12 years worth of data that I transferred to Cumulus (it took me ages, and I’m still trying to correct it!!). I was going to reinstate the error login later when I’d got things going.

To be honest, that should not make a difference, as you can see it came up to 2022 so it did read all files. Then it starts calculating the WindRose which indeed is a time consuming process but there are others with about the same data and it works.

To get things faster you could start with 2021 and 2022 and see where that goes. So yes from the POV of speed that could be a good idea.
It could be a data error somewhere but that would require some deeper research.

The other thing I could try is setting Cumulus up on my windows machine and trying the process there, do you think that would help?

Trying on windows if that is not your machine which you would use operational might speed up things but in the end you would have to get back to the RPi. So yes, it could help but it will not give a solution I think unless it is the memory. While looking for a data issue Windows might help.

In the event that I still can’t get it running, do you want me to pm you with the whole data directory?

Yes, zip it up and PM it.

Sorry about this, but also a bit relieved I’m sort of getting the hang of it. Thanks once again.

Getting something started with CUtils/CMX apparently always has a steep learning curve.
It will run

[HansR]

Hi Andy,

I just looked at your reply noting that the program stopped at the wind rose creation for Linda's site.

This has been something I have noticed when running the website update and watching the generation in the terminal session. Whilst it never stopped for more than a couple of seconds the generation definitely slowed down as the background process worked. Presumable this is a resource hungry process at this point.

Yes, the WindRose is kind of headache process wrt performance. Because I build it from the standard log data (which btw are also used for the solar energy charts but that goes much faster) it always requires the file reading which takes a lot of time and then the calculation itself also is resource hungry as the list of data has to be split in months and then in wind direction and then for every entry the speed for that direction is counted for that month. It is the repeated loops which takes the time. Already several improvements have been made, the latest improvement - which among others integrates the reading and calculation process) has not been released yet.

To improve overall performance (have these extreme resource hungry actions not executed so often) the Thrifty qualifier is introduced to e.g. do this only once a week (or once a year for that matter ) depending on how urgent you think it is.

But that is only useful after you have built it at least once

So having said all the above: it looks as if it stops but it never really does, so what Linda experiences is not normal.

[LindaFNM]

OK, I've been trying again. I ran it, got the same result, then ran as CheckOnly. Log attached.

I'm wondering whether to delete Utils, and start again? The other alternative is to sort all of the bad data.

I tried running htop but I'm so new to this I didn't understand it, and didn't realise it stopped when you Ctrl C.

Oops, tried running website again after a reboot, and I thought the pi had hung as everything froze. I think it may have been working, how long does the process take

Log attached

[HansR]

OK, I've been trying again. I ran it, got the same result, then ran as CheckOnly. Log attached.

The log is the website run and stops after 2012.

I'm wondering whether to delete Utils, and start again? The other alternative is to sort all of the bad data.

Deleting Utils is not useful and start again won't change the data.

Oops, tried running website again after a reboot, and I thought the pi had hung as everything froze. I think it may have been working, how long does the process take

From you first log I would say there was just one more minute to go so it would have taken in total 4m30 for a complete run.

I suggest you zip up the data directory and send it.

[LindaFNM]

Hans,

I have just read your conversation with Andy properly. I have some doubts seeing what your utility does that my data is in a fit state for this. It has been obtained from old records from WLINK and WSWIN and the quality of, especially the wind data is suspect. I would hate for you to waste valuable time trying to solve large amounts of dubious data. I’ve ended up getting less and less concerned about accurate records as I’ve gone on with this!! Also, the early records are every 5 minutes, then changed to every minute when I got the Ecowitt station. There are quite a few gaps too, as in the early days my Pi wasn’t very reliable, I suspect I’d got a setup problem somewhere. It’s only run properly since I re imaged it with Bullseye.

If you really want to progress with this ( and I’m not sure that it’s worth it?), I’ll sort the data folder out tomorrow. Afraid I’ve no idea how to zip from a Raspberry Pi at the moment!

Thank you very much for trying to help.
Linda

[HansR]

Linda,

CumulusUtils can handle all day series wrt changes in intervals and gaps. If there are issues they will probably be in format and completeness per line.

So just zip up the datafolder (if necessary and easier copy to windows first) and send it.
Talking/writing takes time as well.

And to prevent too high expectations: I will not program to fix data. I will look at the data and see what needs to be done, where the problems are and how you can fix things. My contribution will be limited. Your description of what you did with the data I take as input.

[LindaFNM]

Hans,

Thank you, I will look at it. Also, my records since 26 December are not good as my WS80 broke. CumulusMX has been doing a brilliant job at keeping the data sane but I’m having to get daily records from my Davis station to input to the dayfile………
Linda

[LindaFNM]

Hans,

My sincere apologies for probably bombarding you with PMs. I had no idea how the system worked and had assume that showing a message in my outbox was like a ‘stuck’ email. I contacted the Mods yesterday who explained how the system works, now I understand. So sorry once again.

Linda

[HansR]

Hi Linda,

That's OK, no bombardment seen, not even one message. So if you did send, something definitely has gone wrong

[LindaFNM]

I give up ! Next stupid question, and the reason for all of the phantom PMs, I have approx 186 files in the data folder and despite zipping them in 4 sections, the file uploader says they’re too big…… Do you want/mind dozens of files, or do you have any objections to me using your email address on your website please? Thank you.
Linda

[HansR]

I give up ! Next stupid question, and the reason for all of the phantom PMs, I have approx 186 files in the data folder and despite zipping them in 4 sections, the file uploader says they’re too big…… Do you want/mind dozens of files, or do you have any objections to me using your email address on your website please? Thank you.
Linda

Sent you a PM