Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.28.5 (build 3282) - 23 February 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Monthly Log file has stopped updating

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

PaulMy wrote: Tue 28 Nov 2023 7:32 pm Hi,
Your status page shows that realtime is not always current https://maryborough-wx.xyz/wxstatus.php
Your realtime seems to be set at 5 seconds but is not updating for usually more than 1 or 2 minutes or more

Code: Select all

29/11/23 05:12:49 21.8 85 19.1 0 0 194 0.0 10.4 1009.6 SSW 0 km/h C hPa mm 15.4 +0.3 95.0 605.4 7.0 25.8 74 21.8 +0.1 25.1 11:41 21.4 23:40 3 15:18 11 15:11 1012.8 09:00 1008.3 02:49 3.27.1 3263 3 21.8 28.5 0.0 0.00 0 194 0.0 2 1 0 SSW 1077 ft 25.0 0.0 6 0 24.3
and next update

Code: Select all

29/11/23 05:14:44 21.8 85 19.2 0 0 194 0.0 10.4 1009.5 SSW 0 km/h C hPa mm 15.4 +0.3 95.0 605.4 7.0 25.9 74 21.8 +0.1 25.1 11:41 21.4 23:40 3 15:18 11 15:11 1012.8 09:00 1008.3 02:49 3.27.1 3263 2 21.8 28.6 0.0 0.00 0 194 0.0 2 1 0 SSW 1078 ft 25.1 0.0 8 0 24.4
your diag logs show

Code: Select all

2023-11-29 04:02:24.361 Realtime[0]: Not all data is ready, aborting process

2023-11-29 04:02:29.353 Realtime[1]: Not all data is ready, aborting process
2023-11-29 04:02:34.355 Realtime[2]: Start cycle
2023-11-29 04:02:34.356 Realtime[2]: Creating realtime.txt

2023-11-29 04:02:39.355 Realtime[3]: Start cycle
2023-11-29 04:02:39.355 Realtime[3]: Warning, a previous cycle is still processing local files. Skipping this interval.
2023-11-29 04:02:39.380 Realtime[3]: End cycle

2023-11-29 04:02:34.404 Realtime[2]: Creating realtime file - realtimegauges.txt
2023-11-29 04:03:24.356 Realtime[12]: Start cycle
2023-11-29 04:03:24.356 Realtime[12]: Warning, a previous cycle is still processing local files. Skipping this interval.
2023-11-29 04:03:24.357 Realtime[12]: End cycle
2023-11-29 04:03:25.891 FTP[2]: Error uploading /domains/maryborough-wx.xyz/public_html/realtimegauges.txt : Error while uploading the file to the server. See InnerException for more info.
2023-11-29 04:03:25.892 FTP[2]: Inner Exception: Code: 550 Message: /domains/maryborough-wx.xyz/public_html/realtimegauges.txttmp: Temporary hidden file /domains/maryborough-wx.xyz/public_html/.in.realtimegauges.txttmp. already exists
2023-11-29 04:03:25.892 Realtime[2]: Real time files complete
https://maryborough-wx.xyz/realtime.txt
https://maryborough-wx.xyz/realtimegauges.txt
I am just pointing out, but don't know what to suggest to fix it.
So since the 10th of November, my monthly data logs have stopped uploading to Cumulus (however everything else like real-time, dayfile, uploading to my website etc works fine Essentially, this is just leaving me with a blank gap in my weather history from the 10th (which outputs in my NOAA data). I can't remember anything changed on this day so I have no idea how or why this happened. I would be extremely grateful if anyone could help me with this and maybe provide some info on how I might go about getting it working again. The box it runs on is scheduled to reboot at 4am everyday so obviously a simple reboot hasn't been enough to fix it. Thanks.
Strange that your status page shows NOAA update is current and that your NOAA page is only up to November 10 https://maryborough-wx.xyz/Reports/NOAAMO1123.txt

Did you by chance change to PHP Upload and forgot to change the NOAA Upload folder path?

Enjoy,
Paul
I've resolved the issue of the monthly log data showing nothing after the 10th. There was an invalid line in the Nov log file that I had to remove. I think it continued to report on Saratoga as Current since technically the NOAA log file was being uploaded daily, but the data inside it after Nov 10th was corrupt by that bad line of data. I believe this is all fixed now, I have it set to automatically upload daily at 9am I believe.

As for the real-time data issue, it's exactly as you've pointed out; data is uploading in 1-2 minute intervals instead of 5 seconds and Cumulus FTP / weather data doesn't update as fast as it should either, so it triggers the 'Not Current' label in Saratoga. I'm really at a loss with this one.
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

Any other ideas on why my realtime files appear to be processing so slowly?

I checked my Cumulus-MX directory and noticed my realtime.txt is only refreshing once every 1-2 minutes or so. So my guess is that the issue has nothing to do with FTP uploading to the web, but from the datalogger to the box. I notice this pop up a lot in my logs:

2023-11-29 04:02:54.355 Realtime[6]: Warning, a previous cycle is still processing local files. Skipping this interval.

Could this be related to my issue?
20231129-040114.txt
You do not have the required permissions to view the files attached to this post.
User avatar
PaulMy
Posts: 3731
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2+ Cumulus1 / CummulusMX
Operating System: Windows8 / Windows10
Location: Komoka, ON Canada
Contact:

Re: Monthly Log file has stopped updating

Post by PaulMy »

Hi,
I don't think your issue has anything to do with your logger or CMX. That message you quote is exactly your issue. The previous FTP upload has not yet completed when the next one is scheduled. You may need to change your realtime interval to a longer period, i.e. 15 or 20 seconds. You could try the PHP Upload protocol.

Enjoy,
Paul
Davis Vantage Pro2+
C1 www.komokaweather.com/komokaweather-ca
MX www .komokaweather.com/cumulusmxwll/index.htm
MX www.komokaweather.com/cumulusmx/index.php
MX www.komokaweather.com/cumulusmx/index.html

Image
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

Well it definitely wasn't the logger. I changed out Wifilogger V1 and replaced it with V2 tonight, updated firmware to 2.43 and still the same issue.

Tried setting the interval to all sorts of different values and rebooting, still no difference. Has to be either Cumulus or the box it sits on, unable to process the files at a fast enough pace to upload to my template?
freddie
Posts: 2394
Joined: Wed 08 Jun 2011 11:19 am
Weather Station: Davis Vantage Pro 2 + Ecowitt
Operating System: GNU/Linux Ubuntu 22.04 LXC
Location: Alcaston, Shropshire, UK
Contact:

Re: Monthly Log file has stopped updating

Post by freddie »

Have you checked resource usage on the host MX runs on? I'd be surprised if there was an issue as MX is pretty light on resources. It's more likely to be the network speed between MX and where your website is hosted.
Freddie
Image
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

freddie wrote: Wed 29 Nov 2023 2:03 pm Have you checked resource usage on the host MX runs on? I'd be surprised if there was an issue as MX is pretty light on resources. It's more likely to be the network speed between MX and where your website is hosted.
The thing is, the realtime.txt in the MX folder also doesn't refresh regularly - takes up to two minutes or so. That's why I think the problem lies with the box, or from the logger to the box.

Also, I'm 99.99% sure this realtime issue only started to happen last night after I updated Cumulus to the latest version and fixed the Monthly data log issue. It was working just fine before then, we've had regular rainfall and storms in the past week and I never noticed an issue.
User avatar
PaulMy
Posts: 3731
Joined: Sun 28 Sep 2008 11:54 pm
Weather Station: Davis VP2+ Cumulus1 / CummulusMX
Operating System: Windows8 / Windows10
Location: Komoka, ON Canada
Contact:

Re: Monthly Log file has stopped updating

Post by PaulMy »

Hi,
You could try the PHP Upload protocol.
Enjoy,
Paul
Davis Vantage Pro2+
C1 www.komokaweather.com/komokaweather-ca
MX www .komokaweather.com/cumulusmxwll/index.htm
MX www.komokaweather.com/cumulusmx/index.php
MX www.komokaweather.com/cumulusmx/index.html

Image
User avatar
mcrossley
Posts: 12492
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Monthly Log file has stopped updating

Post by mcrossley »

bort483 wrote: Wed 29 Nov 2023 2:06 pm The thing is, the realtime.txt in the MX folder also doesn't refresh regularly - takes up to two minutes or so. That's why I think the problem lies with the box, or from the logger to the box.
Normally the realtime.txt would not be created locally any more. There is no need, MX for some time can upload the contents without the overhead of creating the file.

But, if you want the file locally fine.

The realtime.txt file will not get created though if a real-time period is skipped because the previous cycle has not completed yet. So the file not being created locally is a symptom of the problem not a cause. It sounds like a slow network, or remote server to me.

What upload protocol are you using?

If it is FTP or FTPS then switch on FTP logging and Debug logging. Restart MX and after it has run for say 20 minutes post the latest MX diags file for MX and FTP log file.
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

mcrossley wrote: Wed 29 Nov 2023 4:58 pm
bort483 wrote: Wed 29 Nov 2023 2:06 pm The thing is, the realtime.txt in the MX folder also doesn't refresh regularly - takes up to two minutes or so. That's why I think the problem lies with the box, or from the logger to the box.
Normally the realtime.txt would not be created locally any more. There is no need, MX for some time can upload the contents without the overhead of creating the file.

But, if you want the file locally fine.

The realtime.txt file will not get created though if a real-time period is skipped because the previous cycle has not completed yet. So the file not being created locally is a symptom of the problem not a cause. It sounds like a slow network, or remote server to me.

What upload protocol are you using?

If it is FTP or FTPS then switch on FTP logging and Debug logging. Restart MX and after it has run for say 20 minutes post the latest MX diags file for MX and FTP log file.
I'm using standard FTP. Here's a log with FTP + Debug logging enabled:
20231130-033143.rar
FTP-20231130-033143.rar
You do not have the required permissions to view the files attached to this post.
User avatar
mcrossley
Posts: 12492
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Monthly Log file has stopped updating

Post by mcrossley »

That FTP server is *super* slow!

For example, it takes 8 seconds just to return a directory listing. That isn't waiting for the response, it is returning the file list from start to finish.

You need to get that resolved.
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

Thanks Mark, will take a look at it later today.

And Paul, if all else fails I'll give PHP a shot.
Cheers.
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

So @mcrossley, just to clarify, does it look like the issue lies between me and my server (which is hosted in Singapore)? I've never had a problem with it in the 3 years I've been running it. It's quite possible this is what is causing it (my ISP uses this cable), but I'm in eastern Australia so I'm not sure: https://www.gslstatus.com/ (INDIGO West (Perth to Singapore) Cable Offline)

I don't know what else I can try on my end so I'll just have to wait and see if it clears up.
User avatar
mcrossley
Posts: 12492
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Monthly Log file has stopped updating

Post by mcrossley »

Looking at the log some more, it appears the FTP component is performing a full directory listing to determine if a file exists, normally it would just do a SIZE command on the file in question. As your root folder has 288 items in it and the server is a bit slow, each file is taking around 9 seconds to upload.

This appears to be a bug in the FTP component, another user of the component has reported a similar issue that results in full directory listings being used on some servers. The author is reporting as fixed, but the new version of the component is not yet available. It was fixed four days ago, with the new release "available in a few days"....
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

Damn that makes sense, since I did notice it only started to happen after I updated Cumulus the other night to the latest version.

Hopefully a new version with the fix is released soon.
bort483
Posts: 31
Joined: Tue 19 Sep 2017 9:16 am
Weather Station: Davis Vantage Pro2
Operating System: Ubuntu

Re: Monthly Log file has stopped updating

Post by bort483 »

I've actually managed to fix it. Changed to a web host with servers located here in Aus and not Singapore, using an extended Cyber Monday deal. And all is running well. 8-) No idea why this only started to happen last week after 2 years of being flawless.

Thanks everyone for the help with logs etc.
Post Reply