Welcome to the Cumulus Support forum.

Latest Cumulus MX release 3.9.6 (build 3101) - 20 January 2021 (please see announcement regarding releases since 3.5.0)
Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of the NOAA report and Snow Index drop-down menus to 2030)

Use this Wiki link to Download the Software (Cumulus MX / Cumulus 1 and other related items).

realtime.txt update periodically fails

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

Post Reply
pkiener
Posts: 26
Joined: Mon 24 Feb 2014 10:21 pm
Weather Station: Davis Vantage VUE
Operating System: Windows 10
Location: United States
Contact:

realtime.txt update periodically fails

Post by pkiener »

Cumulus Version 3.6.6 Build 3082
OS: Unix 4.19.118.7 on Rpi 3
Problem: 2nd personal weather web site (powered by Meteotemplate, ver 18.0) gives warning that data is offline. Program utilizes realtime.txt from Cumulus for data source.
Findings: (1) data is being correctly displayed and updated on CumulusMX personal web site; data is correctly generated; (2) realtime.txt (264 byes) and realtimegauges.txt (1,566 bytes) last uploaded to personal web site (hosted by 3rd party) at 4:35:09 AM this morning. Current time 9:20 AM; (3) realtimegauges.txt (1,566 bytes) updated 4:35:09 AM on rasp-pi; (4) realtime.txt (0 bytes) updated 9:20:17 AM on rasp-pi.

From the above findings, it appears that realtime.txt is not being correctly generated with data. Realtime.txt and realtimegaugs.txt thus not being uploaded to personal web site. (NOTE: up until the last upload to web site, data derived from realtime.txt was being displayed correctly)

I note that data used for my personal CumulusMX web site is generated using the template indexT.htm (index.htm is then uploaded to my web site); similar process takes place with realtimegaugesT.txt (realtimegauges.txt being uploaded to web site).

Stopping CumulusMX on rasp-pi and restarting CumulusMX allows realtime.txt to be properly updated again. The problem with realtime.txt has been going on for the past few versions, over a few weeks now. With each new version posted, I have been waiting to see if it will correct my problem, but as of yet, it has not. The problem happens after CumulusMX has been running for a few days (I have not noticed a pattern as to when realtime.txt stops working correctly).

Any help/suggestions would be appreciated! Thank you in advance

Paul
(personal CumulusMX web site: http://www.paulkiener.com/Cumulus/index.htm)
(Meteotemplate web site: http://weather.paulkiener.com)

ExperiMentor
Posts: 213
Joined: Tue 24 Nov 2015 11:30 pm
Weather Station: Fine Offset & Davis Vantage Vue
Operating System: Windows 10; Raspbian Buster
Location: Switzerland

Re: realtime.txt update periodically fails

Post by ExperiMentor »

pkiener wrote:
Thu 04 Jun 2020 2:10 pm
The problem happens after CumulusMX has been running for a few days (I have not noticed a pattern as to when realtime.txt stops working correctly).
This will not fix the root cause or explain why you have experienced this problem when others apparently have not (at least, no-one has reported it).

But it does exemplify why it is good pratice to set your Raspberry Pi do do a daily reboot, in the early hours. Then "has been running for a few days" is never reached.

To make Cumulus run each time the Pi is rebooted (and force reboot in the early hours each day)
On the Pi, type:

Code: Select all

sudo crontab -e
On first run select the text editor you prefer (defaults to #1, nano, the easiest)
Then add the following lines at the end of the file:

Code: Select all

# Start Cumulus as background task 30s after reboot (delay to allow WiFi to startup)
@reboot (sleep 30;cd /home/pi/CumulusMX;sudo mono CumulusMX.exe) &

# Reboot each day at 0253
53 02 * * * sudo reboot
To stop the Pi and restart it without CumulusMX running (eg you need to do that if upgrading the CumulusMX version)
type

Code: Select all

sudo crontab -e
edit to put a # at the start of the line "@reboot..."
Ctrl-X to save the change to crontab and reboot using

Code: Select all

sudo reboot
When your pi restarts, CumulusMX will no longer be running. You can then do your version upgrade or other task.

To revert to normal auto-running of CumulusMX, go through the same again, but this time edit crontab to remove the # from the start of the line "@reboot...". Save changes and reboot - CumulusMX will be running.

water01
Posts: 2053
Joined: Sat 13 Aug 2011 9:33 am
Weather Station: WH1081
Operating System: Windows 10 64bit
Location: Burnham-on-Sea
Contact:

Re: realtime.txt update periodically fails

Post by water01 »

If you are using SFTP there is a problem that I have reported to Mark Crossley and he has investigated it. Mine was caused by a MySql update failing which in turn stopped realtime updates.

Until Mark is able to release a new version the only way out of the problem is to close CumulusMX and restart it and then keep an eye on the realtime updates.
David
Image

pkiener
Posts: 26
Joined: Mon 24 Feb 2014 10:21 pm
Weather Station: Davis Vantage VUE
Operating System: Windows 10
Location: United States
Contact:

Re: realtime.txt update periodically fails

Post by pkiener »

Thank you, David for your reply. I knew there were a few other reports of SFTP failure and realtime.txt, but was not sure that my problem was the same or a different view of a deeper problem. Hope I've been able to shed a little more to help Mark narrow down the problem.

Paul

pkiener
Posts: 26
Joined: Mon 24 Feb 2014 10:21 pm
Weather Station: Davis Vantage VUE
Operating System: Windows 10
Location: United States
Contact:

Re: realtime.txt update periodically fails

Post by pkiener »

I have upgraded CumulusMX to ver 3.6.7, Build 3083 (8 June 2020). I am still getting occasional "Offline" notification, indicating that data via realtime.txt is not being properly uploaded to my web site. Same problem as noted in my previous post in this thread.

I did have logging enabled and the relevant lines, at the time of the last, known good upload, are the following:

2020-06-15 06:45:59.568 Realtime[93]: Uploading - realtime.txt
2020-06-15 06:45:59.568 SFTP: Starting Upload of realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:48:59.568 Error encountered during Realtime file update.
2020-06-15 06:48:59.568 Sharing violation on path /home/pi/CumulusMX/realtime.txt
2020-06-15 06:48:59.569 Realtime[94]: Error, previous upload has been running for too long, attempting to reconnect.
2020-06-15 06:49:17.646 Realtime[94]: Uploading - realtime.txt
2020-06-15 06:49:17.646 SFTP: Starting Upload of realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:49:18.004 SFTP: Renaming /home/kiener/public_html/Cumulus/realtime.txttmp to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:49:18.364 SFTP: Completed uploading realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:49:18.374 Realtime[94]: Uploading - realtimegauges.txt
2020-06-15 06:49:18.375 SFTP: Starting Upload of web/realtimegauges.txt to /home/kiener/public_html/Cumulus/realtimegauges.txt
2020-06-15 06:49:18.863 SFTP: Renaming /home/kiener/public_html/Cumulus/realtimegauges.txttmp to /home/kiener/public_html/Cumulus/realtimegauges.txt
2020-06-15 06:49:19.221 SFTP: Completed uploading web/realtimegauges.txt to /home/kiener/public_html/Cumulus/realtimegauges.txt
2020-06-15 06:50:00.606 Writing log entry for 6/15/2020 6:50:00 AM

I have attached the complete log as I have it to this message.

Any and all help would be greatly appreciated. Thank you!

Paul
You do not have the required permissions to view the files attached to this post.

User avatar
beteljuice
Posts: 3100
Joined: Tue 09 Dec 2008 1:37 pm
Weather Station: None !
Operating System: W10 - Threadripper 16core, etc
Location: Dudley, West Midlands, UK

Re: realtime.txt update periodically fails

Post by beteljuice »

2020-06-15 06:42:59.563 Realtime[92]: Uploading - realtime.txt
2020-06-15 06:42:59.564 SFTP: Starting Upload of realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:43:00.051 SFTP: Renaming /home/kiener/public_html/Cumulus/realtime.txttmp to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:43:00.410 SFTP: Completed uploading realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:43:00.420 Realtime[92]: Uploading - realtimegauges.txt
2020-06-15 06:43:00.421 SFTP: Starting Upload of web/realtimegauges.txt to /home/kiener/public_html/Cumulus/realtimegauges.txt
2020-06-15 06:43:00.896 SFTP: Renaming /home/kiener/public_html/Cumulus/realtimegauges.txttmp to /home/kiener/public_html/Cumulus/realtimegauges.txt
2020-06-15 06:43:01.268 SFTP: Completed uploading web/realtimegauges.txt to /home/kiener/public_html/Cumulus/realtimegauges.txt
2020-06-15 06:45:00.336 Updating CWOP
2020-06-15 06:45:59.568 Realtime[93]: Uploading - realtime.txt
2020-06-15 06:45:59.568 SFTP: Starting Upload of realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:48:59.568 Error encountered during Realtime file update.
2020-06-15 06:48:59.568 Sharing violation on path /home/pi/CumulusMX/realtime.txt
2020-06-15 06:48:59.569 Realtime[94]: Error, previous upload has been running for too long, attempting to reconnect.

2020-06-15 06:49:17.646 Realtime[94]: Uploading - realtime.txt
2020-06-15 06:49:17.646 SFTP: Starting Upload of realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:49:18.004 SFTP: Renaming /home/kiener/public_html/Cumulus/realtime.txttmp to /home/kiener/public_html/Cumulus/realtime.txt
2020-06-15 06:49:18.364 SFTP: Completed uploading realtime.txt to /home/kiener/public_html/Cumulus/realtime.txt

Means nothing to me :lol:
Image
......................Imagine, what you will KNOW tomorrow !

water01
Posts: 2053
Joined: Sat 13 Aug 2011 9:33 am
Weather Station: WH1081
Operating System: Windows 10 64bit
Location: Burnham-on-Sea
Contact:

Re: realtime.txt update periodically fails

Post by water01 »

What is your realtime interval set to as it would appear that the previous upload has not completed before the next one starts, hence the sharing violation. At least CumulusMX now recovers, which prior to 3083 it did not.

Suggest you lengthen your realtime upload interval and see if that solves the problem.
David
Image

Vegit8
Posts: 84
Joined: Fri 27 Sep 2013 2:11 pm
Weather Station: Davis Vantage Pro2
Operating System: Win 10
Location: Somerset
Contact:

Re: realtime.txt update periodically fails

Post by Vegit8 »

Hi
I am not so sure lengthening the repeat time has that much of an effect.
If you push it out too far you begin to lose the sense of the meaning of 'realtime'

I manage two separate systems (mine and a friends) the latter via remote connections.

We have almost identical set ups, with the exception I have a VP2 and he has a WLL and I am running MX on a windows platform, he is running on Ubuntu. He is running 3083, I am some way behind on 3072 because I have not needed to upgrade yet.

I never have the problem with uploads of the realtime.txt file, he has almost constant problems.
My realtime frequency is 20 seconds, his is 45.

Whenever the realtime update stops on his machine it is because the realtimetextT.txt file is locked by ROOT as the Owner (read only)

What confuses me is that all the other files continue to upload via SFTP, I'm guessing the problem with the realtime file is that it is processed each cycle..

I have my suspicions that the problem is exacerbated by two things OS and upload speed.
My friend has problems when he tries to upload other files (to YouTube) - he is only on ADSL - so there appears to be a bandwidth problem.
Consequently the connection times out, leaving the file locked by the OS.

On the next round of uploads MX is unable to unlock the file (code /OS problem?) - and the cycle then continues until MX is stopped and restarted.
A few times I have had to 'unlock' the file manually as it otherwise retains the lock and MX restarts without fixing the problem.

pkiener
Posts: 26
Joined: Mon 24 Feb 2014 10:21 pm
Weather Station: Davis Vantage VUE
Operating System: Windows 10
Location: United States
Contact:

Re: realtime.txt update periodically fails

Post by pkiener »

Thank you, Vegit8 for your reply. My Reaaltime interval is set at 180 seconds (3 minutes). That's considerably longer than what either you or your friend have.
The next time I have a problem, I'll take a look at the status of the realtimetextT.txt file.
Again, thanks for your reply.
Paul

pkiener
Posts: 26
Joined: Mon 24 Feb 2014 10:21 pm
Weather Station: Davis Vantage VUE
Operating System: Windows 10
Location: United States
Contact:

Re: realtime.txt update periodically fails

Post by pkiener »

Vegit8, just curious: where is the file realtimetextT.txt located? I tried looking for it and could not find it. Thanks.
Paul

Vegit8
Posts: 84
Joined: Fri 27 Sep 2013 2:11 pm
Weather Station: Davis Vantage Pro2
Operating System: Win 10
Location: Somerset
Contact:

Re: realtime.txt update periodically fails

Post by Vegit8 »

Hi
Its in /Web

Mapantz
Posts: 902
Joined: Sat 17 Dec 2011 11:55 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 10 x64
Location: Dorset - UK
Contact:

Re: realtime.txt update periodically fails

Post by Mapantz »

realtimetextT.txt doesn't exist unless you make your own version.

realtime.txt sits in the root directory of cmx. ie C:\CumulusMX\
Image

User avatar
mcrossley
Posts: 7566
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Buster Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: realtime.txt update periodically fails

Post by mcrossley »

The realtime.txt file in use issue with SFTP will be fixed (!?) in the next release.

Post Reply