Welcome to the Cumulus Support forum.

Latest Cumulus MX release v3.2.4 build 3060 - 10 December 2019.
    Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014

    See the Wiki for downloading either version.

    Extra Webfiles Missing

    Discussion of version 3 of Cumulus, which runs on Windows, Linux, and OS X. All Cumulus MX queries in here, please.

    Moderator: mcrossley

    Post Reply
    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Extra Webfiles Missing

    Post by Phil23 » Mon 02 Dec 2019 9:44 pm

    Hi All,

    Just looked at my page http://weather.inverellit.com/today-yesterday.htm & found it hadn't updated since 6:00pm on 02/12/2019.

    Looked in the Extra Webfile settings & saw it's entry was missing.
    Actually all entries except the first 10 were gone.

    Checked the Diags & the only thing that stood out was repetitions of this...

    Code: Select all

    2019-12-02 17:10:01.977 Extra web file #8 [\CumulusMX\Cumuluswebtags.txt] not found!
    
    Don't know what triggered it other than the fact that I do have task scheduler shutting down & restarting MX at both 6:00AM & PM.

    In relation to this, can I suggest that Cumulus.ini be included in the files saved in the various backup directories in a future update.

    Cheers

    Phil.
    Image

    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Re: Extra Webfiles Missing

    Post by Phil23 » Tue 03 Dec 2019 1:58 am

    Just been pulling entries from an older ini from last upgrade.

    Only unusual thing I notice is that there is a block of other data after the first 10 entries & then the others continue.
    As in the entries are not in a continuious block, other entries are present after entry 9 then it continues with Entry 10 etc.

    Code: Select all

    [FTP site]
    Host=ftp.inverellit.com
    Port=21
    Username=
    Password=
    Directory=public_html/weather
    AutoUpdate=1
    ActiveFTP=1
    EnableRealtime=1
    RealtimeFTPEnabled=1
    RealtimeTxtFTP=1
    RealtimeInterval=5000
    UpdateInterval=10
    IncludeSTD=1
    IncludeSTDImages=1
    FTPRename=1
    DeleteBeforeUpload=0
    ResizeGraphs=0
    GraphHeight=300
    GraphWidth=750
    ImageFolder=
    ImageCopyRealtime=0
    ExtraLocal0=\CumulusMX\web\gauges-ssT.htm
    ExtraRemote0=/public_html/weather/gauges-ss.htm
    ExtraProcess0=1
    ExtraBinary0=0
    ExtraRealtime0=0
    ExtraFTP0=1
    ExtraLocal1=\CumulusMX\web\realtimeT.xml
    ExtraRemote1=/public_html/weather/realtime.xml
    ExtraProcess1=1
    ExtraBinary1=0
    ExtraRealtime1=1
    ExtraFTP1=1
    ExtraLocal2=\CumulusMX\web\sunbirdT.txt
    ExtraRemote2=/public_html/weather/sunbird.txt
    ExtraProcess2=1
    ExtraBinary2=0
    ExtraRealtime2=1
    ExtraFTP2=1
    ExtraLocal3=\CumulusMX\web\index2t.htm
    ExtraRemote3=/public_html/weather/index2.htm
    ExtraProcess3=1
    ExtraBinary3=0
    ExtraRealtime3=0
    ExtraFTP3=1
    ExtraLocal4=\CumulusMX\web\davcon24T.txt
    ExtraRemote4=/public_html/weather/davcon24.txt
    ExtraProcess4=1
    ExtraBinary4=0
    ExtraRealtime4=1
    ExtraFTP4=1
    ExtraLocal5=\CumulusMX\web\davconfcstT.txt_MX
    ExtraRemote5=/public_html/weather/davconfcst.php
    ExtraProcess5=1
    ExtraBinary5=0
    ExtraRealtime5=0
    ExtraFTP5=1
    ExtraLocal6=\CumulusMX\web\davcon24T.txt
    ExtraRemote6=/public_html/weather2/davcon24.txt
    ExtraProcess6=1
    ExtraBinary6=0
    ExtraRealtime6=0
    ExtraFTP6=1
    ExtraLocal7=\CumulusMX\web\davconfcstT.txt_MX
    ExtraRemote7=/public_html/weather2/davconfcst.php
    ExtraProcess7=1
    ExtraBinary7=0
    ExtraRealtime7=0
    ExtraFTP7=1
    ExtraLocal8=\CumulusMX\Cumuluswebtags.txt
    ExtraRemote8=\CumulusMX\Cumuluswebtags.php
    ExtraProcess8=1
    ExtraBinary8=0
    ExtraRealtime8=0
    ExtraFTP8=0
    ExtraLocal9=\CumulusMX\web\graphsT.htm
    ExtraRemote9=/public_html/weather/graphs.htm
    ExtraProcess9=1
    ExtraBinary9=0
    ExtraRealtime9=0
    ExtraFTP9=1
    ExternalProgram=
    RealtimeProgram=
    ExternalParams=
    RealtimeParams=
    DailyProgram=C:\CumulusMX\Scripts\DayEnd.bat
    DailyParams=
    IncludeSolarChart=0
    IncludeUVChart=0
    IncludeSunshineChart=0
    UTF8encode=1
    ExtraUTF0=0
    ExtraUTF1=0
    ExtraUTF2=0
    ExtraUTF3=0
    ExtraUTF4=0
    ExtraUTF5=0
    ExtraUTF6=0
    ExtraUTF7=0
    ExtraUTF8=0
    ExtraUTF9=0
    Sslftp=0
    FTPlogging=0
    IncludeGraphDataFiles=1
    ExtraLocal10=\CumulusMX\web\kindleT.htm
    ExtraRemote10=/public_html/weather/kindle.htm
    ExtraProcess10=1
    ExtraBinary10=0
    ExtraRealtime10=1
    ExtraFTP10=1
    ExtraUTF10=0
    ExtraLocal11=\CumulusMX\web\datasummaryT.htm
    ExtraRemote11=/public_html/weather/datasummary.php
    ExtraProcess11=1
    ExtraBinary11=0
    ExtraRealtime11=0
    ExtraFTP11=1
    ExtraUTF11=1
    ExtraLocal12=\CumulusMX\data\dayfile.txt
    ExtraRemote12=/public_html/weather/dayfile.txt
    ExtraProcess12=0
    ExtraBinary12=0
    ExtraRealtime12=0
    ExtraFTP12=1
    ExtraUTF12=0
    ExtraLocal13=\CumulusMX\web\today-yesterdayT.htm
    ExtraRemote13=/public_html/weather/today-yesterday.htm
    ExtraProcess13=1
    ExtraBinary13=0
    ExtraRealtime13=0
    ExtraFTP13=1
    ExtraUTF13=0
    ExtraLocal14=\CumulusMX\web\gT.htm
    ExtraRemote14=/public_html/weather/g.htm
    ExtraProcess14=1
    ExtraBinary14=0
    ExtraRealtime14=0
    ExtraFTP14=1
    ExtraUTF14=0
    ExtraLocal15=\CumulusMX\web\betel_monthlyrecord_mxT.html
    ExtraRemote15=/public_html/weather/betel_monthlyrecord_mx.html
    ExtraProcess15=1
    ExtraBinary15=0
    ExtraRealtime15=0
    ExtraFTP15=1
    ExtraUTF15=0
    ExtraLocal16=\CumulusMX\utils\systeminfoTable.txt
    ExtraRemote16=/public_html/weather/sysinfoTable.txt
    ExtraProcess16=1
    ExtraBinary16=0
    ExtraRealtime16=1
    ExtraFTP16=1
    ExtraUTF16=1
    ExtraLocal17=\CumulusMX\web\gaugesT.htm
    ExtraRemote17=/public_html/weather/gauges.htm
    ExtraProcess17=0
    ExtraBinary17=0
    ExtraRealtime17=0
    ExtraFTP17=0
    ExtraUTF17=0
    ExtraLocal18=
    ExtraRemote18=
    ExtraProcess18=0
    ExtraBinary18=0
    ExtraRealtime18=0
    ExtraFTP18=0
    ExtraUTF18=0
    ExtraLocal19=
    ExtraRemote19=
    ExtraProcess19=0
    ExtraBinary19=0
    ExtraRealtime19=0
    ExtraFTP19=0
    ExtraUTF19=0
    ExtraLocal20=
    ExtraRemote20=
    ExtraProcess20=0
    ExtraBinary20=0
    ExtraRealtime20=0
    ExtraFTP20=1
    ExtraUTF20=0
    
    Phil.
    Image

    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Re: Extra Webfiles Missing

    Post by Phil23 » Tue 03 Dec 2019 3:02 am

    Further investigations has found I also lost the Realtimegauges.txt setting & all my NOAA Temperatures & Rainfall values were missing.

    The process of shutting down & restarting MX twice a day has been going on for many months, so can't see it having any relevance to this issue.
    Image

    User avatar
    mcrossley
    Posts: 5807
    Joined: Thu 07 Jan 2010 9:44 pm
    Weather Station: Davis VP2
    Operating System: Stretch Lite rPi
    Location: Wilmslow, Cheshire, UK
    Contact:

    Re: Extra Webfiles Missing

    Post by mcrossley » Tue 03 Dec 2019 11:36 am

    Can't help with the missing entires, but the out of order is just because new options have been added to Extra Files feature. They get added after any existing ones.

    You *can* clean it up and make it sequential (purely for ease of human reading) by renaming Cumulus.ini to something like Cumulsu.ini.bak whilst MX is running, and then either making a minor change to a config item (and reversing it), or shutting down MX. It will then write out a new Cumulus.ini in a logical order.

    Adding the ini file to backup is a good idea.

    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Re: Extra Webfiles Missing

    Post by Phil23 » Tue 03 Dec 2019 8:07 pm

    mcrossley wrote:
    Tue 03 Dec 2019 11:36 am
    You *can* clean it up and make it sequential (purely for ease of human reading) by renaming Cumulus.ini to something like Cumulsu.ini.bak whilst MX is running, and then either making a minor change to a config item (and reversing it), or shutting down MX. It will then write out a new Cumulus.ini in a logical order.
    Thanks Mark,

    That will be a whole lot easier than the manual editing I was thinking of. Maybe a worthwhile addition to the Wiki.

    Incidentally I tried to register on the Wiki some months back & kept failing.

    Hoping you'll add the ini to the files stored in the backup folder for others in the future.

    Cheers.
    Image

    User avatar
    mcrossley
    Posts: 5807
    Joined: Thu 07 Jan 2010 9:44 pm
    Weather Station: Davis VP2
    Operating System: Stretch Lite rPi
    Location: Wilmslow, Cheshire, UK
    Contact:

    Re: Extra Webfiles Missing

    Post by mcrossley » Wed 04 Dec 2019 10:42 am

    One caveat with that ini file process - you will lose any settings that are "ini file only" - i.e. are only ever read by Cumulus and never written.

    Options such as CreateWxnowTx, SpecialLog etc, etc

    So be careful and check your old file against the new to see if anything has "gone" that you need. These options *must* be added back in to the new with Cumulus stopped.

    I guess a safer option would be to get Cumulus to generate the new file - rename that, rename the original back, and cut'n'paste just the FTP section back into the original file.

    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Re: Extra Webfiles Missing

    Post by Phil23 » Thu 05 Dec 2019 5:20 am

    mcrossley wrote:
    Wed 04 Dec 2019 10:42 am
    I guess a safer option would be to get Cumulus to generate the new file - rename that, rename the original back, and cut'n'paste just the FTP section back into the original file.
    I'm Assuming it would be a fairly major re-write to have MX Rename the ini file to bak & then write a the new ini file each time in a logical order?
    Image

    User avatar
    mcrossley
    Posts: 5807
    Joined: Thu 07 Jan 2010 9:44 pm
    Weather Station: Davis VP2
    Operating System: Stretch Lite rPi
    Location: Wilmslow, Cheshire, UK
    Contact:

    Re: Extra Webfiles Missing

    Post by mcrossley » Thu 05 Dec 2019 9:10 am

    Yes, mainly because there are possibly lots of settings in there that are read-only.

    User avatar
    HansR
    Posts: 453
    Joined: Sat 20 Oct 2012 6:53 am
    Weather Station: Davis Vantage Pro 2+
    Operating System: Raspbian GNU/Linux 10 (Buster)
    Location: Wagenborgen (NL)
    Contact:

    Re: Extra Webfiles Missing

    Post by HansR » Thu 05 Dec 2019 10:07 am

    @mcrossley: You might want to modify the ini-library such, that if a parameter does not exist, it will be created when read by the program with a certain default.
    I did modify the library as such and therefore, I am not interested in what is being done by the user. Any deletion will be corrected with an addition of that parameter with a certain default. Especially for read-only parameters (why have these in the ini in the first place?) that would be an easy solution. In other cases the user may have to correct the default again.
    https://meteo-wagenborgen.nl / Cumulus (3.1.2-b3055)
    Davis Vantage Pro 2+ / Raspberry Pi 3 Model B Plus Rev 1.3
    Raspbian GNU/Linux 10 (buster) / Linux 4.19.75-v7+ armv7l
    Mono JIT compiler version 6.4.0.198 (tarball Tue Sep 24 01:45:47 UTC 2019)

    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Re: Extra Webfiles Missing

    Post by Phil23 » Thu 05 Dec 2019 7:50 pm

    Has Happened again, but I got a completely blank default ini file this time.
    Not even the station is set.

    Stations last update was 12:00AM, just before a scheduled midnight shutdown.

    The shutdown is run by the task scheduler which runs a compiled AutoHotKey script.
    All that script does is put focus on the MX window & then send it Ctrl-C,
    so I'm not doing anything dopey like simply killing a task.

    Add to that this has been running for over 12 months with no issues.

    Fortunately I'd upgraded to Version 3059 yesterday arvo, so could just retrieve the ini file from a 6:00pm backup.
    (Thanks Mark).

    Getting the impression it's come about as of a certain update number, but it's still only two random occurrences.

    Phil
    Image

    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Re: Extra Webfiles Missing

    Post by Phil23 » Thu 05 Dec 2019 8:23 pm

    Ok,

    Been looking at the diags logs & looks like my host had a FTP issue just before the 6:00 PM shutdown.
    The 6:00pm shutdown wrote the ini correctly,
    but the 12:00am one wrote it at zero bytes.

    Seems that MX may have been very busy trying to re-establish connections.

    I'm assuming at the 6:00 am restart MX wrote the default Ini file having had nothing to read from the blank one written at mindnight.

    Just detail on the restarts; I shut MX down routinely to allow weatherlink to download updates & then exit.

    MX is then restarted one minute later; 12:01am, 6:01am etc.

    But again, that's been working for months....


    3 diag files attached.

    Cheers.
    You do not have the required permissions to view the files attached to this post.
    Image

    Phil23
    Posts: 346
    Joined: Sat 16 Jul 2016 11:59 pm
    Weather Station: Davis Vantage Pro2 Plus
    Operating System: Win7 Pro
    Location: Australia

    Re: Extra Webfiles Missing

    Post by Phil23 » Thu 05 Dec 2019 9:37 pm

    Don't think this is totally related;

    After upgrading to 3059 @ 3:55pm everything was working fine.
    Then @ 5:49pm realtime uploads started failing.

    But still, the 6:00pm shutdown wrote the ini correctly.

    Code: Select all

    2019-12-05 17:40:00.908 Writing log entry for 5/12/2019 5:40:00 PM
    2019-12-05 17:40:00.922 Written log entry for 5/12/2019 5:40:00 PM
    2019-12-05 17:40:00.923 Writing today.ini, LastUpdateTime = 5/12/2019 5:40:00 PM raindaystart = 215.2 rain counter = 215.2
    2019-12-05 17:40:01.487 Extra web file #8 [\CumulusMX\Cumuluswebtags.txt] not found!
    2019-12-05 17:42:25.666 Reading reception stats
    2019-12-05 17:49:03.489 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:08.497 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:13.498 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:18.501 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:23.512 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:28.513 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:28.598 Error connecting ftp - Timed out trying to connect!
    2019-12-05 17:49:33.513 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:38.513 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:43.514 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:43.599 Error uploading realtime.txt to public_html/weather/realtime.txt : Timed out trying to connect!
    2019-12-05 17:49:48.516 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:53.516 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:58.517 Warning, previous realtime ftp still in progress, skipping this period.
    2019-12-05 17:49:58.618 Error uploading web\realtimegauges.txt to public_html/weather/realtimegauges.txt : Timed out trying to connect!
    2019-12-05 17:50:00.395 Writing log entry for 5/12/2019 5:50:00 PM
    2019-12-05 17:50:00.484 Written log entry for 5/12/2019 5:50:00 PM
    
    FTP errors on realtime files continued till about 6:18pm then came good,
    But then reappeared at 21:38 for a couple of minutes.

    Then there were not apparent errors for the next few hours up till the midnight shutdown that clobbered the ini file.

    Phil
    Image

    User avatar
    mcrossley
    Posts: 5807
    Joined: Thu 07 Jan 2010 9:44 pm
    Weather Station: Davis VP2
    Operating System: Stretch Lite rPi
    Location: Wilmslow, Cheshire, UK
    Contact:

    Re: Extra Webfiles Missing

    Post by mcrossley » Thu 05 Dec 2019 10:47 pm

    Thanks for the files, I'll take a look over the next day or so. CMX must be starting the process of writing the ini but terminating before it completes for some reason.

    Post Reply