Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

CMX crash and realtime retain failure

From build 3044 the development baton passed to Mark Crossley. Mark has been responsible for all the Builds since. He has made the code available on GitHub. It is Mark's hope that others will join in this development, but at the very least he welcomes your ideas for future developments (see Cumulus MX Development suggestions).

Moderator: mcrossley

User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

No. Intermittently it does work (ftps) most of the time it doesn't. Plain ftp works. Sftp says open ssh not permitted. I put it aside for the moment. There are more interesting things in the world :(
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

And before I forget: of course WinSCP works flawlessly on all FTP techniques possible.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
ConligWX
Posts: 1570
Joined: Mon 19 May 2014 10:45 pm
Weather Station: Davis vPro2+ w/DFARS + AirLink
Operating System: Ubuntu 22.04 LTS
Location: Bangor, NI
Contact:

Re: CMX crash and realtime retain failure

Post by ConligWX »

HansR wrote: Thu 08 Apr 2021 5:26 pm There are more interesting things in the world :(

Indeed. I want to know which covid vaccines will turn us all into mutants :lol:
Regards Simon

https://www.conligwx.org - @conligwx
Davis Vantage Pro2 Plus with Daytime FARS • WeatherLink Live • Davis AirLink • PurpleAir •

Image
User avatar
mcrossley
Posts: 12685
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: CMX crash and realtime retain failure

Post by mcrossley »

HansR wrote: Thu 08 Apr 2021 5:26 pm No. Intermittently it does work (ftps) most of the time it doesn't. Plain ftp works. Sftp says open ssh not permitted.
The later versions of the SSH component do support many more key file types, but then we hit some other bugs, and there is no support for running under Mono. :(
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

Ok, so what is the instruction for keygeneration fot the current version used?
I can't find any instruction in the wiki/forum. (Keywords not discriminating enough)
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
mcrossley
Posts: 12685
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: CMX crash and realtime retain failure

Post by mcrossley »

I cannot find the steps I did, but I'm pretty sure I just used OpenSSL on my NAS box to generate an RSA 2048 bit key pair. Extract and copy the public key into the server authorized_keys file, then added the local path to the private key into CMX.
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

I did similar using ssh-keygen with default. Stored the private key directly in the CMX dir and the public in the server part. So far so good. I can access with MobaXterm or other client using a keypair (I use another, similar created keypair) . While running CMX it does not accept openssh. Could be the server part.

To finish off my contribution in this field, attached is the log for my SSH run (private key local in CMX dir, public key on the server according to their specs).

I'll ask the hoster on monday if they intend it only to be for terminals ... :?
You do not have the required permissions to view the files attached to this post.
Hans

https://meteo-wagenborgen.nl
CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
User avatar
mcrossley
Posts: 12685
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: CMX crash and realtime retain failure

Post by mcrossley »

I think that error is saying it does not like the the OpenSSH format private key, it needs to be converted to RSA format...

I found this command (untested)...

Code: Select all

ssh-keygen -p -N "" -m pem -f /path/to/key
That sets a blank password - "" - and converts the file "in place" so will overwrite your existing file. Make a copy first!
User avatar
HansR
Posts: 5870
Joined: Sat 20 Oct 2012 6:53 am
Weather Station: GW1100 (WS80/WH40)
Operating System: Raspberry OS/Bullseye
Location: Wagenborgen (NL)
Contact:

Re: CMX crash and realtime retain failure

Post by HansR »

Yes, that helps. I get connected now. Thnx.
But it's not all right. There are apparently still some paths to set which seems strange to me because it looks correct for realtime.txt. After the first error it disposes the SFTP object and it's back to square one. Tomorrow the rest.
    Schermafbeelding 2021-04-09 225436.jpg
    You do not have the required permissions to view the files attached to this post.
    Hans

    https://meteo-wagenborgen.nl
    CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
    User avatar
    HansR
    Posts: 5870
    Joined: Sat 20 Oct 2012 6:53 am
    Weather Station: GW1100 (WS80/WH40)
    Operating System: Raspberry OS/Bullseye
    Location: Wagenborgen (NL)
    Contact:

    Re: CMX crash and realtime retain failure

    Post by HansR »

    I looked into the failing SFTP uploads and I think it may have to do with ownership of the whole directory structure of my website.

    The owner is not me (my userid or whatever) but the owner is ftp and therefore(?) the SFTP process cannot access the files and issues the error message. I am not sure, but I have no other explanation. I can't use SFTP from WinSCP either because I cannot give a key, only un/pw.

    I will have to contact the provider.
    Hans

    https://meteo-wagenborgen.nl
    CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
    User avatar
    mcrossley
    Posts: 12685
    Joined: Thu 07 Jan 2010 9:44 pm
    Weather Station: Davis VP2/WLL
    Operating System: Bullseye Lite rPi
    Location: Wilmslow, Cheshire, UK
    Contact:

    Re: CMX crash and realtime retain failure

    Post by mcrossley »

    HansR wrote: Sat 10 Apr 2021 7:47 am I can't use SFTP from WinSCP either because I cannot give a key, only un/pw.
    ?? You set the key file under the connection Advanced Site Settings | SSH | Authentication. It also has tools for generating a key pair (it uses PuTTY KeyGen) and to upload your public key to the server.
    User avatar
    HansR
    Posts: 5870
    Joined: Sat 20 Oct 2012 6:53 am
    Weather Station: GW1100 (WS80/WH40)
    Operating System: Raspberry OS/Bullseye
    Location: Wagenborgen (NL)
    Contact:

    Re: CMX crash and realtime retain failure

    Post by HansR »

    OK. I did expect the key entry when modifying the protocol to SSH on the same level as un/pw. Not so, so thanks.
    In the process I also found that the path required two more levels to the root of the server machine. Those had not been visible before without WinSCP on SSH.
    So, everything is working fine now apart from the FTPS protocol.
    I also have to renew the key every year (provider demand), so I prefer FTPS assuming it's working: much easier.
    Hans

    https://meteo-wagenborgen.nl
    CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
    User avatar
    HansR
    Posts: 5870
    Joined: Sat 20 Oct 2012 6:53 am
    Weather Station: GW1100 (WS80/WH40)
    Operating System: Raspberry OS/Bullseye
    Location: Wagenborgen (NL)
    Contact:

    Re: CMX crash and realtime retain failure

    Post by HansR »

    OK, now that I have SFTP working I went back to FTPS and looked at that one.

    I noticed that when setting the protocol for my hoster I have to use port 21 for FTPS (yes I know, it is not common, but still it is true). When I set 21 and save, I go elsewhere to set the path in the extra web files and then looking at the log tail it does connect but it fails again for the datafiles because it tries to do the upload through port 990 (the default. Looking at the settings then indeed the default 990 has been reset iso the 21. Setting it again to 21 it works.

    Apart from that, it still does not upload, looking further... I'll maybe get back on this.

    There is another issue that's open: the MySQL realtime (see first logfile I posted here). As I don't do anything with the SQL itself it seems to be an error in the realtime retainment statement. Is that correct? (I switched the realtime SQL off to avoid disturbance).
    Hans

    https://meteo-wagenborgen.nl
    CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
    User avatar
    mcrossley
    Posts: 12685
    Joined: Thu 07 Jan 2010 9:44 pm
    Weather Station: Davis VP2/WLL
    Operating System: Bullseye Lite rPi
    Location: Wilmslow, Cheshire, UK
    Contact:

    Re: CMX crash and realtime retain failure

    Post by mcrossley »

    Hi Hans, I'd need to see a log with debug switched on to look at the real time SQL retain issue. Unfortunately the current code does not dump the whole SQL statement on error.
    User avatar
    HansR
    Posts: 5870
    Joined: Sat 20 Oct 2012 6:53 am
    Weather Station: GW1100 (WS80/WH40)
    Operating System: Raspberry OS/Bullseye
    Location: Wagenborgen (NL)
    Contact:

    Re: CMX crash and realtime retain failure

    Post by HansR »

    This is the fragment between the setting of the debug and the end of cycle.
    If you need more let me know.

    Code: Select all

    2021-04-11 16:50:33.245 Updating program settings
    2021-04-11 16:50:33.308 Writing Cumulus.ini file
    2021-04-11 16:50:33.339 Completed writing Cumulus.ini file
    2021-04-11 16:50:34.095 LOOP: Data - 8: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-09-0A-4E-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-DA-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-6C-FB
    2021-04-11 16:50:34.095 LOOP: 8 - Data packet is good
    2021-04-11 16:50:36.093 LOOP: Data - 9: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0B-0A-4E-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-DA-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-A1-20
    2021-04-11 16:50:36.094 LOOP: 9 - Data packet is good
    2021-04-11 16:50:38.099 LOOP: Data - 10: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0B-0A-51-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-DA-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-B5-D8
    2021-04-11 16:50:38.100 LOOP: 10 - Data packet is good
    2021-04-11 16:50:40.077 LOOP: Data - 11: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0B-0A-51-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-DA-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-B5-D8
    2021-04-11 16:50:40.077 LOOP: 11 - Data packet is good
    2021-04-11 16:50:42.095 LOOP: Data - 12: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0B-0A-51-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-DA-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-B5-D8
    2021-04-11 16:50:42.095 LOOP: 12 - Data packet is good
    2021-04-11 16:50:44.092 LOOP: Data - 13: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0B-0A-52-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-DA-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-01-D7
    2021-04-11 16:50:44.093 LOOP: 13 - Data packet is good
    2021-04-11 16:50:46.100 LOOP: Data - 14: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0A-0A-53-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-D5-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-D3-C3
    2021-04-11 16:50:46.100 LOOP: 14 - Data packet is good
    2021-04-11 16:50:48.350 LOOP: Data - 15: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0A-0A-52-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-D5-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-4F-D9
    2021-04-11 16:50:48.351 LOOP: 15 - Data packet is good
    2021-04-11 16:50:50.096 LOOP: Data - 16: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0A-0A-52-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-D5-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-4F-D9
    2021-04-11 16:50:50.096 LOOP: 16 - Data packet is good
    2021-04-11 16:50:52.104 LOOP: Data - 17: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-0A-0A-48-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-D5-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-97-11
    2021-04-11 16:50:52.105 LOOP: 17 - Data packet is good
    2021-04-11 16:50:52.635 GetAlCurrent: Outdoor - Waiting for lock
    2021-04-11 16:50:52.636 GetAlCurrent: Outdoor - Has the lock
    2021-04-11 16:50:52.636 GetAlCurrent: Outdoor - Sending GET current conditions request 1 to AL: http://192.168.178.5/v1/current_conditions ...
    2021-04-11 16:50:52.705 GetAlCurrent: Response - {
      "data":{
        "did":"000000000000",
        "name":"FakeAirLink",
        "ts":1618152652,
        "conditions":[ {
            "lsid":000000,
            "data_structure_type":6,
            "temp":61.0,
            "hum":56.1,
            "dew_point":-1,
            "wet_bulb":-1,
            "heat_index":-1,
            "pm_1_last":-1,
            "pm_2p5_last":-1,
            "pm_10_last":-1,
            "pm_1":0.00,
            "pm_2p5":0.00,
            "pm_2p5_last_1_hour":0.00,
            "pm_2p5_last_3_hours":0.01,
            "pm_2p5_last_24_hours":1.32,
            "pm_2p5_nowcast":0.01,
            "pm_10":0.50,
            "pm_10_last_1_hour":0.16,
            "pm_10_last_3_hours":0.20,
            "pm_10_last_24_hours":2.01,
            "pm_10_nowcast":0.2,
            "last_report_time":1618152652,
            "pct_pm_data_last_1_hour":100,
            "pct_pm_data_last_3_hours":100,
            "pct_pm_data_last_24_hours":100,
            "pct_pm_data_nowcast":100
          } ]
        },
        "error":null
      }
    
    2021-04-11 16:50:52.706 DecodeAlCurrent: Outdoor - Found AirLink data
    2021-04-11 16:50:52.706 DecodeAlCurrent: Outdoor - Using temp/hum data
    2021-04-11 16:50:52.706 GetAlCurrent: Outdoor - Releasing lock
    2021-04-11 16:50:53.756 Realtime[196]: Start cycle
    2021-04-11 16:50:53.758 Realtime[196]: Creating realtime.txt
    2021-04-11 16:50:53.762 Realtime[196]: Running SQL command: INSERT IGNORE INTO Realtime (LogDateTime,temp,hum,dew,wspeed,wlatest,bearing,rrate,rfall,press,currentwdir,beaufortnumber,windunit,tempunitnodeg,pressunit,rainunit,windrun,presstrendval,rmonth,ryear,rfallY,intemp,inhum,wchill,temptrend,tempTH,TtempTH,tempTL,TtempTL,windTM,TwindTM,wgustTM,TwgustTM,pressTH,TpressTH,pressTL,TpressTL,version,build,wgust,heatindex,humidex,UV,ET,SolarRad,avgbearing,rhour,forecastnumber,isdaylight,SensorContactLost,wdir,cloudbasevalue,cloudbaseunit,apptemp,SunshineHours,CurrentSolarMax,IsSunny,FeelsLike) Values('21-04-11 16:50:53',5.4,82,2.6,18.3,20.1,328,0.0,3.4,1014.7,'NNW','3','km/h','C','hPa','mm',394.1,'+0.8',24.6,202.8,6.2,20.1,49,1.8,+0.1,7.0,'16:15',2.9,'01:35',34.9,'12:43',46.3,'02:47',1014.7,'16:50',1007.0,'05:14','3.10.5','3122',36.2,5.4,5.4,0.5,0.97,213,325,0.0,1,'1','0','NW',1156,'ft',0.3,0.5,540,'0',1.8)
    2021-04-11 16:50:53.794 Realtime[196]: 1 rows were added.
    2021-04-11 16:50:53.795 Realtime[196]: Running SQL command: DELETE IGNORE FROM Realtime WHERE LogDateTime < DATE_SUB('2021-04-11 16:50:53', INTERVAL retainVal=2 retainUnit=day)
    2021-04-11 16:50:53.819 Realtime[196]: Error encountered during Realtime delete MySQL operation.
    2021-04-11 16:50:53.820 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'retainUnit=day)' at line 1
    2021-04-11 16:50:53.820 Realtime[196]: Processing realtime file - realtimegauges.txt
    2021-04-11 16:50:53.829 Realtime[196]: Processing extra file[1] - utils/airlinkrealtime.txt
    2021-04-11 16:50:54.103 LOOP: Data - 18: 4C-4F-4F-3C-00-DD-06-D0-74-AA-02-31-AA-01-09-0A-4A-01-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-FF-52-FF-FF-FF-FF-FF-FF-FF-00-00-05-D5-00-37-00-95-44-11-00-7B-00-24-04-24-00-3C-00-54-01-FF-FF-FF-FF-FF-FF-FF-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-2A-03-06-4B-82-02-EA-07-0A-0D-9C-22
    2021-04-11 16:50:54.103 LOOP: 18 - Data packet is good
    2021-04-11 16:50:54.366 Realtime[196]: Uploading - realtime.txt
    2021-04-11 16:50:54.550 Realtime[196]: Uploading - realtimegauges.txt
    2021-04-11 16:50:54.725 Realtime[196]: Uploading extra web file[1] utils/airlinkrealtime.txttmp to /public_html/airlinkrealtime.txt
    2021-04-11 16:50:54.910 Realtime[196]: End cycle
    
    Hans

    https://meteo-wagenborgen.nl
    CMX build 4017+ ● RPi 3B+ ● Raspbian Linux 6.1.21-v7+ armv7l ● dotnet 8.0.3
    Post Reply