cumulus stops shortly after startup

[Nossie]

follow up from this -

viewtopic.php?t=21418

Appears it's not just cumulus utils but also cumulus mx

I backed up and restored everything on 16/07 and I don't think my cmx install has been running correctly since... it seems to start - shows data and then data stops - when working with utils, the data from the last week is there - and I have finally managed it to force an update for today...

any idea why it would suddenly stop?

Also, I turned the co2 sensor back on a few days ago - think it might have been off since the 16/07 in error. However although it's showing on the display it's not being registered on extra sensors \o/

Screenshot 2023-07-24 022708.png

[HansR]

1) @Ian: the logfile stops at 2023-07-24 01:38:00.32 while the last update on the site is 24/07/2023 01:51:56. Is CMX still running? If no please restart and keep it running. Post the logfile when running long time so that we can see the json uploads as well or does CMX stop by itself (without stop command and without a message in the log)

2) Other investigators: we investigated this in the thread shown by Ian in his post (and continued in PM). We first thought it was CUtils but it is not. CMX loses contact with the sensors and apparently does not recover, it does send files but those don't have new data or do not arrive at their destination (realtime and json). No idea on this side.

[Nossie]

@Ian: the logfile stops at 2023-07-24 01:38:00.32 while the last update on the site is 24/07/2023 01:51:56. Is CMX still running?

That was the last time I made some changes - I managed to get utils to properly update including the graphs (it updated on the 17 - then I forced it to update on the 22nd thinking it might have fixed it... and yesterday - but I can't get CMX to remain connected

I've attached a much larger log from then until now.

[HansR]

ÓK, thanks.

It does not seem to be able to read its sensors. Has the device been configured correctly or has it received any change because the problem seems to be in the sensor reading/device configuration? Are the data on ecowitt.net complete and what you expect?

[Nossie]

works fine

eco.png

Nothing out of the usual

[Nossie]

I've attached another log.

pretty lost with this one, think I should try turning off upload.php and see if it makes a difference?

[HansR]

I don't see why you should be lost with the llast log, I see no change with the previous ones.

Your problem is the sensors are not being read, you should check the CMX <-> GW1100 connection/configuration. CMX doesn't do anything if it dos not read the sensors.

[Nossie]

No.

My problem is that the data is there and it's being stored - sure the CO2 unit is not transferring and the realtime sensors only update once but why should that bring down the whole of CMX?

it works on ecowitt software and hardware fine. I have changed nothing since I wiped the OS before and restored, multiple times putting back the same folders and files.

Surely there should be more than just "look ma', no hands" when CMX expects data but magically runs out - When there are megabytes of megabytes of data to be had, why do you think I can force the website to update and suddenly a weeks worth of data reappears again?

From my options - the only one I currently see as viable is to rip out utils - wipe out CMX completely and import my data and start again - while not a massive issue I can only imagine the path that will take me down.

you should check the CMX <-> GW1100 connection/configuration.

all the sensors are connected to the weatherstation equivalent of the GW1/1000 directly using the API .... and since I've now forced cmxutils to update until approx. present time - and we have data - I think we can safely say that there is data. For the most part, it's not that there is no data - it's that it starts, updates with new data and then stops.



So I changed it back to SFTP, restarted and ran the system as normal - not exactly what I expected but it's more than what I was getting before!

Code: Select all

pi@Europa:~ $ ./updateMX.sh
2023-07-24 23:12:54.432 CumulusUtils Initial: Shutting down the default listener
2023-07-24 23:12:54.459  CUstrings[xx].ini : CUstringsEN.ini looked for.
2023-07-24 23:12:54.468 CumulusUtils version: 6.20.0
2023-07-24 23:12:54.471  CuSupport constructor : Unit Wind (m/s, mph, km/h, kts): mph
2023-07-24 23:12:54.474  CuSupport constructor : Unit Pressure (mb,hPa,inHg): hPa
2023-07-24 23:12:54.477  CuSupport constructor : Unit Rain (mm,in): mm
2023-07-24 23:12:54.479  CuSupport constructor : Unit T (C,F): ?C
2023-07-24 23:12:54.482  CuSupport constructor : Unit Distance (m, mi, km, kn): mi
2023-07-24 23:12:54.503 InetSupport: Constructor start
2023-07-24 23:13:09.635 CumulusUtils : ----------------------------
2023-07-24 23:13:09.643 CumulusUtils : Entering Main

Unhandled Exception:
System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef (System.Boolean& success) [0x00025] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.NativeEventCalls.SetEvent (Microsoft.Win32.SafeHandles.SafeWaitHandle handle) [0x00002] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.EventWaitHandle.Set () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle.Set()
  at System.IOAsyncResult.set_IsCompleted (System.Boolean value) [0x00024] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete () [0x00037] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.Socket.<BeginConnectCallback>m__21 (System.IOAsyncResult ares) [0x000f0] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef (System.Boolean& success) [0x00025] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.NativeEventCalls.SetEvent (Microsoft.Win32.SafeHandles.SafeWaitHandle handle) [0x00002] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.EventWaitHandle.Set () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle.Set()
  at System.IOAsyncResult.set_IsCompleted (System.Boolean value) [0x00024] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete () [0x00037] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.Socket.<BeginConnectCallback>m__21 (System.IOAsyncResult ares) [0x000f0] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
2023-07-24 23:13:11.004 CumulusUtils Initial: Shutting down the default listener
2023-07-24 23:13:11.033  CUstrings[xx].ini : CUstringsEN.ini looked for.
2023-07-24 23:13:11.048 CumulusUtils version: 6.20.0
2023-07-24 23:13:11.051  CuSupport constructor : Unit Wind (m/s, mph, km/h, kts): mph
2023-07-24 23:13:11.054  CuSupport constructor : Unit Pressure (mb,hPa,inHg): hPa
2023-07-24 23:13:11.058  CuSupport constructor : Unit Rain (mm,in): mm
2023-07-24 23:13:11.061  CuSupport constructor : Unit T (C,F): ?C
2023-07-24 23:13:11.064  CuSupport constructor : Unit Distance (m, mi, km, kn): mi
2023-07-24 23:13:11.092 InetSupport: Constructor start
2023-07-24 23:13:26.231 CumulusUtils : ----------------------------
2023-07-24 23:13:26.234 CumulusUtils : Entering Main

Unhandled Exception:
System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef (System.Boolean& success) [0x00025] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.NativeEventCalls.SetEvent (Microsoft.Win32.SafeHandles.SafeWaitHandle handle) [0x00002] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.EventWaitHandle.Set () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle.Set()
  at System.IOAsyncResult.set_IsCompleted (System.Boolean value) [0x00024] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete () [0x00037] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.Socket.<BeginConnectCallback>m__21 (System.IOAsyncResult ares) [0x000f0] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef (System.Boolean& success) [0x00025] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.NativeEventCalls.SetEvent (Microsoft.Win32.SafeHandles.SafeWaitHandle handle) [0x00002] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.EventWaitHandle.Set () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle.Set()
  at System.IOAsyncResult.set_IsCompleted (System.Boolean value) [0x00024] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete () [0x00037] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.Socket.<BeginConnectCallback>m__21 (System.IOAsyncResult ares) [0x000f0] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
2023-07-24 23:13:26.947 CumulusUtils Initial: Shutting down the default listener
2023-07-24 23:13:26.976  CUstrings[xx].ini : CUstringsEN.ini looked for.
2023-07-24 23:13:26.985 CumulusUtils version: 6.20.0
2023-07-24 23:13:26.988  CuSupport constructor : Unit Wind (m/s, mph, km/h, kts): mph
2023-07-24 23:13:26.991  CuSupport constructor : Unit Pressure (mb,hPa,inHg): hPa
2023-07-24 23:13:26.994  CuSupport constructor : Unit Rain (mm,in): mm
2023-07-24 23:13:26.997  CuSupport constructor : Unit T (C,F): ?C
2023-07-24 23:13:27.000  CuSupport constructor : Unit Distance (m, mi, km, kn): mi
2023-07-24 23:13:27.024 InetSupport: Constructor start
2023-07-24 23:13:42.126 CumulusUtils : ----------------------------
2023-07-24 23:13:42.130 CumulusUtils : Entering Main

Unhandled Exception:
System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef (System.Boolean& success) [0x00025] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.NativeEventCalls.SetEvent (Microsoft.Win32.SafeHandles.SafeWaitHandle handle) [0x00002] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.EventWaitHandle.Set () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle.Set()
  at System.IOAsyncResult.set_IsCompleted (System.Boolean value) [0x00024] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete () [0x00037] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.Socket.<BeginConnectCallback>m__21 (System.IOAsyncResult ares) [0x000f0] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
2023-07-24 23:13:42.147 HelpTexts Contructor: start
[ERROR] FATAL UNHANDLED EXCEPTION: System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef (System.Boolean& success) [0x00025] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.NativeEventCalls.SetEvent (Microsoft.Win32.SafeHandles.SafeWaitHandle handle) [0x00002] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading.EventWaitHandle.Set () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle.Set()
  at System.IOAsyncResult.set_IsCompleted (System.Boolean value) [0x00024] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete () [0x00037] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Net.Sockets.Socket.<BeginConnectCallback>m__21 (System.IOAsyncResult ares) [0x000f0] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
pi@Europa:~ $                                          

[Nossie]

realtimegauges.txt from tonight

Code: Select all

{"date":"23:23",
"temp":"11.0",
"tempTL":"10.8",
"tempTH":"17.5",
"intemp":"20.0",
"dew":"9.1",
"dewpointTL":"7.3",
"dewpointTH":"10.6",
"apptemp":"10.8",
"apptempTL":"10.6",
"apptempTH":"17.5",
"feelslike":"11.0",
"feelslikeTL":"10.8",
"feelslikeTH":"17.8",
"wchill":"11.0",
"wchillTL":"10.8",
"heatindex":"11.0",
"heatindexTH":"17.5",
"humidex":"11.9",
"wlatest":"0",
"wspeed":"0",
"wgust":"0",
"wgustTM":"6",
"bearing":"345",
"avgbearing":"0",
"press":"1010.1",
"pressTL":"1009.3",
"pressTH":"1010.4",
"pressL":"963.4",
"pressH":"1045.2",
"rfall":"0.0",
"rrate":"0.0",
"rrateTM":"0.0",
"hum":"88",
"humTL":"58",
"humTH":"89",
"inhum":"57",
"SensorContactLost":"0",
"forecast":"",
"tempunit":"C",
"windunit":"mph",
"pressunit":"hPa",
"rainunit":"mm",
"temptrend":"-1.0",
"TtempTL":"22:45",
"TtempTH":"14:55",
"TdewpointTL":"11:20",
"TdewpointTH":"14:55",
"TapptempTL":"23:00",
"TapptempTH":"14:55",
"TfeelslikeTL":"23:00",
"TfeelslikeTH":"14:55",
"TwchillTL":"22:45",
"TheatindexTH":"14:55",
"TrrateTM":"10:00",
"ThourlyrainTH":"10:00",
"LastRainTipISO":"2023-07-23 20:05",
"hourlyrainTH":"0.0",
"ThumTL":"17:00",
"ThumTH":"22:35",
"TpressTL":"17:55",
"TpressTH":"13:15",
"presstrendval":"0.2",
"Tbeaufort":"F1",
"TwgustTM":"11:50",
"windTM":"2",
"bearingTM":"40",
"timeUTC":"2023,7,24,22,23,55",
"BearingRangeFrom10":"000",
"BearingRangeTo10":"000",
"UV":"0.0",
"UVTH":"5.0",
"SolarRad":"0",
"SolarTM":"558",
"CurrentSolarMax":"0",
"domwinddir":"NNE",
"WindRoseData":[38,32,53,21,45,28,8,13,10,1,6,7,14,40,58,47],
"windrun":"10.1",
"cloudbasevalue":"782",
"cloudbaseunit":"ft",
"version":"3.25.2",
"build":"3245",
"ver":"12"}

Yep - data! (this is after I changed it back to phpupload)

[HansR]

My problem is that the data is there and it's being stored - sure the CO2 unit is not transferring and the realtime sensors only update once but why should that bring down the whole of CMX?

No , there is no data: what you see is an historical update from ecowitt.net but reading the sensors is not done. And that's what it says: contact wit the sensors is lost.

From my options - the only one I currently see as viable is to rip out utils - wipe out CMX completely and import my data and start again - while not a massive issue I can only imagine the path that will take me down.

That brute force option would not give you understanding of what is going on and might not give result. Only do things if you understand the underlying cause.

you should check the CMX <-> GW1100 connection/configuration.

all the sensors are connected to the weatherstation equivalent of the GW1/1000 directly using the API .... and since I've now forced cmxutils to update until approx. present time - and we have data - I think we can safely say that there is data. For the most part, it's not that there is no data - it's that it starts, updates with new data and then stops.

Yes, but your weatherstation is not connected to CMX. You are only looking at historical data, it is not fetching the data 'realtime' as it says it is not connected to to the sensors. Did you read the logfile at the end?

So I changed it back to SFTP, restarted and ran the system as normal - not exactly what I expected but it's more than what I was getting before!

You are doing the wrong things here (this required to many paths to change).
Reactivate the upload.php and look at the connection between the GW1000 and CMX.

You are saying

all the sensors are connected to the weatherstation equivalent of the GW1/1000 directly using the API

while the logfile says:

Code: Select all

2023-07-24 06:22:08.222 Station type: 14 - HTTP Ecowitt

And the fact that you have an updated realtimegauges does not say anything: the LED on the site is still red and no updates are coming, the data apparently has moved from 16 to 23 july and stopped again.

I am out, maybe somebody else can have some effect.