CumulusMX Crashed today Unable to connect to station

[bab5871]

Got an email from Wunderground my station hadn't reported in 2 hours. Logged in to check and saw this error... This is on a RPi3 with the latest Jesse. Davis Vantage Vue with the Davis brand USB logging cable.

_wapi_connect: error looking up socket handle 0x10 (error 10038)
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x12
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x12
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
_wapi_connect: error looking up socket handle 0x10
System.IO.IOException: Input/output error
at System.IO.Ports.SerialPortStream.ThrowIOException () [0x00000] in <filename unknown>:0
at System.IO.Ports.SerialPortStream.DiscardInBuffer () [0x00000] in <filename unknown>:0
at System.IO.Ports.SerialPort.DiscardInBuffer () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.Ports.SerialPort:DiscardInBuffer ()
at ?.? (System.IO.Ports.SerialPort ?, System.String ?) [0x00000] in <filename unknown>:0
at ?.? () [0x00000] in <filename unknown>:0
at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
**** An error has occurred - please zip up the MXdiags folder and post it in the forum ****
Press Enter to terminate


Tried to Ctrl+c and start it again and got this...


pi@piwx:/CumulusMX $ sudo mono CumulusMX.exe
Current culture: English (United States)
Serial: B525C117C0B69EB4A9981EF9BA84F15F
Cumulus MX v.3.0.0 build 3038
Today.ini = data/today.ini
Last update=2016-07-01T07:10:00
Unable to connect to station
Cumulus running at: http://*:8998/
(Replace * with any IP address on this machine, or localhost)
Starting web socket server on port 8002
7/1/2016 9:22:53 AM
Type Ctrl-C to terminate

I rebooted the PI and started screen and started cumulusmx and it fired up just fine. mxdiags dir is attached. Any ideas?

[steve]

Some kind of problem with the USB connection, the serial/USB device disappeared:

2016-07-01 09:22:52.748 Serial device = /dev/ttyUSB0
2016-07-01 09:22:52.754 No such file or directory

Cumulus isn't given any information as to what happened, your best bet would be to look in your system logs to see if there's anything in there about that device.

[bab5871]

Interesting to say the least. Here's what I see in /var/log/messages

Jul 1 06:25:04 piwx rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="552" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul 1 07:13:27 piwx kernel: [3751405.770335] usb 1-1.3: USB disconnect, device number 5
Jul 1 07:13:27 piwx kernel: [3751405.770795] cp210x ttyUSB0: cp210x converter now disconnected from ttyUSB0
Jul 1 07:13:27 piwx kernel: [3751405.770838] cp210x 1-1.3:1.0: device disconnected
Jul 1 07:13:28 piwx kernel: [3751406.011725] usb 1-1.3: new full-speed USB device number 6 using dwc_otg
Jul 1 07:13:28 piwx kernel: [3751406.118208] usb 1-1.3: New USB device found, idVendor=10c4, idProduct=ea60
Jul 1 07:13:28 piwx kernel: [3751406.118230] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jul 1 07:13:28 piwx kernel: [3751406.118244] usb 1-1.3: Product: CP2102 USB to UART Bridge Controller
Jul 1 07:13:28 piwx kernel: [3751406.118257] usb 1-1.3: Manufacturer: Silicon Labs
Jul 1 07:13:28 piwx kernel: [3751406.118269] usb 1-1.3: SerialNumber: 0001
Jul 1 07:13:28 piwx kernel: [3751406.121118] cp210x 1-1.3:1.0: cp210x converter detected
Jul 1 07:13:28 piwx kernel: [3751406.121421] usb 1-1.3: cp210x converter now attached to ttyUSB1
Jul 1 09:24:37 piwx rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="552" x-info="http://www.rsyslog.com"] exiting on signal 15.

Ignore the syslog stuff, just there for showing timing of the events, syslog is fine.

So it dumped ttyUSB0, then found it again on ttyUSB1... hmm.

[steve]

I did wonder if it had become /dev/ttyUSB1 - I've seen that before on other people's systems - but I have no idea why it did it. I believe you can force it to always be /dev/ttyUSB0 and this has been discussed in the forum, but I can't remember how you do it just at the moment, and I don't know what the effect would have been in this case anyway. I suspect that Cumulus wouldn't have coped any better with its current code, but it probably wouldn't have then required a reboot of the Pi.

[bab5871]

Keep having more issues. It's like it freezes the data and continues to upload the same value to wunderground. Latest mxdiags crash.



pi@piwx:/CumulusMX $ sudo mono CumulusMX.exe
Current culture: English (United States)
Serial: B525C117C0B69EB4A9981EF9BA84F15F
Cumulus MX v.3.0.0 build 3038
Today.ini = data/today.ini
Last update=2016-07-08T12:50:00
Connected to station
12:56:47 PM Downloading Archive Data
12:56:47 PM Downloading Archive Data
12:56:47 PM Normal running
Cumulus running at: http://*:8998/
(Replace * with any IP address on this machine, or localhost)
Starting web socket server on port 8002
7/8/2016 12:56:47 PM
Type Ctrl-C to terminate
System.IO.IOException: Input/output error
at System.IO.Ports.SerialPortStream.ThrowIOException () [0x00000] in <filename unknown>:0
at System.IO.Ports.SerialPortStream.DiscardInBuffer () [0x00000] in <filename unknown>:0
at System.IO.Ports.SerialPort.DiscardInBuffer () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.Ports.SerialPort:DiscardInBuffer ()
at î·.î (System.IO.Ports.SerialPort î, System.String î) [0x00000] in <filename unknown>:0
at î·.î () [0x00000] in <filename unknown>:0
at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
**** An error has occurred - please zip up the MXdiags folder and post it in the forum ****
Press Enter to terminate

_wapi_thread_disown_mutex: error looking up thread by ID
_wapi_thread_disown_mutex: error looking up thread by ID
_wapi_thread_disown_mutex: error looking up thread by ID
pi@piwx:/CumulusMX $

[steve]

It's like it freezes the data and continues to upload the same value to wunderground.

Yes, that what it does, if it can't read any new data. It just uses the last data that it read. Cumulus (including v1) has always done that. Your system is taking away the device that MX is using, so it can't read any more data from that device.

[BigOkie]

I did wonder if it had become /dev/ttyUSB1 - I've seen that before on other people's systems - but I have no idea why it did it. I believe you can force it to always be /dev/ttyUSB0 and this has been discussed in the forum, but I can't remember how you do it just at the moment, and I don't know what the effect would have been in this case anyway. I suspect that Cumulus wouldn't have coped any better with its current code, but it probably wouldn't have then required a reboot of the Pi.

It's called udev rules. Might be able to get some information from this page:

https://www.element14.com/community/com ... sb-devices

Here is another that might be a little more helpful.

https://www.domoticz.com/wiki/PersistentUSBDevices

[bab5871]

Is there any way to build in an alert of some sort? Otherwise we have absolutely no idea if this crashed as it still uploads data. Or just have it stop if it has too many uploads of the same value? Not sure what a solution would be here.

[steve]

Yes, quite possibly, in a future version. It's still in beta, and still in development (when I find the time and enthusiasm to write any code). It should really be handling that exception and not allowing the thread to crash, at least. But this is just a symptom, and not the root cause of the problem.

[bab5871]

Yeah I'm still trying to figure out why it does that. I had this running on VWS on Windows XP and it would drop the connection to the USB as well. Luckily VWS would stop uploading and I'd get an alert from Wunderground so I could reboot it. I've got no idea where to start other than it's the console/data logger/USB cable.