Page 1 of 3
System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 5:04 pm
by SJG1976
20160620-150023.txt
Not sure if this is to do with the Certificate Update Earlier.
I'm not getting the following in the MXDiags.
Code: Select all
2016-06-20 17:58:46.999 System.OutOfMemoryException: Out of memory
at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
at (wrapper alloc) object:AllocString (intptr,int)
at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
at . () [0x00000] in <filename unknown>:0
at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:58:47.499 System.OutOfMemoryException: Out of memory
at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
at (wrapper alloc) object:AllocString (intptr,int)
at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
at . () [0x00000] in <filename unknown>:0
at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:58:48.000 System.OutOfMemoryException: Out of memory
at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
at (wrapper alloc) object:AllocString (intptr,int)
at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
at . () [0x00000] in <filename unknown>:0
at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:58:48.500 System.OutOfMemoryException: Out of memory
at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
at (wrapper alloc) object:AllocString (intptr,int)
at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
at . () [0x00000] in <filename unknown>:0
at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:59:00.775 Writing log entry for 20/06/2016 17:59:00
2016-06-20 17:59:00.777 Written log entry for 20/06/2016 17:59:00
2016-06-20 18:00:00.780 Hour changed:18
Web Page Updates Stop working, and no communication to the Station seems to take place.
A Stop and Re-Start will start it all going again, but it seems to loose the Data then.
I had 9.8mm of rain, and after a re-start it's back to 0.0mm
Re: System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 5:29 pm
by steve
You have a corruption in your June log which needs fixing:
2016-06-20 15:00:28.484 Error at line 3355 of data/Jun16log.txt : Input string was not in the correct format
2016-06-20 15:00:28.484 Please edit the file to correct the error
And also problems in today.ini, which is why today's rain figure is resetting.
I don't think the out of memory issue can be directly because of the certificate update, but perhaps indirectly because you now have Twitter working - or do you? Was it updating Twitter OK up until this point? You could try turning Twitter off to see if the problem goes away.
If you turn debug logging on, there may be more useful information logged.
Re: System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 5:35 pm
by SJG1976
Thanks Steve,
Had 2 Rows on NULL in the Jun16.txt File.
That's now been resolved and working.
Yes Twitter is working and posting updates at the Required times.
Will see if I can dig out the today.ini and see what's up there as well.
Do you sleep on the Keyboard, you seem to reply very quickly
Can't see anything wrong in today.ini though
Re: System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 5:43 pm
by steve
Your today.ini has been overwritten now so it's OK again, except for the start of day rain counter, which got reset because of the earlier problem with today.ini. You can use the 'today's rain editor' to correct it.
I do spend too much time at the keyboard - I'm here all day because of my job, and then I tend to carry on into the evening. When I'm doing other things I tend to look in regularly on the forum and the internet in general.
Re: System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 6:00 pm
by SJG1976
Thanks Steve.
Well I think I've done it...
Not sure the correct way, but hey ho.
First backed up today.ini and Jun16.txt
Deleted today.ini
Edited Jun16.txt and removed all the Data before the Rain Started.
Startup up again, and let it download all the Data from the station.
That didn't seem to have the desired effect though.
Re: System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 6:15 pm
by steve
I don't follow that - what were you trying to fix with that procedure? If you just wanted to edit today's rain, the editor would do that. If you wanted to 'rewind', to re-download data from the logger, the safest, easiest way to do that is to use one of the backups Cumulus creates.
What was the desired effect that what you did didn't have?
Re: System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 6:33 pm
by SJG1976
The idea was to start again from Midnight, and get all the data back from the console.
Failed miserably though.
I've made a right mess now.
Think I just use today as a Learning Curve.
And Just Amend today's rain to the correct amount.
Thanks for your Patience Steve.
Re: System Out Of Memory CumulusMX 3038
Posted: Mon 20 Jun 2016 6:43 pm
by SJG1976
Well the Restoring from Backup was much easier.
Was getting in a right mess there.
So in the end.
I Copied all the GOOD backup Data over the DATA File Directory.
Started again.
Now it's all good.
Easy when you do it the right way
Thanks for you help.
Re: System Out Of Memory CumulusMX 3038
Posted: Tue 21 Jun 2016 3:38 am
by SJG1976
Well it looks like it is a Twitter posting Update issue.
I can only think the Certificates screwed it up somehow?
Been running with Twitter on, and the Realtime Updates stop, and the Twitter update stops after about the 2nd Tweet.
Still can't see what's wrong though?
Which Extra Debug logs do I need to switch on is it just the FTP logs?
Re: System Out Of Memory CumulusMX 3038
Posted: Tue 21 Jun 2016 6:57 am
by SJG1976
I've attached the latest logs.
Looks like it isn't TWITTER?
Code: Select all
2016-06-21 07:15:00.324 Writing today.ini, LastUpdateTime = 21/06/2016 07:15:00 raindaystart = 409.4 rain counter = 409.4
2016-06-21 07:15:00.326 http://wow.metoffice.gov.uk/automaticreading?siteid=30826477&siteAuthenticationKey=******&dateutc=2016-06-21+06%3A15%3A00&winddir=240&windspeedmph=2.7&windgustmph=6.0&humidity=84&tempf=61.4&rainin=0.00&dailyrainin=0.00&baromin=29.975&dewptf=56.5&UV=0.6&solarradiation=188&softwaretype=Cumulus%20v3.0.0&action=updateraw
Web tag error
Exception: i=28831 len=41158
inputText.Length=41158
token=ByMonthRainRateH mon=10
System.OutOfMemoryException: Out of memory
at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
at (wrapper alloc) object:AllocString (intptr,int)
at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
at . () [0x00000] in <filename unknown>:0
2016-06-21 07:15:00.535 System.OutOfMemoryException: Out of memory
at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
at (wrapper alloc) object:AllocString (intptr,int)
at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
at . () [0x00000] in <filename unknown>:0
2016-06-21 07:15:05.728 WOW Response: OK:
2016-06-21 07:16:00.321 Writing log entry for 21/06/2016 07:16:00
2016-06-21 07:16:00.321 max gust: 6.0
2016-06-21 07:16:00.322 Written log entry for 21/06/2016 07:16:00
2016-06-21 07:16:00.323 Error writing today.ini: Out of memory
Perhaps just a coincidence.
Re: System Out Of Memory CumulusMX 3038
Posted: Tue 21 Jun 2016 7:39 am
by steve
Debug logging adds more info to the diags files, so what you've done is correct.
The trouble with an out of memory error is that the code which suffers from and reports it may not be the code which causes it. It may not even be the same program causing it. It's really a process of elimination. It is sometimes possible to run some kind of 'leak detection', but I don't know much about that on Linux. I'll see if I can find anything out.
Re: System Out Of Memory CumulusMX 3038
Posted: Tue 21 Jun 2016 7:48 am
by SJG1976
Thanks Steve,
I'm too much of a Linux Newbie, and a little knowledge can be dangerous!
I've not been having much luck with this Raspi Pi3.
I've now got the Cumulus mounted on a USB Flash Drive (not sure if that would cause any issues)
And just use the SD Card for the Main Raspi Stuff on a Pi3.
As the SD Card I used got corrupt after a power cut.
So it may be that?
I really don't know.
My CumulsMX running on the older Pi on just an SD Card has been working with no issues.
Re: System Out Of Memory CumulusMX 3038
Posted: Tue 21 Jun 2016 7:00 pm
by SJG1976
Well still not got anywhere...
Thought I'd try with the default twitter updates rather than a twitter.txt file.
Still not got anywhere, so now trying with no Twitter updates, as i think it's something to do with that now...
As it only started to happen after I got the certificates working and twitter updating (it has updated but seems to fail after 2 updates)
Attached are the last logs if anyone can spot anything...
Will update again tomorrow and let you know if it runs through the night with no errors.
Although the Error seems to be straight after a WOW Upload?
Re: System Out Of Memory CumulusMX 3038
Posted: Tue 21 Jun 2016 7:16 pm
by steve
Nothing helpful in that diags file. Turning off Twitter is the logical thing to try, as the problem apparently started when you got that working. But in that file, you only updated Twitter twice before the out of memory errors started, which doesn't seem to fit with that.
Do you have ftp logging turned on? If so, is the ftplog.txt file getting very big?
Re: System Out Of Memory CumulusMX 3038
Posted: Tue 21 Jun 2016 7:23 pm
by SJG1976
I've switched the FTP log off, yesterday, as I didn't find anything in there that look liked it errored.
Yes the 2 Updates went through fine, which doesn't seem to fit with it being a Twitter issue.
Unless the Certs have screwed up the WOW Updates?
I'm not sure how they work (although they are all updated as well)
Will it make a difference running Cumulus from a /mnt/usbstorage mount?
All seemd to work unless I have twitter on, so will see how it goes tonight.
Not sure it helps but when the Error happend I'm also un-able to access the
http://pi:8998 address.
I ssh and view the logs doing a tail -f and see them scrolling through.
Just can't put my finger on it.
Next stage. Will be a re-image (although I need a new SD Card)
And try again with all the same settings on the same mounted USB Flash drive.
Thanks for looking again though.