System Out Of Memory CumulusMX 3038

[SJG1976]

20160620-150023.txt

Not sure if this is to do with the Certificate Update Earlier.

I'm not getting the following in the MXDiags.

Code: Select all

2016-06-20 17:58:46.999 System.OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
  at (wrapper alloc) object:AllocString (intptr,int)
  at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
  at . () [0x00000] in <filename unknown>:0
  at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:58:47.499 System.OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
  at (wrapper alloc) object:AllocString (intptr,int)
  at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
  at . () [0x00000] in <filename unknown>:0
  at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:58:48.000 System.OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
  at (wrapper alloc) object:AllocString (intptr,int)
  at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
  at . () [0x00000] in <filename unknown>:0
  at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:58:48.500 System.OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
  at (wrapper alloc) object:AllocString (intptr,int)
  at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0
  at . () [0x00000] in <filename unknown>:0
  at .ToString () [0x00000] in <filename unknown>:0
2016-06-20 17:59:00.775 Writing log entry for 20/06/2016 17:59:00
2016-06-20 17:59:00.777 Written log entry for 20/06/2016 17:59:00
2016-06-20 18:00:00.780 Hour changed:18

Web Page Updates Stop working, and no communication to the Station seems to take place.
A Stop and Re-Start will start it all going again, but it seems to loose the Data then.

I had 9.8mm of rain, and after a re-start it's back to 0.0mm

[steve]

You have a corruption in your June log which needs fixing:

2016-06-20 15:00:28.484 Error at line 3355 of data/Jun16log.txt : Input string was not in the correct format
2016-06-20 15:00:28.484 Please edit the file to correct the error

And also problems in today.ini, which is why today's rain figure is resetting.

I don't think the out of memory issue can be directly because of the certificate update, but perhaps indirectly because you now have Twitter working - or do you? Was it updating Twitter OK up until this point? You could try turning Twitter off to see if the problem goes away.

If you turn debug logging on, there may be more useful information logged.

[SJG1976]

Thanks Steve,

Had 2 Rows on NULL in the Jun16.txt File.

That's now been resolved and working.
Yes Twitter is working and posting updates at the Required times.

Will see if I can dig out the today.ini and see what's up there as well.

Do you sleep on the Keyboard, you seem to reply very quickly

Can't see anything wrong in today.ini though

[steve]

Your today.ini has been overwritten now so it's OK again, except for the start of day rain counter, which got reset because of the earlier problem with today.ini. You can use the 'today's rain editor' to correct it.

I do spend too much time at the keyboard - I'm here all day because of my job, and then I tend to carry on into the evening. When I'm doing other things I tend to look in regularly on the forum and the internet in general.

[SJG1976]

Thanks Steve.

Well I think I've done it...

Not sure the correct way, but hey ho.

First backed up today.ini and Jun16.txt
Deleted today.ini

Edited Jun16.txt and removed all the Data before the Rain Started.
Startup up again, and let it download all the Data from the station.

That didn't seem to have the desired effect though.

[steve]

I don't follow that - what were you trying to fix with that procedure? If you just wanted to edit today's rain, the editor would do that. If you wanted to 'rewind', to re-download data from the logger, the safest, easiest way to do that is to use one of the backups Cumulus creates.

What was the desired effect that what you did didn't have?

[SJG1976]

The idea was to start again from Midnight, and get all the data back from the console.

Failed miserably though.

I've made a right mess now.

Think I just use today as a Learning Curve.
And Just Amend today's rain to the correct amount.

Thanks for your Patience Steve.

[SJG1976]

Well the Restoring from Backup was much easier.

Was getting in a right mess there.

So in the end.

I Copied all the GOOD backup Data over the DATA File Directory.
Started again.

Now it's all good.

Easy when you do it the right way

Thanks for you help.

[SJG1976]

Well it looks like it is a Twitter posting Update issue.
I can only think the Certificates screwed it up somehow?

Been running with Twitter on, and the Realtime Updates stop, and the Twitter update stops after about the 2nd Tweet.

Still can't see what's wrong though?
Which Extra Debug logs do I need to switch on is it just the FTP logs?

[SJG1976]

I've attached the latest logs.
Looks like it isn't TWITTER?

Code: Select all

2016-06-21 07:15:00.324 Writing today.ini, LastUpdateTime = 21/06/2016 07:15:00 raindaystart = 409.4 rain counter = 409.4
2016-06-21 07:15:00.326 http://wow.metoffice.gov.uk/automaticreading?siteid=30826477&siteAuthenticationKey=******&dateutc=2016-06-21+06%3A15%3A00&winddir=240&windspeedmph=2.7&windgustmph=6.0&humidity=84&tempf=61.4&rainin=0.00&dailyrainin=0.00&baromin=29.975&dewptf=56.5&UV=0.6&solarradiation=188&softwaretype=Cumulus%20v3.0.0&action=updateraw
Web tag error
Exception: i=28831 len=41158
inputText.Length=41158
token=ByMonthRainRateH mon=10
System.OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
  at (wrapper alloc) object:AllocString (intptr,int)
  at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0 
  at . () [0x00000] in <filename unknown>:0 
2016-06-21 07:15:00.535 System.OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int)
  at (wrapper alloc) object:AllocString (intptr,int)
  at System.String.Concat (System.String str0, System.String str1) [0x00000] in <filename unknown>:0 
  at . () [0x00000] in <filename unknown>:0 
2016-06-21 07:15:05.728 WOW Response: OK: 
2016-06-21 07:16:00.321 Writing log entry for 21/06/2016 07:16:00
2016-06-21 07:16:00.321 max gust: 6.0
2016-06-21 07:16:00.322 Written log entry for 21/06/2016 07:16:00
2016-06-21 07:16:00.323 Error writing today.ini: Out of memory

Perhaps just a coincidence.

[steve]

Debug logging adds more info to the diags files, so what you've done is correct.

The trouble with an out of memory error is that the code which suffers from and reports it may not be the code which causes it. It may not even be the same program causing it. It's really a process of elimination. It is sometimes possible to run some kind of 'leak detection', but I don't know much about that on Linux. I'll see if I can find anything out.

[SJG1976]

Thanks Steve,

I'm too much of a Linux Newbie, and a little knowledge can be dangerous!

I've not been having much luck with this Raspi Pi3.
I've now got the Cumulus mounted on a USB Flash Drive (not sure if that would cause any issues)

And just use the SD Card for the Main Raspi Stuff on a Pi3.

As the SD Card I used got corrupt after a power cut.
So it may be that?

I really don't know.

My CumulsMX running on the older Pi on just an SD Card has been working with no issues.

[SJG1976]

Well still not got anywhere...

Thought I'd try with the default twitter updates rather than a twitter.txt file.

Still not got anywhere, so now trying with no Twitter updates, as i think it's something to do with that now...

As it only started to happen after I got the certificates working and twitter updating (it has updated but seems to fail after 2 updates)

Attached are the last logs if anyone can spot anything...
Will update again tomorrow and let you know if it runs through the night with no errors.

Although the Error seems to be straight after a WOW Upload?

[steve]

Nothing helpful in that diags file. Turning off Twitter is the logical thing to try, as the problem apparently started when you got that working. But in that file, you only updated Twitter twice before the out of memory errors started, which doesn't seem to fit with that.

Do you have ftp logging turned on? If so, is the ftplog.txt file getting very big?

[SJG1976]

I've switched the FTP log off, yesterday, as I didn't find anything in there that look liked it errored.


Yes the 2 Updates went through fine, which doesn't seem to fit with it being a Twitter issue.
Unless the Certs have screwed up the WOW Updates?

I'm not sure how they work (although they are all updated as well)

Will it make a difference running Cumulus from a /mnt/usbstorage mount?

All seemd to work unless I have twitter on, so will see how it goes tonight.

Not sure it helps but when the Error happend I'm also un-able to access the http://pi:8998 address.
I ssh and view the logs doing a tail -f and see them scrolling through.

Just can't put my finger on it.

Next stage. Will be a re-image (although I need a new SD Card)

And try again with all the same settings on the same mounted USB Flash drive.

Thanks for looking again though.