Raspberry Pi

[Adrian Hudson]

Hi Steve,

Obviously I don't have your version with the timings coded in but:

If I remove all Extra Files then the Pi takes about 45s to generate the standard files - i.e. the CPU runs at 90%-100% for that time before returning to a few %

If I add CUtags.php (for the Saratoga templates) generation only (no ftp) - so the standard files plus this file are generated - the CPU runs 90%-100% for about 1m 20s

If I then add cumuluswebtags.php generation only it takes about 2m 15s at high CPU.

By the way, my Pi is a model B (an early one if that makes any difference - I don't think so) with a pretty standard Raspian distro. All I have done is add VNC. I could try wiping the Pi and starting again but that's probably a bit drastic.

Very odd that one Pi should take a couple of seconds (yours) and another take - well, a long time.

Edit: Forgot to say, on Windows 7 PC. Not a particularly powerful one, it takes about 2-3 seconds to generate the standard files and process 3 extra files.

[steve]

Having left my Pi running overnight I am now seeing similar things to you and I understand why. Most of the time is taken up creating the graph data files:

Create standard files: 14 s
Create graph data : 150 s
Upload standard files: 15 s
Upload graph data files: 26 s

Just after startup, my 'recent data' for the graphs was quite small, as it was pre-loaded from log entries. After running overnight, it will now have considerably more data in it, as it's at 1-minute intervals.

As it happens, while changing something today for build 3007, I noticed that it is creating two sets of json data files. The 'd3' ones are for the current graphs, but I can't remember what the other ones were for. Anyway, I've taken that code out, so that should in theory halve the time taken to create the files (they don't get uploaded automatically so the upload time won't change).

The above logging (which also includes 'extra files') is in build 3007, so you'll be able to see your breakdown.

[henkg]

Seconds upload (number of seconds shown by 'top' with a CPU % of 95)
3006: 100 seconds
3007: 20 seconds
Five times faster: quite an approvement!

[steve]

Have you taken into account that just after startup, unless you use a logging interval of one minute, there is less recent data to process?

[henkg]

I do not understand (since I do not understand how and what data is used). First check was after the interval period, not immedidately after updating to 3007. Anyway, I will check again tomorrow.

[steve]

When it starts up, it loads in the last 24 hours of data from the log files, to use for the graphs, so the number of points plotted depends on the logging interval. While it is running, it is adding a new entry once a minute (and discarding entries more than 24 hours old). So, unless you are using a log interval of one minute, there are fewer points plotted for the first 24 hours after startup.

[Adrian Hudson]

When it starts up, it loads in the last 24 hours of data from the log files, to use for the graphs, so the number of points plotted depends on the logging interval. While it is running, it is adding a new entry once a minute (and discarding entries more than 24 hours old). So, unless you are using a log interval of one minute, there are fewer points plotted for the first 24 hours after startup.

Ahh, I was wondering what you meant earlier too.

Anyway, its faster than it was now

21 seconds to make the HTML files
15 s to make the graph data files
85 s to create the extra files which consist of CUtags.txt and cumuluswebtags.txt that are processed for tags
4 s to upload everything

So, if you want to look anywhere - any speeding up of processing of extra files would bear the greatest fruit.

[Adrian Hudson]

another oddity on the Pi

When I closed it down by pressing a key earlier today (while it was running normally) I got this:

2015-01-13 16:35:17.470 Done creating graph data files, creating extra files
2015-01-13 16:36:38.395 Done creating extra files
2015-01-13 16:36:38.605 Uploading extra files
2015-01-13 16:36:39.915 Done uploading extra files, uploading standard files
2015-01-13 16:36:41.663 Done uploading standard files, uploading graph data files
2015-01-13 16:36:43.096 Done uploading graph data files
Key pressed - terminating
2015-01-13 16:37:25.960 Cumulus closing
2015-01-13 16:37:25.961 Writing Cumulus.ini file
2015-01-13 16:37:26.421 Closing connection
2015-01-13 16:37:26.440 Station stopped
2015-01-13 16:37:26.678 System.InvalidOperationException: Specified port is not open.
at System.IO.Ports.SerialPort.CheckOpen () [0x00000] in <filename unknown>:0
at System.IO.Ports.SerialPort.get_BytesToRead () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.Ports.SerialPort:get_BytesToRead ()
at A.cf7e9d95b4bca10284aa98260770f84ed.cef318bad79e6e2ae25ef095f05a3381a (Int32 c06ddffdff9a9e3ca800b3fbd0094d363) [0x00000] in <filename unknown>:0
System.InvalidOperationException: Specified port is not open.
at System.IO.Ports.SerialPort.CheckOpen () [0x00000] in <filename unknown>:0
at System.IO.Ports.SerialPort.DiscardInBuffer () [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.Ports.SerialPort:DiscardInBuffer ()
at A.cf7e9d95b4bca10284aa98260770f84ed.c0b733c832f79110a0f49c86a0a9ed7fb (System.IO.Ports.SerialPort c280b81a22ba57650c0bd1f113b9e6173, System.String c6f2e2480619541814a900d39230835dd) [0x00000] in <filename unknown>:0
at A.cf7e9d95b4bca10284aa98260770f84ed.Start () [0x00000] in <filename unknown>:0
at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0

[steve]

Any errors that get logged (and there are a number that can appear) when closing down can be ignored, unless you believe they are indicative of a problem. They are mostly timing related, in this case the main thread has closed the port but the data reading thread hasn't got the message to close yet.

[Adrian Hudson]

Any errors that get logged (and there are a number that can appear) when closing down can be ignored, unless you believe they are indicative of a problem. They are mostly timing related, in this case the main thread has closed the port but the data reading thread hasn't got the message to close yet.

OK thanks Steve

[WoodburyMan]

CumulusMX 3.007 is now running on my B+. As far as I can tell everything is running fine and smooth. Everything is updating as it should to my site, extra files and all. So easy to copy over. Just edited my config file to point to the new paths for some extra files I have and that was it. I set it to load automatically via rc.local so it loads on startup. (note I had to do this in order for it to load.. it did not like being started from another path or directory via absolute path...

Code: Select all

cd /home/pi/CumulusMXDist && sudo mono CumulusMX.exe

And I set up a Samba server to easily copy files over and backup to my Desktop every few days...
http://raspberrywebserver.com/serveradm ... twork.html

How to you get that timing information for how long it takes to generate the graphs?
I can see via top and htop CPU useage and timing.. im getting 1.26, 1/13, 1.06 averages (Dual core so I assume 2.0 is max). Like all of yours I get my CPU pegged to 100% by mono when all my graphics and extra files are generated. It seems all the regular FTP upload generated files takes about 2 minutes to generate total for me... The biggest is CUTags.txt...

Code: Select all

ExtraLocal0=/home/pi/CumulusMXDist/WXtags-template-files/CUtags.txt
ExtraRemote0=/grudzien.us/CUtags.php
ExtraProcess0=1
ExtraBinary0=0
ExtraRealtime0=0
ExtraFTP0=1
ExtraUTF0=0
ExtraLocal1=/home/pi/CumulusMXDist/WXtags-template-files/realtimegaugesT.txt
ExtraRemote1=/grudzien.us/realtimegauges.txt
ExtraProcess1=1
ExtraBinary1=0
ExtraRealtime1=1
ExtraFTP1=1
ExtraUTF1=0
ExtraLocal2=/home/pi/CumulusMXDist/WXtags-template-files/gauges-ss-basic.htm
ExtraRemote2=/grudzien.us/gauges-ss.htm
ExtraProcess2=1
ExtraBinary2=0
ExtraRealtime2=0
ExtraFTP2=1
ExtraUTF2=0
ExtraLocal3=/home/pi/CumulusMXDist/WXtags-template-files/realtime.xml
ExtraRemote3=/grudzien.us/realtime.xml
ExtraProcess3=1
ExtraBinary3=0
ExtraRealtime3=1
ExtraFTP3=1
ExtraUTF3=0
ExtraLocal4=/home/pi/CumulusMXDist/WXtags-template-files/sunbird.txt
ExtraRemote4=/grudzien.us/sunbird.txt
ExtraProcess4=1
ExtraBinary4=0
ExtraRealtime4=0
ExtraFTP4=1
ExtraUTF4=0

*(EDIT* I unchecked processing on all my extra files and still have almost 2 minutes of 100% CPU useage from Mono every 5 minutes when items are generated. I do have years of data, all recorded at 1 minute intervals.

[tumutbound]

The latest release (3007) seems to have addressed the CPU usage issue as well.
I'm running sar/sadc on my RPi B+ and for the last day or so, average CPU usage is around 20%, mostly user with around 3-4% of that as system usage.
AS the RPi is averaging 80% idle, I'll have to find something else for it to do!

[steve]

How to you get that timing information for how long it takes to generate the graphs?

Add Logging=1 to [Station] in Cumulus.ini (with it stopped).

[henkg]

CPU usage > 95% with 3007 (after running 24h)
2015-01-15 08:00:00.552 Creating standard HTML files
2015-01-15 08:00:14.517 Done creating standard HTML files, creating graph data files
2015-01-15 08:01:04.224 Done creating graph data files, creating extra files
2015-01-15 08:01:04.238 Done creating extra files
2015-01-15 08:01:04.486 Uploading extra files
2015-01-15 08:01:04.499 Done uploading extra files, uploading standard files
2015-01-15 08:01:05.954 Done uploading standard files, uploading graph data files
2015-01-15 08:01:07.366 Done uploading graph data files
Total: 70 seconds.
Remarks:
- Only one extra file: realtimegaugesT.txt
- FTP upload is within my LAN, to a Synology NAS, DS412+. This might be faster than to a hosting provider
I will keep CumulusMX running for another 24h before updating to 3008 (or 3009)

[steve]

We can't expect great performance from the Pi, but the time taken to produce the graph data is long compared to everything else. I'll have a look some time to see if I can optimise it a bit.