Mmmmm interesting

[The PIT]

I've been messing around with the mysql php scripts to produce graphs for 30 days.
Works very well apart from the temperature graphs and Barometer. I need change the format in regards to the Barometer so I'm not to bothered about that one.

Any when the graphs are produced there a few spikes to zero. Being sad I've looked through the day file on one of the days concerned and there aren't any jumps to zero in the data.

So I'm presuming there is a hiccup in the mysql database itself. Anyway to look through that too see if anything is amiss I've got a free version of Toad but that seems to stall after a bit.

graphTempOut.png

[Mapantz]

Are you not able to go to phpMyAdmin and look at the dayfile table?

[The PIT]

I haven't phpadmin installed. However I have fired up log file viewer in mx and that doesn't show any zero figures during one of the days concerned. Neither do the monthly stats.

I haven't done much to alter the files graphsettings.php

<?php
/*!
* Simple static trend graphs for Cumulus MX
*
* Created by Mark Crossley, January 2015
*
* Released under GNU GENERAL PUBLIC LICENSE, Version 2, June 1991
* See the enclosed License file
*
* File encoding = UTF-8
*
*/

$GRAPH = array();
############################################################################
# CONFIGURATION INFORMATION
############################################################################
$GRAPH['version'] = '4.3.2';
$GRAPH['width'] = 1200;
$GRAPH['height'] = 400;
$GRAPH['jsonloc'] = 'c:/Program Files (X86)/CumulusMx/web';
$GRAPH['jpgraphloc'] = 'c:/php/includes/'; // Path to jpgraph, or put it in your PHP path
// NOTE: This is the *server* operating system path to the file, so unless you have added
// the web folder to your PHP path, it will be something like "/home/<userid>/www/jpgraph/"
$GRAPH['cachetime'] = 10; // number of minutes to cache images
$GRAPH['hours'] = 720; // number of hours to graph
$GRAPH['uom'] = array('temp'=>'C', 'rain'=>'mm', 'wind'=>'mph', 'baro'=>'hpa', 'cloudbase'=>'feet');
$GRAPH['rosePoints'] = 16; // 8 or 16
$GRAPH['roseSize'] = 400;
// Localised Compass point array
$GRAPH['compass'] = array('N','NNE','NE','ENE','E','ESE','SE','SSE','S','SSW','SW','WSW','W','WNW','NW','NNW');

#---------------------------------------------------------------------------

############################################################################
# Includes for JpGraph
############################################################################
include $GRAPH['jpgraphloc'] . 'jpgraph.php';
include $GRAPH['jpgraphloc'] . 'jpgraph_line.php';
include $GRAPH['jpgraphloc'] . 'jpgraph_scatter.php';
include $GRAPH['jpgraphloc'] . 'jpgraph_date.php';
include $GRAPH['jpgraphloc'] . 'jpgraph_plotline.php';
include $GRAPH['jpgraphloc'] . 'jpgraph_windrose.php';

// Set the TZ to UTC so times display in 'station' time
//define('TIMEZONE', 'Europe/London');
// date_default_timezone_set('UTC'); Removed as set the incorrect time
// echo date(DATE_RFC2822);
// Current field names (matches tag fields) used
$GRAPH['cvalues'] = array(
'date','time','temp','hum','dew','wspeed','wlatest','bearing','rrate',
'rfall','press','currentwdir','beaufortnumber','windunit','tempunitnodeg','pressunit','rainunit',
'windrun','presstrendval','rmonth','ryear','rfallY','intemp','inhum','wchill',
'temptrend','tempTH','TtempTH','tempTL','TtempTL',
'windTM','TwindTM','wgustTM','TwgustTM',
'pressTH','TpressTH','pressTL','TpressTL',
'version','build',
'wgust','heatindex','humidex','UV','ET','SolarRad','avgbearing',
'rhour','forecastnumber','isdaylight','SensorContactLost','wdir','cloudbasevalue','cloudbaseunit',
'apptemp','SunshineHours','CurrentSolarmax','IsSunny');


############################################################################
# COMMON FUNCTIONS
############################################################################

function graph_common($graph) {
// global $graph;

// Remove the default theme
$graph->graph_theme = null;

// Ensure anti-aliasing is off.
$graph->img->SetAntiAliasing(false);

// Setup margin and titles
$graph->SetMargin(50, 20, 20, 85);

$graph->xaxis->HideLine(false);
$graph->xaxis->HideTicks(false, false);
$graph->xaxis->SetPos('min');
$graph->xgrid->SetFill(false);
$graph->xgrid->SetColor('gray');
$graph->xgrid->SetLineStyle('dotted');
$graph->xgrid->Show(true, false);

$graph->yaxis->HideLine(false);
$graph->yaxis->HideTicks(false, false);
$graph->yaxis->scale->SetGrace(5, 5);
//$graph->yaxis->SetTitleMargin(32);
$graph->ygrid->SetFill(false);
$graph->ygrid->SetColor('gray');
$graph->ygrid->SetLineStyle('dotted');

// Plot area settings
$graph->SetBox($aDrawPlotFrame=true, $aPlotFrameColor=array(100,100,100), $aPlotFrameWeight=1);
$graph->SetBackgroundGradient($aFrom=array(200,220,220), $aTo=array(250,255,255), $aGradType=GRAD_MIDHOR, $aStyle=BGRAD_FRAME);

$graph->legend->SetFillColor('#d0d0d0');
$graph->legend->SetFrameWeight(1);
$graph->legend->SetPos(0.5, 0.98, 'center', 'bottom');
$graph->legend->SetLayout(LEGEND_HOR);

// Adjust the start time for an "even" 6 hours
// $graph->xaxis->scale->SetTimeAlign(HOURADJ_12);

// Force labels to only be displayed every 6 hours, tick every hour
$graph->xaxis->scale->ticks->Set(48*60*60, 48*60*60);
//$graph->xaxis->scale->ticks->Set(1*60);

// Use hour:minute format for the labels
//$graph->xaxis->scale->SetDateFormat('H:i');
$graph->xaxis->SetLabelAngle(90);


}

function get_data($fields) {
global $GRAPH;
$retVal = array();

include 'db_ro_details.php';

// Connect to the database
$mysqli = new mysqli($db_host, $db_user, $db_pass, $db_name);
if ($mysqli->connect_errno) {
die('Failed to connect to the database server - ' . $mysqli->connect_error);
}

$cols = '';
foreach ($fields as $fld) {
if (array_search($fld, $GRAPH['cvalues'])) {
$cols .= $fld . ',';
} else {
die("!!Failed to match field: $fld");
}
}
$cols = rtrim($cols, ',');

$query = "SELECT unix_timestamp(LogDateTime) AS time, $cols
FROM realtime
WHERE LogDateTime >= now() - INTERVAL " .$GRAPH['hours']. " HOUR
ORDER BY time";

$result = $mysqli->query($query);
if (!$result) {
die('ERROR - Bad Select Statement: ' . $mysqli->error . '<br><br>' . $query);
}


// get the field names in teh returned data
$flds = $result->fetch_fields();
$keys = array();
for ($i = 0; $i < count($flds); $i++) {
$keys[] = $flds[$i]->name;
}

// fetch the SQL data
while ($row = $result->fetch_assoc()) {
foreach($keys as $key) {
$retVal[$key][] = $row[$key];
}
}

// close connection
//$mysqli->close();
mysqli_close($mysqli);

return($retVal);
}

############################################################################
# END OF SCRIPT
############################################################################
?>


and graphtempout hasn't been altered.



<?php
/*!
* Simple static trend graphs for Cumulus MX
*
* Created by Mark Crossley, January 2015
*
* Released under GNU GENERAL PUBLIC LICENSE, Version 2, June 1991
* See the enclosed License file
*
* File encoding = UTF-8
*
*/

require_once 'graphSettings2.php';

$name = basename($_SERVER['PHP_SELF'], '.php').'.png';

// Create the graph and set a scale.
$graph = new Graph($GRAPH['width'], $GRAPH['height'], $name, $GRAPH['cachetime']);
$graph->SetScale('datlin');

//fetch the data
$data = get_data(array('temp','dew','apptemp','wchill'));

graph_common($graph);

$graph->title->Set('Temperature (°' . $GRAPH['uom']['temp']. ')');

// Create the linear plot
$lineplot1 = new LinePlot($data['temp'], $data['time']);
$lineplot1->SetWeight(2);
$lineplot1->SetLegend('Temperature');

$lineplot2 = new LinePlot($data['dew'], $data['time']);
$lineplot2->SetWeight(2);
$lineplot2->SetLegend('Dewpoint');

$lineplot3 = new LinePlot($data['apptemp'], $data['time']);
$lineplot3->SetWeight(2);
$lineplot3->SetLegend('Apparent');

$lineplot4 = new LinePlot($data['wchill'], $data['time']);
$lineplot4->SetWeight(2);
$lineplot4->SetLegend('Wind Chill');

$line = new PlotLine(HORIZONTAL,0,"blue@0.5",2);
$graph->AddLine($line);
$graph->SetClipping(true);

// Add the plot to the graph
$graph->Add($lineplot2);
$graph->Add($lineplot3);
$graph->Add($lineplot4);
$graph->Add($lineplot1);

$lineplot1->SetColor("#B22222:1.3");
$lineplot2->SetColor("#2222B2:1.3");
$lineplot3->SetColor("#22B222:1.3");

// Display the graph
@unlink(CACHE_DIR . $name);
$graph->Stroke();

?>

If I knew what I was doing in php I would print the results out.

[The PIT]

Just printed the array of data and there isn't any zero's in the data so it's an issue the plotting and jpgraph

[The PIT]

Just tried the latest version of jpgraph no difference. I wonder if it's because I'm using php 7.4.10 which is making it do something strange.

[TRB]

Over the 9 year life of my station, on old Cumulus until installing MX 2 or 3 months ago, it was common to get an occasional spike to zero or even to minus 40.

For 100% of the many times it was always visible in the *log.txt files but not always quickly found.

Usually it was just a single line of data in my 5 minute updates and I would either edit it or delete the whole line if there were other corruptions.

To help quickly find the suss data I used https://download.cnet.com/CSVed/3000-2351_4-52868.html

I note in you case it is only the external temp graph affected. Is that not suggesting data rather than a graphing issue?

[mcrossley]

Trev, do you have Spike Removal settings enabled, they may be useful for you?

[The PIT]

The Barometer does the same I been through the data in that file after creating a json file and loading into notepad++. Possible I could have missed it. I'll try your file maybe tonight but I'm half asleep right now.

graphPress.png

[The PIT]

You can also step through the data in cumulusmx and there isn't anything there to cause the issue .

[The PIT]

Looked at the data files and they are fine so it looks like a bug in jpgraph.

[TRB]

Trev, do you have Spike Removal settings enabled, they may be useful for you?

Thanks Mark,

I have to admit I’ve never properly understood spike removal in a Cumulus context.

As an old electronics tech to me a spike was a very brief voltage transient and measured much more by the shortness of time rather than amplitude.

My Cumulus focus on spike removal was with wind gusts and again my thinking was directed to how brief the gusts were. So perhaps my technical training has pushed me in the wrong direction and now confronted with a review of settings I see that we should be discussing only amplitude and not duration. What I don’t have is the definition of a spike. My guess is an out of character variation between one data value and the next data value.

Checking my old Cumulus settings (see pic) I have temperature set at what I assume is a default 999 degrees. That in itself remains confusing. What does 999 mean? Let’s say it was set to 50 degrees. Does that mean a spike is declared when the temp rises above 50 or when the temp changes by 50, say from -10 to +40?

What happens when a spike setting is zero? I set my wind spike values to zero yesterday and nothing has changed.

Speaking of wind... since installing MX I have been puzzled by graphs with flat tops. Here are today’s wind values (6 hours worth) and a comparison between old and new Cumulus (Cumuli?). I got the old using Select-A-Graph with the same data file. Paranoid about data loss on my WS dedicated laptop I auto-back-up its data to my primary PC every hour and there I can run old Cumulus with current data.

The red 'Wind' is quite similar in shape and amplitude. The green ‘Gust’ graph of old has spiky tops and bottoms (that make sense to me) whereas MX has unrealistic flat tops. Also I cannot explain why the old and new green lines basically track each other until time 0700. At about 0620 both peak at around 34 and at 0640 peak at 25 yet after 0700 MX displays much higher values than old Cumulus – yet they both source the same data file.

[mcrossley]

Spike removal checks subsequent values, and if the new value changes by more than the spike value from the previous value it is ignored.


By default MX uses a 10 minute max gust window, so a gust value will "hold" for ten minutes unless it is exceeded, then that value will hold for ten minutes etc. This is normal behaviour, the gust speed is the max speed recorded in the last 10 minutes.

It looks like your C1 is not set to use 10 minute gust values. It has an option for the wind graph "Plot latest gust"...

On the wind graph, Cumulus normally plots the highest gust from the last 10 minutes. Selecting this option causes Cumulus to plot the latest 'gust' value from the station instead. The effect of this depends on the type of station you have, and how it generates this 'latest' value; you may prefer it this way, or you may not. Note that changing this does not cause the graph to be redrawn, it just changes the method of plotting from that point onwards.

[TRB]

Thanks again Mark

Three things...

1. My C1 did have the ‘plot latest gust’ flag set as shown in both the pic and the cumulus.ini value thus... PlotLatestGust=1

Also there seems no equivalent PlotLatestGust flag in MX so is that not an option?

ws options.jpg

2. Also set in C1 was the 10 min average calc. I zeroed that value in MX ini file and from that moment the wind speed chart displayed more detail after time 10:00

ws options2.jpg

wind graph.jpg

3. Despite the agreed logic I remained puzzled by your comment in terms of results.... Spike removal checks subsequent values, and if the new value changes by more than the spike value from the previous value it is ignored.

I mentioned that as a test I set the wind and gust spike value to zero so why did it not ignore every subsequent value after that and display a flat line?

[mcrossley]

1. No, that option does not exist in MX
2. OK - but that is not the normal way wind speed is presented. A 10 minute (and or 2 minute) average is "standard".
3. That will be a bug in MX for wind spike removal! Fixed for the next release.

[The PIT]

Looks my thread has been hijacked.

Anyways I think the plunges down to zero or below are when I close MX prior to a reboot of the machine.

I close mx by ctrl c so it should fill the data on the restart but doesn't seem to.