Cumulus Support

Posted: **Tue 10 Oct 2023 2:23 pm**

This concerns values logged in a CustomLog.

If I log a webtag on a five minute interval, I expect the value to be what it is at the moment it is recorded e.g. the instantaneous temperature at 6h00, 6h05, 6h10 etc... It could just as well be the average of recorded values of a sensor. So I wonder??

And how is that on a daily log of that temperature? I expect it to the average of all instantaneous values.
Is that correct?

With lightning: the nr of strikes I expect the sum in that interval but no: it is the sum on the day and it resets at midnight.
For the corresponding lightning distance (whatever we think about the usefulness of that value) I would expect it to the average value over that interval and not e.g. the last value.

How are these values on a daily basis? The nr of strikes is clear: that is the total, but is the distance really the average distance over the day?

Is there a generic to say what the value represents?

And would it be an idea to have that information - including the webtag dimension - stored somewhere to be asked when needed?
I know this is quite a demand but should/could I make a request of this?

Posted: **Tue 10 Oct 2023 6:13 pm**

I worked at Kew Observatory in the late 60s. We did weather observations every three hours between 06Z and 21Z inclusive. We also had a photo-thermograph in the famous North Wall Screen. When the photo was developed we could analyse it, recording hourly temperatures with these going on a blue form (the number has disappeared into the mists of time). The daily mean was the sum of the hourly temps divided by 24. There was something with subtracting the first midnight value from the second, dividing by 2 and adjusting the mean with the result. The details of this has also disappeared into the mists of time. But in those days most stations only had manually read hourly values - think these were averaged over the 24 hours.
Wind was different, we had a pressure-tube anemograph. We analysed the mean wind (direction and speed) in each hour and somehow (again mists) got a daily mean. gusts were the highest in each hour.
Not sure if this helps, but I do get surprised sometimes at how much precision we strive for, in something that doesn't have much precision.
It was fun working there - a Dines rate of rainfall recorder, a Bibby rate of rain recorder, an aspirated thermograph and many other (probably) unique instruments.

Posted: **Tue 10 Oct 2023 6:23 pm**

Thanks for this piece of history. I had this type of discussion before regarding the average day temperature where one argued that it IS (maxT+minT)/2 because that was how it was done in the 19th century (and it made it as a configurable option in CMX). That's all very well but we progressed in our measurement and storage and as such our calculations changed.

But my question was more specific on how the webtag values are made. It is a difference if a daily value is the last read value at midnight or the average value of all observations that day. Computers brought a lot, also confusion

Posted: **Tue 10 Oct 2023 7:33 pm**

And @Mark: the date in the CustomLogs apparently is written as in the locale. Unfortunately CUtils users sometimes use a different locale for CUtils than they use for CMX. If an en-US user writes the date it will be MM/dd/YY and there is no way of knowing with what locale it has been written.

Or is it only the separators and is it always dd-MM-YY ??

Could it be done in a fixed style ?
Any suggestions otherwise?

Posted: **Tue 10 Oct 2023 7:41 pm**

I just replied in the other thread. The format is "dd/MM/yy" just like the other log files. So the order is fixed, the separator locale dependent.

The values logged are the point in time values when the entry is created.

Posted: **Tue 10 Oct 2023 8:20 pm**

mcrossley wrote: ↑Tue 10 Oct 2023 7:41 pm I just replied in the other thread. The format is "dd/MM/yy" just like the other log files. So the order is fixed, the separator locale dependent.

The values logged are the point in time values when the entry is created.

OK. Thnx.

Posted: **Wed 11 Oct 2023 11:56 pm**

Not sure if this is a request for a feature, a change or simply a bug report.

Anyway, another issue I encounter is the naming of the Ecowitt CO2 sensor webtags. I use the webtag names as variables in the javascript procedures generated. However a name like CO2-24h is interpreted by javascript as an equation 2 - 24. In other words I cannot chart the CO2 sensor webtags.

Is it possible to rename those webtags with underscores?

Afaics these are the only webtags with a - (minus) in the name.

Posted: **Thu 12 Oct 2023 8:15 am**

HansR wrote: ↑Tue 10 Oct 2023 8:20 pm
mcrossley wrote: ↑Tue 10 Oct 2023 7:41 pm I just replied in the other thread. The format is "dd/MM/yy" just like the other log files. So the order is fixed, the separator locale dependent.

The values logged are the point in time values when the entry is created.
OK. Thnx.

Returning to this issue of the values : Agreed for the interval logs but for the daily logs it is different as the result is always the midnight value of the webtag. That seems pretty pointless, I would say a sum or an average would have meaning as a midnight value (mostly) has not.

Don't know why I can't remove my own post but OK, I would have, reason below.

Rethinking it again: Not required, there are enough webtags having the information e.g. daily summary webtags for temp or rain etc... monthly or yearly development of temperature can be logged daily. It is up to the user not to e.g. log plain temperature on a daily basis (unless he has a reason)...

Posted: **Thu 12 Oct 2023 8:58 am**

And an additional issue found while handling the CustomLogs is wrt the JSON PHP incremental uploads.

If the user adds a new log (and as such new variables in the JSON) the upload of that JSON gets a 500 error. Should it be able to just add new variables with the incremental upload or should that only be possible with creating a new JSON?

To show what is happening I renamed a log, In this case I changed the table name from AllTempTags to TempTags, effectively creating a new log and deleting the old one, which as an effect had the variable names in the JSON changed from e.g. AllTempTagstemp to TempTagstemp.

Code: Select all

2023-10-12 11:21:07.137 Information Uploading => customlogsRecentdata.json from utils/customlogsRecentdata.json
2023-10-12 11:21:07.137 Information Upload File values: localfile: utils/customlogsRecentdata.json
2023-10-12 11:21:07.137 Information Upload File values: remotefile: customlogsRecentdata.json
2023-10-12 11:21:07.137 Information Incremental = True; filename = customlogsRecentdata.json; ext = .json; HoursInGraph = 72
2023-10-12 11:21:07.165 Information InetPhp: customlogsRecentdata.json: Response code = 500: InternalServerError
2023-10-12 11:21:07.165 Information InetPhp: customlogsRecentdata.json: Response text follows:
Unzipping data
Opening text file customlogsRecentdata.json for appending

2023-10-12 11:21:07.166 Information PHP UploadFile: Failed

I would expect the Upload incremental to simply add those new names to the json at the front (it does in a new json). The old ones will no longer be used but will still exist in the json until they will be fased out at the end of the length of the period in the json.

But apparently it does not work this way. Why not and could it be?

Posted: **Fri 13 Oct 2023 11:02 am**

If the "append" option is being used and no file is found or the file is zero length, then the script attempts to write just the new data to the filename provided using fopen(name, "w").

That will either create a new file if none exists or overwrite an existing file - assuming the correct permissions are in place.

Code: Select all

    // next check we have an existing file to append
    // if not then just write the new data to a dest file
    echo "Opening text file $name for appending\n";
    if (!file_exists($name) || filesize($name) === 0) {
        echo "No existing file to append data - $name\n";
        WriteFile($name, $data);
        exitCode(200);
    }

You should see the message "No existing file to append data - $name" in the returned response.

Posted: **Fri 13 Oct 2023 12:26 pm**

mcrossley wrote: ↑Fri 13 Oct 2023 11:02 am You should see the message "No existing file to append data - $name" in the returned response.

Well, obviously it does not do that, the returned message is above.

We may continue this in the new thread on the JSON mixup issue or wherever you wish.

Posted: **Fri 13 Oct 2023 1:40 pm**

I just tested it by letting MX run, it was uploading the tempdata.json file incrementally. I deleted the file on the server and got...

Code: Select all

2023-10-13 14:36:00.262 PHP[Int]: Uploading graph data file: tempdata.json
2023-10-13 14:36:00.263 PHP[Int]: Uploading to tempdata.json
2023-10-13 14:36:00.264 PHP[Int]: Sending via GET
2023-10-13 14:36:00.279 PHP[Int]: Upload to tempdata.json: Response code = 200: OK
2023-10-13 14:36:00.281 PHP[Int]: Upload to tempdata.json: Response text follows:
Opening text file tempdata.json for appending
No existing file to append data - tempdata.json
Writing text file tempdata.json
Write complete

The resulting file on the server just had data for the one increment in it.

So, the code does work for me.

Are you sure there are no permissions issues?

Posted: **Fri 13 Oct 2023 1:41 pm**

Perhaps have a look in the web server error log and see what is causing the error 500.

Posted: **Fri 13 Oct 2023 1:51 pm**

mcrossley wrote: ↑Fri 13 Oct 2023 1:40 pm I just tested it by letting MX run, it was uploading the tempdata.json file incrementally. I deleted the file on the server and got...
Code: Select all
2023-10-13 14:36:00.262 PHP[Int]: Uploading graph data file: tempdata.json
2023-10-13 14:36:00.263 PHP[Int]: Uploading to tempdata.json
2023-10-13 14:36:00.264 PHP[Int]: Sending via GET
2023-10-13 14:36:00.279 PHP[Int]: Upload to tempdata.json: Response code = 200: OK
2023-10-13 14:36:00.281 PHP[Int]: Upload to tempdata.json: Response text follows:
Opening text file tempdata.json for appending
No existing file to append data - tempdata.json
Writing text file tempdata.json
Write complete
The resulting file on the server just had data for the one increment in it.

So, the code does work for me.

Are you sure there are no permissions issues?

If I delete the file on the server all is OK for me as well, there are no permission issues.

I have no idea what caused the old data to persist in the middle of the new data but we will see. I created a new thread on this and currently data is correctly being added. At some point I will start modifying the definitions of the JSON and see what happens. I will try to that as controlled as possible and tell you than exactly what happened.

I just redefined the nr of hrs in the charts to 24 hrs. Effectively that trimmed the JSONs (all of them) to 24 hrs and that did reset my charts to 24 hrs which they remained when going back to 72 hrs which they are now. That is the current status, new data is added fine every 10 minutes, two data pairs at the time per json.

We'll take it from here in the other thread.

Posted: **Fri 13 Oct 2023 2:02 pm**

mcrossley wrote: ↑Fri 13 Oct 2023 1:41 pm Perhaps have a look in the web server error log and see what is causing the error 500.

The only error in the Apache errorlog I see at 2023-10-12 11:21 is:

Code: Select all

2023-10-12 12:21:04	Error		648686 [INFO] [2946050] [T9] [217.105.82.254:54110-2#APVH_meteo-wagenborgen.nl:443] Error while parsing request: (null)!

Not much to go on (for me).

Cumulus Support

Custom Logs

Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs

Re: Custom Logs