JSON one value more than other JSON

[HansR]

I found that the tempdata.json has one value more than the humdata.json.

That poses some algorithmic issues when using both files with temperature as the lead filling other arrays, it breaks on the last value.
I did not check nor tried other json's but I assume the problem may occur with the others as well.

Is it possible to have exactly the same number of values in related json files?

[freddie]

The time data is also included in the JSON files. Perhaps you could line up your arrays using the time values?

I do take your point that the number of values should ideally be identical, but my suggestion could be used as a temporary measure until MX changes.

[HansR]

The time data is also included in the JSON files. Perhaps you could line up your arrays using the time values?

I do take your point that the number of values should ideally be identical, but my suggestion could be used as a temporary measure until MX changes.

Well, I thought about that but it is slightly more complex.
Yes, the idea is that all variables in the json have the same number of values.

In my ChartsCompiler that is used for more complex equations to fill a result array (as in example below where I use the length of Temperature to loop over the arrays of Temperature, Humidity and Windspeed).

Code: Select all

        for (var i = 0; i < Temperature.length; i++) {
            pwsFWI.push([Temperature[i][0], 6.112 * Math.exp(17.62 * Temperature[i][1] / (243.12 + Temperature[i][1])) *
                (1.0 - Humidity[i][1] / 100.0) * WindSpeed[i][1]
            ]);
        }

If the length of those are arrays is inherently different from each other, then yes I will have to adjust the generated code. But so far this is only a recent effect I can see and changing this code is quite complex, probably more so than synchronizing the JSON's length, so I will wait what Mark says

[mcrossley]

Different length I would expect to be rare, different start and end times less so. Can you check what you are seeing?

I'll have to check, but I think the data insert is asynchronous from the read. The generation of the graph JSON files takes a finite time, so more data could have been added (and trimmed) at some point during that process.

Solving that problem properly would require adding locking and queuing of processes, not trivial. However, I'll take a look later, there may be a simpler solution.

I'm laid up with COVID at the moment, so my enthusiasm to do anything much comes and goes...

[HansR]

I had this trouble for more than a day since yesterday but somewhere afterwards (until I write this reply), apparently it synchronized again and the relevant charts properly load again (not all charts have this complexity.

My initial analysis proved that it went wrong on the last element of Temperature, giving an undefined on the last element of Humidity (see my fragment above) so Humidity was one element short compared to Temperature. I did not check dates, just the nr of elements, should have probably.

If it happens again, I will analyse again on the date/time of first and last element (checking if it is shifted somewhere or just missing the last element or somewhere in the middle). As I have correct JSON's now , there is not much I can analyse.

I'm laid up with COVID at the moment, so my enthusiasm to do anything much comes and goes...

No hurry and no analysis from my side so don't bother I would say, just get better calmly and we'll pick it up again when relevant

[mcrossley]

I'm feeling a lot better today now I have finally got up

I have checked and afaics the data insert, and graph file generation are synchronous. So, the files should be in sync.

I think we need to dig deeper and look at the start/end time stamps in each file.

[freddie]

Glad to hear you're feeling better @mcrossley.

I generate my own JSON files based on what is read from my Monthly/Daily database tables, keyed on LogDateTime. That way you guarantee identical data sequences. I'm not sure what the status of the internal MX database is, but perhaps do the same from there if possible?

[mcrossley]

I'm assuming we are talking about the recent data graph files rather than the daily.

These are just a dump of all the required values in the internal recent data SQLite table. The table is appended to every minute and cleaned up every 10 minutes. All the values are held in the same table.

[HansR]

I'm assuming we are talking about the recent data graph files rather than the daily.

These are just a dump of all the required values in the internal recent data SQLite table. The table is appended to every minute and cleaned up every 10 minutes. All the values are held in the same table.

Yes, it occurred with the RECENT data graph JSON files.

So, if I understand you well this happens all synchronized, appended every minute and cleaned up every ten minutes. Then the question becomes: how was it possible they got out of sync?

And btw @freddie and @mcrossley: I do a lot of data handling but wrt to the JSON data I don't generate what CMX does already. I only generate a JSON for data CMX does not supply. So far, where those data were mixed, no problem has ever occurred. It was only today and yesterday I saw the problem. I must say: those are not charts I look at every day.

And indeed good to hear Covid has left you already

[mcrossley]

Still not up to doing much physically - a potter in the garden is enough at the mo - but I'm not feeling awful like the past few days now.

The inserts are done one the minute, and the graph files generated immediately afterwards in the same function call.

To be clear is the inconsistency between CMX generated files, or between CMX and CUtils generated files?

[HansR]

Still not up to doing much physically - a potter in the garden is enough at the mo - but I'm not feeling awful like the past few days now.
[...]
To be clear is the inconsistency between CMX generated files, or between CMX and CUtils generated files?

Good.

The inconsistency is between CMX data files. More specifically: so far I only have an issue with humidity and temperature.

I just had the problem again, I saved the JSONs before continuing so the evidence is fixed now.
I just loaded the json's in VS Code and determined that

• humdata.json has 4318 entries (for "hum")
• tempdata.json has 4319 entries (for "temp")
• It is the first record which is missing for hum. The first entry for "hum" is [1666295460000, 97], the first entry for "temp" is [1666295400000, 11.9],

So far I did not check other variables or files, I'll do on request. If you want the files yourself, let me know.
This is where the problem originates.

I must say I have had several network problems here (some leading to real issues), not enough bandwidth it seems but it would surprise me if that were the cause. Will have glassfiber in a year anyway

[mcrossley]

OK, 1 minute difference. I think I have the problem and a solution. I have emailed you...

[HansR]

Yes, thanks.
And the first impression is that the problem went away after activation of the fix.
So unless otherwise reported, let's say it is solved.

[mcrossley]

Let's leave it a few days to be sure.

[HansR]

Mark, it's working fine so far