top10Table - little problem

[Dador]

Hi,

I have had a rain gauge for a few months, but the top10Table does not count the periods with and without precipitation. I don't know where it comes from. Is it because a whole year has to pass? I am sure that during this period 10 records were collected for dry and wet periods.
There are no errors in the logs. The table is created and overwritten every day at midnight. What can you do about it?

Link to the table:
https://cmx.pogodarybnik.pl/top10.htm

[HansR]

Hi,

I have had a rain gauge for a few months, but the top10Table does not count the periods with and without precipitation. I don't know where it comes from. Is it because a whole year has to pass? I am sure that during this period 10 records were collected for dry and wet periods.
There are no errors in the logs. The table is created and overwritten every day at midnight. What can you do about it?

The dry and wet periods are determined from the dayfile. I really have no clue what is happening here and nobody else has had this issue. It is not necessary to have a full year pass, just some days with and without rain should already produce the list.

Can you post your dayfile so I can examine tomorrow? Thanks.

[Dador]

Can you post your dayfile so I can examine tomorrow? Thanks.

Maybe you'll notice something I'm not seeing .

[HansR]

I tried it on my test environment and the counting gets confused because of the long dry period at the start (the period between day when you started the station and the day you added the rain meter). To be honest, because the rain meter is seen as a primary sensor for CMX to function properly, I can't see this as a bug.

I advise to set the startdate ( parameter RecordsBeganDate in section general) to the date you added the rainmeter. Or even better: set it to 01/01/23 (Note the format is dd/MM/yy). That should solve the problem.

[Dador]

I advise to set the startdate ( parameter RecordsBeganDate in section general) to the date you added the rainmeter. Or even better: set it to 01/01/23 (Note the format is dd/MM/yy). That should solve the problem.

In fact, shifting the date to, for example, January 1, 2023 "removed" the problem. But after doing that, all records prior to that date are not included. That's not what I meant.
I've tried manually removing the values ​​for Max rainfall rate and Total rainfall, but then I get errors. Perhaps it would help if the program skipped blank fields, but I don't know if that would work.

Code: Select all

Error DayfileValue.SetValues: line is: 04/05/22,0,0,00:00,7.7,02:30,20.7,11:54,1017.0,00:56,1019.4,23:34,,00:00,,14.0,0.0,0,00:00,46,14:41,89,06:21,0.00,0.0,20.7,11:54,20.9,11:52,6.6,02:30,0.0,00:00,7.7,02:30,11.2,10:20,5.0,02:30,0,4.6,0.3,0,00:00,0.0,00:00,21.4,11:54,7.7,02:30,22.3,11:52,5104.3,,
Error DayfileValue.SetValues: Field maxRainRate is Empty

Well, for now, I'll leave it as is.

[HansR]

In fact, shifting the date to, for example, January 1, 2023 "removed" the problem. But after doing that, all records prior to that date are not included. That's not what I meant.
I've tried manually removing the values ​​for Max rainfall rate and Total rainfall, but then I get errors. Perhaps it would help if the program skipped blank fields, but I don't know if that would work.

Code: Select all

Error DayfileValue.SetValues: line is: 04/05/22,0,0,00:00,7.7,02:30,20.7,11:54,1017.0,00:56,1019.4,23:34,,00:00,,14.0,0.0,0,00:00,46,14:41,89,06:21,0.00,0.0,20.7,11:54,20.9,11:52,6.6,02:30,0.0,00:00,7.7,02:30,11.2,10:20,5.0,02:30,0,4.6,0.3,0,00:00,0.0,00:00,21.4,11:54,7.7,02:30,22.3,11:52,5104.3,,
Error DayfileValue.SetValues: Field maxRainRate is Empty

Well, for now, I'll leave it as is.

Skipping empty fields is difficult because there are no empty fields wrt rain. If there is no rain the value is 0 in the dayfile which is perfectly legal. The only problem which arises is that you get an extremely long dry period (184 days I believe it was) because no rain is registered. I cannot determine an absent rainmeter on the values of the rain in dayfile, that would not work in the Sahara

Setting the value at the day you have the rainmeter installed indeed skips all data before that date. The days before is only 184 days, you will get over it once you are a year further in time.

My argument to set it to the first of january is because having a startdate somewhere in the year will give strange bias for the statistical functions (charts) of CUtils. But it is all up to you. We all want long series but series start on the first day of the year just because we look at weather values per year, month, week and day. So it all is broken up on year limits. That's how life goes.

You can also see the days before the presence of the rainmeter (or before the first of january) as experimenting the new station.

Oh.... and the edits you made are not allowed: rainrate must have a value. Actually All values in dayfile must be NOT NULL (i.e. must have a value) so two comma's after each other are not allowed.

[HansR]

And one more remark: with your edits you suggest dayfile.txt could be manipulated with empty values to indicate absence of a sensor. If you really would want that it is CMX which must change the format and meaning of the fields in dayfile.txt. I have some doubts Mark would allow that and make those changes which would be required

[HansR]

OK. I'll be looking into the sorting again on why the calculation is disturbed.
No guarantee

[Dador]

Skipping empty fields is difficult because there are no empty fields wrt rain. If there is no rain the value is 0 in the dayfile which is perfectly legal. The only problem which arises is that you get an extremely long dry period (184 days I believe it was) because no rain is registered. I cannot determine an absent rainmeter on the values of the rain in dayfile, that would not work in the Sahara

Sure, I understand that empty fields are a programming problem.

Oh.... and the edits you made are not allowed: rainrate must have a value. Actually All values in dayfile must be NOT NULL (i.e. must have a value) so two comma's after each other are not allowed.

This is a bit of an inconvenience when saving weather data. Because "0" is always some value. Since we have "0" for rain, it means it didn't rain, "0" for wind means it wasn't blowing, and "0" for temperature means it was zero degrees that day or at that time. "0" is therefore problematic. For certain statistical situations, it would be much better if you could use empty fields instead of "0".

Ecowitt stations have separate sensors where everyone can work independently. Let's say that in July my temperature sensor breaks down. CMX will report the lack of a sensor in dayfile.txt as "0". Of course, later in the statistics it will come out that some day in July had an average air temperature of zero degrees, which is a complete lie and distortion. In such a situation, if we do not want to lose the remaining data for that day, it would be best to replace the faulty temperature with an empty field, which would not be taken into account for statistical calculations. This is my point of view.

OK. I'll be looking into the sorting again on why the calculation is disturbed.

Please look at my table today. I changed the date to 01/01/23. Despite over 100 measurement days, it has only 4 dry and wet periods, which is not enough in my opinion. It is as if the dry and wet periods are combined with the number of months with precipitation. By analogy, my problem will solve itself in August when I have 10 months of rainfall.

[HansR]

This is a bit of an inconvenience when saving weather data. Because "0" is always some value. Since we have "0" for rain, it means it didn't rain, "0" for wind means it wasn't blowing, and "0" for temperature means it was zero degrees that day or at that time. "0" is therefore problematic. For certain statistical situations, it would be much better if you could use empty fields instead of "0".

Ecowitt stations have separate sensors where everyone can work independently. Let's say that in July my temperature sensor breaks down. CMX will report the lack of a sensor in dayfile.txt as "0". Of course, later in the statistics it will come out that some day in July had an average air temperature of zero degrees, which is a complete lie and distortion. In such a situation, if we do not want to lose the remaining data for that day, it would be best to replace the faulty temperature with an empty field, which would not be taken into account for statistical calculations. This is my point of view.

I understand your point but CMX currently does not support null values in the database (dayfile and other txt files), so that cannot be used by CUtils as a distinction.

OK. I'll be looking into the sorting again on why the calculation is disturbed.

Please look at my table today. I changed the date to 01/01/23. Despite over 100 measurement days, it has only 4 dry and wet periods, which is not enough in my opinion. It is as if the dry and wet periods are combined with the number of months with precipitation. By analogy, my problem will solve itself in August when I have 10 months of rainfall.

Yes, I noticed something was weird so I took a deeper dive and found something so the problem has been promoted to 'Bug'. It probably entered the top10 code when I did a technical update of the module (version 6.16.0) Sorry for that.
Now I need to design the solution.

Your problem will not solve when you have 10 months of rainfall because the dry/wet periods go over the month boundaries it is a switch between wet (rain detected on a day) and dry (no rain detected) so after 20 days with alternating dry and wet weather you should already see 10 periods.

Anyway... no more discussion, I'll look into it.

[HansR]

Hi,

Attached you find a beta 1 for version 6.17.4 which I think has the wet and dry periods corrected. Could you try and feed back?
It is surprising that nobody ever saw this: it is the oldest module and the sorting algorithm never changed. Pretty bizar if I may say so.
So thanks for the observation

[EDIT:] but note: CUtils uses 0.2 mm and up as rain but ignores the 0.1 day measurements. That is different from your beteljuice dayfile table which apparently ignores up to 0.5 mm of rain. That has to do with setting. CUtils has no setting for this but uses 0.2 mm as a limit hardcoded (intentionally to make comparison between sites easier).

Regards,

[Dador]

Works well

[HansR]

Good. I will release 6.17.4 (or maybe 6.18.0) after the next release of CMX because there is one other thing - maybe more - I need to do.

[Dador]

Hi Hans,

in 6.17.3 I noticed the problem with the table again. Currently, only 6 out of 10 records are created. Something got messed up again.

[HansR]

I'll be back this evening and look at it.