number of processes?

[mcrossley]

Results from 08h10 this morning: more processes and errors in the log so contains probably more info

Thanks Hans, what is obvious is that the process counts increases occur at your interval processing. Errors that occur in your real-time uploads do not cause an increase.

The interval is logging two extra files with the same source name "web/HWActueelT.txt" (I'll add the destination name to the logging!).
But there appears to be only one response: "HWActueel.txt: Response code = 200: OK"
Is the other one uploaded to "MeteoWagenborgenData.txt"?

When the process count increases, after the interval update you do get a message...
"PHP[Int]: Upload process complete"

That message is only output when all the tasks for the upload have flagged back to CMX they have run to completion. So at the moment, I cannot see anything obviously wrong.

[mcrossley]

I have found a possible contention in token parser if two processes read attempt to read the same template file at the same time - it was opening the file in the default read/write mode.

I still do not see how that would cause a hung process, it should just log an exception in the output file and continue, but it needs fixing and then retesting to see if it is the cause.

[HansR]

Thanks Hans, what is obvious is that the process counts increases occur at your interval processing. Errors that occur in your real-time uploads do not cause an increase.

Yes, agree with that analysis.

The interval is logging two extra files with the same source name "web/HWActueelT.txt" (I'll add the destination name to the logging!).
But there appears to be only one response: "HWActueel.txt: Response code = 200: OK"
Is the other one uploaded to "MeteoWagenborgenData.txt"?

Yes the second one becomes "MeteoWagenborgenData.txt".
And that file is found on the server side so it does upload though I can't trace back if it did when it did not report back.

I have found a possible contention in token parser if two processes read attempt to read the same template file at the same time - it was opening the file in the default read/write mode.

I still do not see how that would cause a hung process, it should just log an exception in the output file and continue, but it needs fixing and then retesting to see if it is the cause.

OK. Can retest today, I'll be out of the loop for three days afterwards.

[mcrossley]

After I made a few tweaks, it runs on Windows fine, no increase in memory or thread count after a day of running lots of uploads at one-minute intervals. The exact same configuration on Linux - fail, the process count increases at random intervals.

For the technically inclined, the uploads are each run as a separate task, normally these tasks would each run on a thread allocated from a thread pool. Once the task completes, the thread is marked as available for use again by another task. What appears to be happening under Mono is that more threads get added to the pool as time goes on, which would imply they are not being freed correctly.

So, I have another build where I flag the tasks as "long running", that means they bypass the thread pool and are always* started as new threads. This seems to be working much better under Mono. A test instance has been running for 24 hours and has 31 processes. There have a couple of increments, but they could be due to other causes.

The build is actually going to be v3.25.0 as it contains some other changes around the FTP component. Hans is away for a few days, so is anybody else willing to test this build for me?

[*] long running is supposed to be a "hint", but it appears to be coded such that it is treated as gospel

[AndyKF650]

Hi Mark

I am quite happy to test this for you. I am running CMX/CUtils via mono on a RPI 4 edit also running PHP upload

For information my CMX processes are currently at 41 (used Hans' test script)

One other thing is that in my RPi /tmp folder I am seeing increasing numbers of zero byte files looking like this
CASESENSITIVETEST1a9b866d457e4eb089721051e92c7d05. Currently after two days this is up to 90 count

This maybe to do with the issues we are having.

[BigOkie]

After I made a few tweaks, it runs on Windows fine, no increase in memory or thread count after a day of running lots of uploads at one-minute intervals. The exact same configuration on Linux - fail, the process count increases at random intervals.

For the technically inclined, the uploads are each run as a separate task, normally these tasks would each run on a thread allocated from a thread pool. Once the task completes, the thread is marked as available for use again by another task. What appears to be happening under Mono is that more threads get added to the pool as time goes on, which would imply they are not being freed correctly.

So, I have another build where I flag the tasks as "long running", that means they bypass the thread pool and are always* started as new threads. This seems to be working much better under Mono. A test instance has been running for 24 hours and has 31 processes. There have a couple of increments, but they could be due to other causes.

The build is actually going to be v3.25.0 as it contains some other changes around the FTP component. Hans is away for a few days, so is anybody else willing to test this build for me?

[*] long running is supposed to be a "hint", but it appears to be coded such that it is treated as gospel

I don't usually pay much attention to htop when I run this; just updated to b3235 last night and I see 22 mono processes running currently. My previous version (I think it was 3223) I didn't even pay much attention to how many processes were running but it was greater than 1 for sure. Not sure if it was less than my current of 22 (which has been running now for about 15 hours). Attached is my htop dump.

2023-03-23_082230.png

[mcrossley]

22 is pretty normal, on the low side.

[broadstairs]

My Linux system has about 18 in total, but remember I do not use the default website so not many uploads.

Stuart

[Nossie]

Just to add ...

I've not tried to view the number of processes running because I run my system in quite a different way

However, I have had an event a few days or so ago where starting cumulus pegged mono to 150-200% which works out to be about 2 cores at 100% on an overclocked pi

This did cause the pi to shutdown once it hit 75C+ not sure if it's related but I restarted my docker stack and it happened again, so I tried a few different things and eventually it went away for a few days and came back. This included only running that one container which kept things manageable. Anyway the issue disappeared and has been fine for now but it might be related.

[philpugh]

Just to add another set of stats!

I run three instances of CumulusMX on a Pi 4 (32 bit OS).

It's been running for several days ( it auto restarts the three CMX instances on Monday 1 a.m.) and I have the following:

Main weather station (Ecowitt WH65 with additional temp (2 off) plus WH57 Lightning detector and CO2 monitor WH 45) Total 22 processes
Secondary system (Ecowitt WS68 with Temp/Humidity sensor also logging the same CO2 monitor WH 45) Total 20 processes
Experimental system (Ecowitt GW2001 no aditional sensors) Total 15 Processes.

I host my own website on the same RPi 4 and all transfers are done by copy. I use CUTILS to generate the websites. I don't use the default websites.

Running b 3235 Mono 6.12.0.182

[banaunnuh]

I have found a possible contention in token parser if two processes read attempt to read the same template file at the same time - it was opening the file in the default read/write mode.
pikashow

192.168.l.l

[mcrossley]

I think I have fixed parser contention issues in the next release.