Cumulus Support

HansR wrote: ↑Mon 20 Mar 2023 7:14 am Results from 08h10 this morning: more processes and errors in the log so contains probably more info

mcrossley wrote: ↑Mon 20 Mar 2023 11:01 am Thanks Hans, what is obvious is that the process counts increases occur at your interval processing. Errors that occur in your real-time uploads do not cause an increase.

mcrossley wrote: ↑Mon 20 Mar 2023 11:01 am The interval is logging two extra files with the same source name "web/HWActueelT.txt" (I'll add the destination name to the logging!).
But there appears to be only one response: "HWActueel.txt: Response code = 200: OK"
Is the other one uploaded to "MeteoWagenborgenData.txt"?

mcrossley wrote: ↑Mon 20 Mar 2023 11:29 am I have found a possible contention in token parser if two processes read attempt to read the same template file at the same time - it was opening the file in the default read/write mode.

I still do not see how that would cause a hung process, it should just log an exception in the output file and continue, but it needs fixing and then retesting to see if it is the cause.

mcrossley wrote: ↑Wed 22 Mar 2023 2:56 pm After I made a few tweaks, it runs on Windows fine, no increase in memory or thread count after a day of running lots of uploads at one-minute intervals. The exact same configuration on Linux - fail, the process count increases at random intervals.

For the technically inclined, the uploads are each run as a separate task, normally these tasks would each run on a thread allocated from a thread pool. Once the task completes, the thread is marked as available for use again by another task. What appears to be happening under Mono is that more threads get added to the pool as time goes on, which would imply they are not being freed correctly.

So, I have another build where I flag the tasks as "long running", that means they bypass the thread pool and are always* started as new threads. This seems to be working much better under Mono. A test instance has been running for 24 hours and has 31 processes. There have a couple of increments, but they could be due to other causes.

The build is actually going to be v3.25.0 as it contains some other changes around the FTP component. Hans is away for a few days, so is anybody else willing to test this build for me?

[*] long running is supposed to be a "hint", but it appears to be coded such that it is treated as gospel