MDEV-31095 tpool - restrict threadpool concurrency during bufferpool load

Add threadpool functionality to restrict concurrency during "batch"
periods (where tasks are added in rapid succession).
This will throttle thread creation more agressively than usual, while
keeping performance at least on-par.

One of these cases is bufferpool load, where async read IOs are executed
without any throttling. There can be as much as 650K read IOs for
loading 10GB buffer pool.

Another one is recovery, where "fake read" IOs are executed.

Why there are more threads than we expect?
Worker threads are not be recognized as idle, until they return to the
standby list, and to return to that list, they need to acquire
mutex currently held in the submit_task(). In those cases, submit_task()
has no worker to wake, and would create threads until default concurrency
level (2*ncpus) is satisfied. Only after that throttling would happen.
This commit is contained in:
Vladislav Vaintroub 2023-04-25 12:00:06 +02:00 committed by Vladislav Vaintroub
commit e33e2fa949
4 changed files with 43 additions and 11 deletions

View file

@ -694,7 +694,9 @@ static void buf_dump_load_func(void *)
#ifdef WITH_WSREP
if (!get_wsrep_recovery()) {
#endif /* WITH_WSREP */
srv_thread_pool->set_concurrency(srv_n_read_io_threads);
buf_load();
srv_thread_pool->set_concurrency();
#ifdef WITH_WSREP
}
#endif /* WITH_WSREP */