2019-04-30 11:42:43 -07:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2009-12-03 10:59:42 -07:00
|
|
|
/*
|
|
|
|
* Common Block IO controller cgroup interface
|
|
|
|
*
|
|
|
|
* Based on ideas and code from CFQ, CFS and BFQ:
|
|
|
|
* Copyright (C) 2003 Jens Axboe <axboe@kernel.dk>
|
|
|
|
*
|
|
|
|
* Copyright (C) 2008 Fabio Checconi <fabio@gandalf.sssup.it>
|
|
|
|
* Paolo Valente <paolo.valente@unimore.it>
|
|
|
|
*
|
|
|
|
* Copyright (C) 2009 Vivek Goyal <vgoyal@redhat.com>
|
|
|
|
* Nauman Rafique <nauman@google.com>
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
*
|
|
|
|
* For policy-specific per-blkcg data:
|
|
|
|
* Copyright (C) 2015 Paolo Valente <paolo.valente@unimore.it>
|
|
|
|
* Arianna Avanzini <avanzini.arianna@gmail.com>
|
2009-12-03 10:59:42 -07:00
|
|
|
*/
|
|
|
|
#include <linux/ioprio.h>
|
2009-12-03 10:59:49 -07:00
|
|
|
#include <linux/kdev_t.h>
|
2009-12-04 08:36:41 -07:00
|
|
|
#include <linux/module.h>
|
2017-02-02 11:15:33 -07:00
|
|
|
#include <linux/sched/signal.h>
|
2009-12-07 01:29:39 -07:00
|
|
|
#include <linux/err.h>
|
2010-04-01 15:01:41 -07:00
|
|
|
#include <linux/blkdev.h>
|
2015-05-22 14:13:37 -07:00
|
|
|
#include <linux/backing-dev.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 01:04:11 -07:00
|
|
|
#include <linux/slab.h>
|
2012-03-05 14:15:00 -07:00
|
|
|
#include <linux/delay.h>
|
2012-03-19 15:10:56 -07:00
|
|
|
#include <linux/atomic.h>
|
2015-08-18 14:55:31 -07:00
|
|
|
#include <linux/ctype.h>
|
2022-02-09 11:20:45 -07:00
|
|
|
#include <linux/resume_user_mode.h>
|
2019-07-09 14:41:29 -07:00
|
|
|
#include <linux/psi.h>
|
2021-11-23 11:53:12 -07:00
|
|
|
#include <linux/part_stat.h>
|
2012-03-05 14:15:12 -07:00
|
|
|
#include "blk.h"
|
2022-02-11 03:11:49 -07:00
|
|
|
#include "blk-cgroup.h"
|
2021-06-17 17:44:44 -07:00
|
|
|
#include "blk-ioprio.h"
|
2021-10-05 08:11:56 -07:00
|
|
|
#include "blk-throttle.h"
|
2009-12-04 08:36:42 -07:00
|
|
|
|
2023-06-09 16:42:49 -07:00
|
|
|
static void __blkcg_rstat_flush(struct blkcg *blkcg, int cpu);
|
|
|
|
|
2015-07-09 13:39:47 -07:00
|
|
|
/*
|
|
|
|
* blkcg_pol_mutex protects blkcg_policy[] and policy [de]activation.
|
|
|
|
* blkcg_pol_register_mutex nests outside of it and synchronizes entire
|
|
|
|
* policy [un]register operations including cgroup file additions /
|
|
|
|
* removals. Putting cgroup file registration outside blkcg_pol_mutex
|
|
|
|
* allows grabbing it from cgroup callbacks.
|
|
|
|
*/
|
|
|
|
static DEFINE_MUTEX(blkcg_pol_register_mutex);
|
2012-04-13 13:11:26 -07:00
|
|
|
static DEFINE_MUTEX(blkcg_pol_mutex);
|
2012-03-05 14:15:13 -07:00
|
|
|
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
struct blkcg blkcg_root;
|
2012-04-16 13:57:25 -07:00
|
|
|
EXPORT_SYMBOL_GPL(blkcg_root);
|
2009-12-04 08:36:41 -07:00
|
|
|
|
2015-05-22 14:13:21 -07:00
|
|
|
struct cgroup_subsys_state * const blkcg_root_css = &blkcg_root.css;
|
2019-06-27 13:39:48 -07:00
|
|
|
EXPORT_SYMBOL_GPL(blkcg_root_css);
|
2015-05-22 14:13:21 -07:00
|
|
|
|
2012-04-16 13:57:25 -07:00
|
|
|
static struct blkcg_policy *blkcg_policy[BLKCG_MAX_POLS];
|
2012-03-05 14:15:04 -07:00
|
|
|
|
2015-07-09 13:39:49 -07:00
|
|
|
static LIST_HEAD(all_blkcgs); /* protected by blkcg_pol_mutex */
|
|
|
|
|
2019-07-16 07:58:31 -07:00
|
|
|
bool blkcg_debug_stats = false;
|
2018-07-03 08:14:52 -07:00
|
|
|
|
2023-06-09 16:42:49 -07:00
|
|
|
static DEFINE_RAW_SPINLOCK(blkg_stat_lock);
|
|
|
|
|
2021-07-06 18:56:49 -07:00
|
|
|
#define BLKG_DESTROY_BATCH_SIZE 64
|
|
|
|
|
2022-11-04 17:59:01 -07:00
|
|
|
/*
|
|
|
|
* Lockless lists for tracking IO stats update
|
|
|
|
*
|
|
|
|
* New IO stats are stored in the percpu iostat_cpu within blkcg_gq (blkg).
|
|
|
|
* There are multiple blkg's (one for each block device) attached to each
|
|
|
|
* blkcg. The rstat code keeps track of which cpu has IO stats updated,
|
|
|
|
* but it doesn't know which blkg has the updated stats. If there are many
|
|
|
|
* block devices in a system, the cost of iterating all the blkg's to flush
|
|
|
|
* out the IO stats can be high. To reduce such overhead, a set of percpu
|
|
|
|
* lockless lists (lhead) per blkcg are used to track the set of recently
|
|
|
|
* updated iostat_cpu's since the last flush. An iostat_cpu will be put
|
|
|
|
* onto the lockless list on the update side [blk_cgroup_bio_start()] if
|
|
|
|
* not there yet and then removed when being flushed [blkcg_rstat_flush()].
|
|
|
|
* References to blkg are gotten and then put back in the process to
|
|
|
|
* protect against blkg removal.
|
|
|
|
*
|
|
|
|
* Return: 0 if successful or -ENOMEM if allocation fails.
|
|
|
|
*/
|
|
|
|
static int init_blkcg_llists(struct blkcg *blkcg)
|
|
|
|
{
|
|
|
|
int cpu;
|
|
|
|
|
|
|
|
blkcg->lhead = alloc_percpu_gfp(struct llist_head, GFP_KERNEL);
|
|
|
|
if (!blkcg->lhead)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
for_each_possible_cpu(cpu)
|
|
|
|
init_llist_head(per_cpu_ptr(blkcg->lhead, cpu));
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2022-04-19 21:27:20 -07:00
|
|
|
/**
|
|
|
|
* blkcg_css - find the current css
|
|
|
|
*
|
|
|
|
* Find the css associated with either the kthread or the current task.
|
|
|
|
* This may return a dying css, so it is up to the caller to use tryget logic
|
|
|
|
* to confirm it is alive and well.
|
|
|
|
*/
|
|
|
|
static struct cgroup_subsys_state *blkcg_css(void)
|
|
|
|
{
|
|
|
|
struct cgroup_subsys_state *css;
|
|
|
|
|
|
|
|
css = kthread_blkcg();
|
|
|
|
if (css)
|
|
|
|
return css;
|
|
|
|
return task_css(current, io_cgrp_id);
|
|
|
|
}
|
|
|
|
|
2012-04-13 13:11:33 -07:00
|
|
|
static bool blkcg_policy_enabled(struct request_queue *q,
|
2012-04-16 13:57:25 -07:00
|
|
|
const struct blkcg_policy *pol)
|
2012-04-13 13:11:33 -07:00
|
|
|
{
|
|
|
|
return pol && test_bit(pol->plid, q->blkcg_pols);
|
|
|
|
}
|
|
|
|
|
block: avoid calling blkg_free() in atomic context
blkg_free() can currently be called in atomic context, either spin lock is
held, or run in rcu callback. Meantime either request queue's release
handler or ->pd_free_fn can sleep.
Fix the issue by scheduling a work function for freeing blkcg_gq the
instance.
[ 148.553894] BUG: sleeping function called from invalid context at block/blk-sysfs.c:767
[ 148.557381] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 0, name: swapper/13
[ 148.560741] preempt_count: 101, expected: 0
[ 148.562577] RCU nest depth: 0, expected: 0
[ 148.564379] 1 lock held by swapper/13/0:
[ 148.566127] #0: ffffffff82615f80 (rcu_callback){....}-{0:0}, at: rcu_lock_acquire+0x0/0x1b
[ 148.569640] Preemption disabled at:
[ 148.569642] [<ffffffff8123f9c3>] ___slab_alloc+0x554/0x661
[ 148.573559] CPU: 13 PID: 0 Comm: swapper/13 Kdump: loaded Not tainted 5.17.0_up+ #110
[ 148.576834] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-1.fc33 04/01/2014
[ 148.579768] Call Trace:
[ 148.580567] <IRQ>
[ 148.581262] dump_stack_lvl+0x56/0x7c
[ 148.582367] ? ___slab_alloc+0x554/0x661
[ 148.583526] __might_resched+0x1af/0x1c8
[ 148.584678] blk_release_queue+0x24/0x109
[ 148.585861] kobject_cleanup+0xc9/0xfe
[ 148.586979] blkg_free+0x46/0x63
[ 148.587962] rcu_do_batch+0x1c5/0x3db
[ 148.589057] rcu_core+0x14a/0x184
[ 148.590065] __do_softirq+0x14d/0x2c7
[ 148.591167] __irq_exit_rcu+0x7a/0xd4
[ 148.592264] sysvec_apic_timer_interrupt+0x82/0xa5
[ 148.593649] </IRQ>
[ 148.594354] <TASK>
[ 148.595058] asm_sysvec_apic_timer_interrupt+0x12/0x20
Cc: Tejun Heo <tj@kernel.org>
Fixes: 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt")
Reported-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/linux-block/20220322093322.GA27283@lst.de/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/20220323011308.2010380-1-ming.lei@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-03-22 18:13:08 -07:00
|
|
|
static void blkg_free_workfn(struct work_struct *work)
|
2012-03-05 14:15:14 -07:00
|
|
|
{
|
block: avoid calling blkg_free() in atomic context
blkg_free() can currently be called in atomic context, either spin lock is
held, or run in rcu callback. Meantime either request queue's release
handler or ->pd_free_fn can sleep.
Fix the issue by scheduling a work function for freeing blkcg_gq the
instance.
[ 148.553894] BUG: sleeping function called from invalid context at block/blk-sysfs.c:767
[ 148.557381] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 0, name: swapper/13
[ 148.560741] preempt_count: 101, expected: 0
[ 148.562577] RCU nest depth: 0, expected: 0
[ 148.564379] 1 lock held by swapper/13/0:
[ 148.566127] #0: ffffffff82615f80 (rcu_callback){....}-{0:0}, at: rcu_lock_acquire+0x0/0x1b
[ 148.569640] Preemption disabled at:
[ 148.569642] [<ffffffff8123f9c3>] ___slab_alloc+0x554/0x661
[ 148.573559] CPU: 13 PID: 0 Comm: swapper/13 Kdump: loaded Not tainted 5.17.0_up+ #110
[ 148.576834] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-1.fc33 04/01/2014
[ 148.579768] Call Trace:
[ 148.580567] <IRQ>
[ 148.581262] dump_stack_lvl+0x56/0x7c
[ 148.582367] ? ___slab_alloc+0x554/0x661
[ 148.583526] __might_resched+0x1af/0x1c8
[ 148.584678] blk_release_queue+0x24/0x109
[ 148.585861] kobject_cleanup+0xc9/0xfe
[ 148.586979] blkg_free+0x46/0x63
[ 148.587962] rcu_do_batch+0x1c5/0x3db
[ 148.589057] rcu_core+0x14a/0x184
[ 148.590065] __do_softirq+0x14d/0x2c7
[ 148.591167] __irq_exit_rcu+0x7a/0xd4
[ 148.592264] sysvec_apic_timer_interrupt+0x82/0xa5
[ 148.593649] </IRQ>
[ 148.594354] <TASK>
[ 148.595058] asm_sysvec_apic_timer_interrupt+0x12/0x20
Cc: Tejun Heo <tj@kernel.org>
Fixes: 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt")
Reported-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/linux-block/20220322093322.GA27283@lst.de/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/20220323011308.2010380-1-ming.lei@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-03-22 18:13:08 -07:00
|
|
|
struct blkcg_gq *blkg = container_of(work, struct blkcg_gq,
|
|
|
|
free_work);
|
2023-02-14 11:33:08 -07:00
|
|
|
struct request_queue *q = blkg->q;
|
2012-03-05 14:15:20 -07:00
|
|
|
int i;
|
2012-03-05 14:15:16 -07:00
|
|
|
|
2023-01-19 04:03:50 -07:00
|
|
|
/*
|
|
|
|
* pd_free_fn() can also be called from blkcg_deactivate_policy(),
|
|
|
|
* in order to make sure pd_free_fn() is called in order, the deletion
|
2023-02-14 11:33:04 -07:00
|
|
|
* of the list blkg->q_node is delayed to here from blkg_destroy(), and
|
2023-01-19 04:03:50 -07:00
|
|
|
* blkcg_mutex is used to synchronize blkg_free_workfn() and
|
|
|
|
* blkcg_deactivate_policy().
|
|
|
|
*/
|
2023-02-14 11:33:08 -07:00
|
|
|
mutex_lock(&q->blkcg_mutex);
|
2013-05-14 13:52:31 -07:00
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++)
|
2015-08-18 14:55:11 -07:00
|
|
|
if (blkg->pd[i])
|
|
|
|
blkcg_policy[i]->pd_free_fn(blkg->pd[i]);
|
2023-01-19 04:03:48 -07:00
|
|
|
if (blkg->parent)
|
|
|
|
blkg_put(blkg->parent);
|
2023-08-17 07:17:51 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
2023-02-14 11:33:04 -07:00
|
|
|
list_del_init(&blkg->q_node);
|
2023-08-17 07:17:51 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2023-02-14 11:33:08 -07:00
|
|
|
mutex_unlock(&q->blkcg_mutex);
|
2012-03-05 14:15:20 -07:00
|
|
|
|
2023-02-14 11:33:08 -07:00
|
|
|
blk_put_queue(q);
|
2019-11-07 12:18:03 -07:00
|
|
|
free_percpu(blkg->iostat_cpu);
|
2019-06-13 15:30:39 -07:00
|
|
|
percpu_ref_exit(&blkg->refcnt);
|
2012-03-05 14:15:16 -07:00
|
|
|
kfree(blkg);
|
2012-03-05 14:15:14 -07:00
|
|
|
}
|
|
|
|
|
block: avoid calling blkg_free() in atomic context
blkg_free() can currently be called in atomic context, either spin lock is
held, or run in rcu callback. Meantime either request queue's release
handler or ->pd_free_fn can sleep.
Fix the issue by scheduling a work function for freeing blkcg_gq the
instance.
[ 148.553894] BUG: sleeping function called from invalid context at block/blk-sysfs.c:767
[ 148.557381] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 0, name: swapper/13
[ 148.560741] preempt_count: 101, expected: 0
[ 148.562577] RCU nest depth: 0, expected: 0
[ 148.564379] 1 lock held by swapper/13/0:
[ 148.566127] #0: ffffffff82615f80 (rcu_callback){....}-{0:0}, at: rcu_lock_acquire+0x0/0x1b
[ 148.569640] Preemption disabled at:
[ 148.569642] [<ffffffff8123f9c3>] ___slab_alloc+0x554/0x661
[ 148.573559] CPU: 13 PID: 0 Comm: swapper/13 Kdump: loaded Not tainted 5.17.0_up+ #110
[ 148.576834] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-1.fc33 04/01/2014
[ 148.579768] Call Trace:
[ 148.580567] <IRQ>
[ 148.581262] dump_stack_lvl+0x56/0x7c
[ 148.582367] ? ___slab_alloc+0x554/0x661
[ 148.583526] __might_resched+0x1af/0x1c8
[ 148.584678] blk_release_queue+0x24/0x109
[ 148.585861] kobject_cleanup+0xc9/0xfe
[ 148.586979] blkg_free+0x46/0x63
[ 148.587962] rcu_do_batch+0x1c5/0x3db
[ 148.589057] rcu_core+0x14a/0x184
[ 148.590065] __do_softirq+0x14d/0x2c7
[ 148.591167] __irq_exit_rcu+0x7a/0xd4
[ 148.592264] sysvec_apic_timer_interrupt+0x82/0xa5
[ 148.593649] </IRQ>
[ 148.594354] <TASK>
[ 148.595058] asm_sysvec_apic_timer_interrupt+0x12/0x20
Cc: Tejun Heo <tj@kernel.org>
Fixes: 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt")
Reported-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/linux-block/20220322093322.GA27283@lst.de/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/20220323011308.2010380-1-ming.lei@redhat.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-03-22 18:13:08 -07:00
|
|
|
/**
|
|
|
|
* blkg_free - free a blkg
|
|
|
|
* @blkg: blkg to free
|
|
|
|
*
|
|
|
|
* Free @blkg which may be partially allocated.
|
|
|
|
*/
|
|
|
|
static void blkg_free(struct blkcg_gq *blkg)
|
|
|
|
{
|
|
|
|
if (!blkg)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Both ->pd_free_fn() and request queue's release handler may
|
|
|
|
* sleep, so free us by scheduling one work func
|
|
|
|
*/
|
|
|
|
INIT_WORK(&blkg->free_work, blkg_free_workfn);
|
|
|
|
schedule_work(&blkg->free_work);
|
|
|
|
}
|
|
|
|
|
2018-12-05 10:10:38 -07:00
|
|
|
static void __blkg_release(struct rcu_head *rcu)
|
|
|
|
{
|
|
|
|
struct blkcg_gq *blkg = container_of(rcu, struct blkcg_gq, rcu_head);
|
2023-06-09 16:42:49 -07:00
|
|
|
struct blkcg *blkcg = blkg->blkcg;
|
|
|
|
int cpu;
|
2018-12-05 10:10:38 -07:00
|
|
|
|
2023-03-26 17:49:53 -07:00
|
|
|
#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
|
2019-06-27 13:39:52 -07:00
|
|
|
WARN_ON(!bio_list_empty(&blkg->async_bios));
|
2023-03-26 17:49:53 -07:00
|
|
|
#endif
|
2023-06-09 16:42:49 -07:00
|
|
|
/*
|
|
|
|
* Flush all the non-empty percpu lockless lists before releasing
|
|
|
|
* us, given these stat belongs to us.
|
|
|
|
*
|
|
|
|
* blkg_stat_lock is for serializing blkg stat update
|
|
|
|
*/
|
|
|
|
for_each_possible_cpu(cpu)
|
|
|
|
__blkcg_rstat_flush(blkcg, cpu);
|
2019-06-27 13:39:52 -07:00
|
|
|
|
2018-12-05 10:10:38 -07:00
|
|
|
/* release the blkcg and parent blkg refs this blkg has been holding */
|
|
|
|
css_put(&blkg->blkcg->css);
|
|
|
|
blkg_free(blkg);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* A group is RCU protected, but having an rcu lock does not mean that one
|
|
|
|
* can access all the fields of blkg and assume these are valid. For
|
|
|
|
* example, don't try to follow throtl_data and request queue links.
|
|
|
|
*
|
|
|
|
* Having a reference to blkg under an rcu allows accesses to only values
|
|
|
|
* local to groups like group stats and group rate limits.
|
|
|
|
*/
|
|
|
|
static void blkg_release(struct percpu_ref *ref)
|
|
|
|
{
|
|
|
|
struct blkcg_gq *blkg = container_of(ref, struct blkcg_gq, refcnt);
|
|
|
|
|
|
|
|
call_rcu(&blkg->rcu_head, __blkg_release);
|
|
|
|
}
|
|
|
|
|
2023-03-26 17:49:53 -07:00
|
|
|
#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
|
|
|
|
static struct workqueue_struct *blkcg_punt_bio_wq;
|
|
|
|
|
2019-06-27 13:39:52 -07:00
|
|
|
static void blkg_async_bio_workfn(struct work_struct *work)
|
|
|
|
{
|
|
|
|
struct blkcg_gq *blkg = container_of(work, struct blkcg_gq,
|
|
|
|
async_bio_work);
|
|
|
|
struct bio_list bios = BIO_EMPTY_LIST;
|
|
|
|
struct bio *bio;
|
blkcg: add plugging support for punt bio
The test and the explaination of the patch as bellow.
Before test we added more debug code in blkg_async_bio_workfn():
int count = 0
if (bios.head && bios.head->bi_next) {
need_plug = true;
blk_start_plug(&plug);
}
while ((bio = bio_list_pop(&bios))) {
/*io_punt is a sysctl user interface to control the print*/
if(io_punt) {
printk("[%s:%d] bio start,size:%llu,%d count=%d plug?%d\n",
current->comm, current->pid, bio->bi_iter.bi_sector,
(bio->bi_iter.bi_size)>>9, count++, need_plug);
}
submit_bio(bio);
}
if (need_plug)
blk_finish_plug(&plug);
Steps that need to be set to trigger *PUNT* io before testing:
mount -t btrfs -o compress=lzo /dev/sda6 /btrfs
mount -t cgroup2 nodev /cgroup2
mkdir /cgroup2/cg3
echo "+io" > /cgroup2/cgroup.subtree_control
echo "8:0 wbps=1048576000" > /cgroup2/cg3/io.max #1000M/s
echo $$ > /cgroup2/cg3/cgroup.procs
Then use dd command to test btrfs PUNT io in current shell:
dd if=/dev/zero of=/btrfs/file bs=64K count=100000
Test hardware environment as below:
[root@localhost btrfs]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
With above debug code, test command and test environment, I did the
tests under 3 different system loads, which are triggered by stress:
1, Run 64 threads by command "stress -c 64 &"
[53615.975974] [kworker/u66:18:1490] bio start,size:45583056,8 count=0 plug?1
[53615.975980] [kworker/u66:18:1490] bio start,size:45583064,8 count=1 plug?1
[53615.975984] [kworker/u66:18:1490] bio start,size:45583072,8 count=2 plug?1
[53615.975987] [kworker/u66:18:1490] bio start,size:45583080,8 count=3 plug?1
[53615.975990] [kworker/u66:18:1490] bio start,size:45583088,8 count=4 plug?1
[53615.975993] [kworker/u66:18:1490] bio start,size:45583096,8 count=5 plug?1
... ...
[53615.977041] [kworker/u66:18:1490] bio start,size:45585480,8 count=303 plug?1
[53615.977044] [kworker/u66:18:1490] bio start,size:45585488,8 count=304 plug?1
[53615.977047] [kworker/u66:18:1490] bio start,size:45585496,8 count=305 plug?1
[53615.977050] [kworker/u66:18:1490] bio start,size:45585504,8 count=306 plug?1
[53615.977053] [kworker/u66:18:1490] bio start,size:45585512,8 count=307 plug?1
[53615.977056] [kworker/u66:18:1490] bio start,size:45585520,8 count=308 plug?1
[53615.977058] [kworker/u66:18:1490] bio start,size:45585528,8 count=309 plug?1
2, Run 32 threads by command "stress -c 32 &"
[50586.290521] [kworker/u66:6:32351] bio start,size:45806496,8 count=0 plug?1
[50586.290526] [kworker/u66:6:32351] bio start,size:45806504,8 count=1 plug?1
[50586.290529] [kworker/u66:6:32351] bio start,size:45806512,8 count=2 plug?1
[50586.290531] [kworker/u66:6:32351] bio start,size:45806520,8 count=3 plug?1
[50586.290533] [kworker/u66:6:32351] bio start,size:45806528,8 count=4 plug?1
[50586.290535] [kworker/u66:6:32351] bio start,size:45806536,8 count=5 plug?1
... ...
[50586.299640] [kworker/u66:5:32350] bio start,size:45808576,8 count=252 plug?1
[50586.299643] [kworker/u66:5:32350] bio start,size:45808584,8 count=253 plug?1
[50586.299646] [kworker/u66:5:32350] bio start,size:45808592,8 count=254 plug?1
[50586.299649] [kworker/u66:5:32350] bio start,size:45808600,8 count=255 plug?1
[50586.299652] [kworker/u66:5:32350] bio start,size:45808608,8 count=256 plug?1
[50586.299663] [kworker/u66:5:32350] bio start,size:45808616,8 count=257 plug?1
[50586.299665] [kworker/u66:5:32350] bio start,size:45808624,8 count=258 plug?1
[50586.299668] [kworker/u66:5:32350] bio start,size:45808632,8 count=259 plug?1
3, Don't run thread by stress
[50861.355246] [kworker/u66:19:32376] bio start,size:13544504,8 count=0 plug?0
[50861.355288] [kworker/u66:19:32376] bio start,size:13544512,8 count=0 plug?0
[50861.355322] [kworker/u66:19:32376] bio start,size:13544520,8 count=0 plug?0
[50861.355353] [kworker/u66:19:32376] bio start,size:13544528,8 count=0 plug?0
[50861.355392] [kworker/u66:19:32376] bio start,size:13544536,8 count=0 plug?0
[50861.355431] [kworker/u66:19:32376] bio start,size:13544544,8 count=0 plug?0
[50861.355468] [kworker/u66:19:32376] bio start,size:13544552,8 count=0 plug?0
[50861.355499] [kworker/u66:19:32376] bio start,size:13544560,8 count=0 plug?0
[50861.355532] [kworker/u66:19:32376] bio start,size:13544568,8 count=0 plug?0
[50861.355575] [kworker/u66:19:32376] bio start,size:13544576,8 count=0 plug?0
[50861.355618] [kworker/u66:19:32376] bio start,size:13544584,8 count=0 plug?0
[50861.355659] [kworker/u66:19:32376] bio start,size:13544592,8 count=0 plug?0
[50861.355740] [kworker/u66:0:32346] bio start,size:13544600,8 count=0 plug?1
[50861.355748] [kworker/u66:0:32346] bio start,size:13544608,8 count=1 plug?1
[50861.355962] [kworker/u66:2:32347] bio start,size:13544616,8 count=0 plug?0
[50861.356272] [kworker/u66:7:31962] bio start,size:13544624,8 count=0 plug?0
[50861.356446] [kworker/u66:7:31962] bio start,size:13544632,8 count=0 plug?0
[50861.356567] [kworker/u66:7:31962] bio start,size:13544640,8 count=0 plug?0
[50861.356707] [kworker/u66:19:32376] bio start,size:13544648,8 count=0 plug?0
[50861.356748] [kworker/u66:15:32355] bio start,size:13544656,8 count=0 plug?0
[50861.356825] [kworker/u66:17:31970] bio start,size:13544664,8 count=0 plug?0
Analysis of above 3 test results with different system load:
>From above test, we can see more and more continuous bios can be plugged
with system load increasing. When run "stress -c 64 &", 310 continuous
bios are plugged; When run "stress -c 32 &", 260 continuous bios are
plugged; When don't run stress, at most only 2 continuous bios are
plugged, in most cases, bio_list only contains one single bio.
How to explain above phenomenon:
We know, in submit_bio(), if the bio is a REQ_CGROUP_PUNT io, it will
queue a work to workqueue blkcg_punt_bio_wq. But when the workqueue is
scheduled, it depends on the system load. When system load is low, the
workqueue will be quickly scheduled, and the bio in bio_list will be
quickly processed in blkg_async_bio_workfn(), so there is less chance
that the same io submit thread can add multiple continuous bios to
bio_list before workqueue is scheduled to run. The analysis aligned with
above test "3".
When system load is high, there is some delay before the workqueue can
be scheduled to run, the higher the system load the greater the delay.
So there is more chance that the same io submit thread can add multiple
continuous bios to bio_list. Then when the workqueue is scheduled to run,
there are more continuous bios in bio_list, which will be processed in
blkg_async_bio_workfn(). The analysis aligned with above test "1" and "2".
According to test, we can get io performance improved with the patch,
especially when system load is higher. Another optimazition is to use
the plug only when bio_list contains at least 2 bios.
Signed-off-by: Xianting Tian <tian.xianting@h3c.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-09 23:15:06 -07:00
|
|
|
struct blk_plug plug;
|
|
|
|
bool need_plug = false;
|
2019-06-27 13:39:52 -07:00
|
|
|
|
|
|
|
/* as long as there are pending bios, @blkg can't go away */
|
2023-03-26 17:49:52 -07:00
|
|
|
spin_lock(&blkg->async_bio_lock);
|
2024-03-28 01:41:45 -07:00
|
|
|
bio_list_merge_init(&bios, &blkg->async_bios);
|
2023-03-26 17:49:52 -07:00
|
|
|
spin_unlock(&blkg->async_bio_lock);
|
2019-06-27 13:39:52 -07:00
|
|
|
|
blkcg: add plugging support for punt bio
The test and the explaination of the patch as bellow.
Before test we added more debug code in blkg_async_bio_workfn():
int count = 0
if (bios.head && bios.head->bi_next) {
need_plug = true;
blk_start_plug(&plug);
}
while ((bio = bio_list_pop(&bios))) {
/*io_punt is a sysctl user interface to control the print*/
if(io_punt) {
printk("[%s:%d] bio start,size:%llu,%d count=%d plug?%d\n",
current->comm, current->pid, bio->bi_iter.bi_sector,
(bio->bi_iter.bi_size)>>9, count++, need_plug);
}
submit_bio(bio);
}
if (need_plug)
blk_finish_plug(&plug);
Steps that need to be set to trigger *PUNT* io before testing:
mount -t btrfs -o compress=lzo /dev/sda6 /btrfs
mount -t cgroup2 nodev /cgroup2
mkdir /cgroup2/cg3
echo "+io" > /cgroup2/cgroup.subtree_control
echo "8:0 wbps=1048576000" > /cgroup2/cg3/io.max #1000M/s
echo $$ > /cgroup2/cg3/cgroup.procs
Then use dd command to test btrfs PUNT io in current shell:
dd if=/dev/zero of=/btrfs/file bs=64K count=100000
Test hardware environment as below:
[root@localhost btrfs]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
With above debug code, test command and test environment, I did the
tests under 3 different system loads, which are triggered by stress:
1, Run 64 threads by command "stress -c 64 &"
[53615.975974] [kworker/u66:18:1490] bio start,size:45583056,8 count=0 plug?1
[53615.975980] [kworker/u66:18:1490] bio start,size:45583064,8 count=1 plug?1
[53615.975984] [kworker/u66:18:1490] bio start,size:45583072,8 count=2 plug?1
[53615.975987] [kworker/u66:18:1490] bio start,size:45583080,8 count=3 plug?1
[53615.975990] [kworker/u66:18:1490] bio start,size:45583088,8 count=4 plug?1
[53615.975993] [kworker/u66:18:1490] bio start,size:45583096,8 count=5 plug?1
... ...
[53615.977041] [kworker/u66:18:1490] bio start,size:45585480,8 count=303 plug?1
[53615.977044] [kworker/u66:18:1490] bio start,size:45585488,8 count=304 plug?1
[53615.977047] [kworker/u66:18:1490] bio start,size:45585496,8 count=305 plug?1
[53615.977050] [kworker/u66:18:1490] bio start,size:45585504,8 count=306 plug?1
[53615.977053] [kworker/u66:18:1490] bio start,size:45585512,8 count=307 plug?1
[53615.977056] [kworker/u66:18:1490] bio start,size:45585520,8 count=308 plug?1
[53615.977058] [kworker/u66:18:1490] bio start,size:45585528,8 count=309 plug?1
2, Run 32 threads by command "stress -c 32 &"
[50586.290521] [kworker/u66:6:32351] bio start,size:45806496,8 count=0 plug?1
[50586.290526] [kworker/u66:6:32351] bio start,size:45806504,8 count=1 plug?1
[50586.290529] [kworker/u66:6:32351] bio start,size:45806512,8 count=2 plug?1
[50586.290531] [kworker/u66:6:32351] bio start,size:45806520,8 count=3 plug?1
[50586.290533] [kworker/u66:6:32351] bio start,size:45806528,8 count=4 plug?1
[50586.290535] [kworker/u66:6:32351] bio start,size:45806536,8 count=5 plug?1
... ...
[50586.299640] [kworker/u66:5:32350] bio start,size:45808576,8 count=252 plug?1
[50586.299643] [kworker/u66:5:32350] bio start,size:45808584,8 count=253 plug?1
[50586.299646] [kworker/u66:5:32350] bio start,size:45808592,8 count=254 plug?1
[50586.299649] [kworker/u66:5:32350] bio start,size:45808600,8 count=255 plug?1
[50586.299652] [kworker/u66:5:32350] bio start,size:45808608,8 count=256 plug?1
[50586.299663] [kworker/u66:5:32350] bio start,size:45808616,8 count=257 plug?1
[50586.299665] [kworker/u66:5:32350] bio start,size:45808624,8 count=258 plug?1
[50586.299668] [kworker/u66:5:32350] bio start,size:45808632,8 count=259 plug?1
3, Don't run thread by stress
[50861.355246] [kworker/u66:19:32376] bio start,size:13544504,8 count=0 plug?0
[50861.355288] [kworker/u66:19:32376] bio start,size:13544512,8 count=0 plug?0
[50861.355322] [kworker/u66:19:32376] bio start,size:13544520,8 count=0 plug?0
[50861.355353] [kworker/u66:19:32376] bio start,size:13544528,8 count=0 plug?0
[50861.355392] [kworker/u66:19:32376] bio start,size:13544536,8 count=0 plug?0
[50861.355431] [kworker/u66:19:32376] bio start,size:13544544,8 count=0 plug?0
[50861.355468] [kworker/u66:19:32376] bio start,size:13544552,8 count=0 plug?0
[50861.355499] [kworker/u66:19:32376] bio start,size:13544560,8 count=0 plug?0
[50861.355532] [kworker/u66:19:32376] bio start,size:13544568,8 count=0 plug?0
[50861.355575] [kworker/u66:19:32376] bio start,size:13544576,8 count=0 plug?0
[50861.355618] [kworker/u66:19:32376] bio start,size:13544584,8 count=0 plug?0
[50861.355659] [kworker/u66:19:32376] bio start,size:13544592,8 count=0 plug?0
[50861.355740] [kworker/u66:0:32346] bio start,size:13544600,8 count=0 plug?1
[50861.355748] [kworker/u66:0:32346] bio start,size:13544608,8 count=1 plug?1
[50861.355962] [kworker/u66:2:32347] bio start,size:13544616,8 count=0 plug?0
[50861.356272] [kworker/u66:7:31962] bio start,size:13544624,8 count=0 plug?0
[50861.356446] [kworker/u66:7:31962] bio start,size:13544632,8 count=0 plug?0
[50861.356567] [kworker/u66:7:31962] bio start,size:13544640,8 count=0 plug?0
[50861.356707] [kworker/u66:19:32376] bio start,size:13544648,8 count=0 plug?0
[50861.356748] [kworker/u66:15:32355] bio start,size:13544656,8 count=0 plug?0
[50861.356825] [kworker/u66:17:31970] bio start,size:13544664,8 count=0 plug?0
Analysis of above 3 test results with different system load:
>From above test, we can see more and more continuous bios can be plugged
with system load increasing. When run "stress -c 64 &", 310 continuous
bios are plugged; When run "stress -c 32 &", 260 continuous bios are
plugged; When don't run stress, at most only 2 continuous bios are
plugged, in most cases, bio_list only contains one single bio.
How to explain above phenomenon:
We know, in submit_bio(), if the bio is a REQ_CGROUP_PUNT io, it will
queue a work to workqueue blkcg_punt_bio_wq. But when the workqueue is
scheduled, it depends on the system load. When system load is low, the
workqueue will be quickly scheduled, and the bio in bio_list will be
quickly processed in blkg_async_bio_workfn(), so there is less chance
that the same io submit thread can add multiple continuous bios to
bio_list before workqueue is scheduled to run. The analysis aligned with
above test "3".
When system load is high, there is some delay before the workqueue can
be scheduled to run, the higher the system load the greater the delay.
So there is more chance that the same io submit thread can add multiple
continuous bios to bio_list. Then when the workqueue is scheduled to run,
there are more continuous bios in bio_list, which will be processed in
blkg_async_bio_workfn(). The analysis aligned with above test "1" and "2".
According to test, we can get io performance improved with the patch,
especially when system load is higher. Another optimazition is to use
the plug only when bio_list contains at least 2 bios.
Signed-off-by: Xianting Tian <tian.xianting@h3c.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-09 23:15:06 -07:00
|
|
|
/* start plug only when bio_list contains at least 2 bios */
|
|
|
|
if (bios.head && bios.head->bi_next) {
|
|
|
|
need_plug = true;
|
|
|
|
blk_start_plug(&plug);
|
|
|
|
}
|
2019-06-27 13:39:52 -07:00
|
|
|
while ((bio = bio_list_pop(&bios)))
|
|
|
|
submit_bio(bio);
|
blkcg: add plugging support for punt bio
The test and the explaination of the patch as bellow.
Before test we added more debug code in blkg_async_bio_workfn():
int count = 0
if (bios.head && bios.head->bi_next) {
need_plug = true;
blk_start_plug(&plug);
}
while ((bio = bio_list_pop(&bios))) {
/*io_punt is a sysctl user interface to control the print*/
if(io_punt) {
printk("[%s:%d] bio start,size:%llu,%d count=%d plug?%d\n",
current->comm, current->pid, bio->bi_iter.bi_sector,
(bio->bi_iter.bi_size)>>9, count++, need_plug);
}
submit_bio(bio);
}
if (need_plug)
blk_finish_plug(&plug);
Steps that need to be set to trigger *PUNT* io before testing:
mount -t btrfs -o compress=lzo /dev/sda6 /btrfs
mount -t cgroup2 nodev /cgroup2
mkdir /cgroup2/cg3
echo "+io" > /cgroup2/cgroup.subtree_control
echo "8:0 wbps=1048576000" > /cgroup2/cg3/io.max #1000M/s
echo $$ > /cgroup2/cg3/cgroup.procs
Then use dd command to test btrfs PUNT io in current shell:
dd if=/dev/zero of=/btrfs/file bs=64K count=100000
Test hardware environment as below:
[root@localhost btrfs]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
With above debug code, test command and test environment, I did the
tests under 3 different system loads, which are triggered by stress:
1, Run 64 threads by command "stress -c 64 &"
[53615.975974] [kworker/u66:18:1490] bio start,size:45583056,8 count=0 plug?1
[53615.975980] [kworker/u66:18:1490] bio start,size:45583064,8 count=1 plug?1
[53615.975984] [kworker/u66:18:1490] bio start,size:45583072,8 count=2 plug?1
[53615.975987] [kworker/u66:18:1490] bio start,size:45583080,8 count=3 plug?1
[53615.975990] [kworker/u66:18:1490] bio start,size:45583088,8 count=4 plug?1
[53615.975993] [kworker/u66:18:1490] bio start,size:45583096,8 count=5 plug?1
... ...
[53615.977041] [kworker/u66:18:1490] bio start,size:45585480,8 count=303 plug?1
[53615.977044] [kworker/u66:18:1490] bio start,size:45585488,8 count=304 plug?1
[53615.977047] [kworker/u66:18:1490] bio start,size:45585496,8 count=305 plug?1
[53615.977050] [kworker/u66:18:1490] bio start,size:45585504,8 count=306 plug?1
[53615.977053] [kworker/u66:18:1490] bio start,size:45585512,8 count=307 plug?1
[53615.977056] [kworker/u66:18:1490] bio start,size:45585520,8 count=308 plug?1
[53615.977058] [kworker/u66:18:1490] bio start,size:45585528,8 count=309 plug?1
2, Run 32 threads by command "stress -c 32 &"
[50586.290521] [kworker/u66:6:32351] bio start,size:45806496,8 count=0 plug?1
[50586.290526] [kworker/u66:6:32351] bio start,size:45806504,8 count=1 plug?1
[50586.290529] [kworker/u66:6:32351] bio start,size:45806512,8 count=2 plug?1
[50586.290531] [kworker/u66:6:32351] bio start,size:45806520,8 count=3 plug?1
[50586.290533] [kworker/u66:6:32351] bio start,size:45806528,8 count=4 plug?1
[50586.290535] [kworker/u66:6:32351] bio start,size:45806536,8 count=5 plug?1
... ...
[50586.299640] [kworker/u66:5:32350] bio start,size:45808576,8 count=252 plug?1
[50586.299643] [kworker/u66:5:32350] bio start,size:45808584,8 count=253 plug?1
[50586.299646] [kworker/u66:5:32350] bio start,size:45808592,8 count=254 plug?1
[50586.299649] [kworker/u66:5:32350] bio start,size:45808600,8 count=255 plug?1
[50586.299652] [kworker/u66:5:32350] bio start,size:45808608,8 count=256 plug?1
[50586.299663] [kworker/u66:5:32350] bio start,size:45808616,8 count=257 plug?1
[50586.299665] [kworker/u66:5:32350] bio start,size:45808624,8 count=258 plug?1
[50586.299668] [kworker/u66:5:32350] bio start,size:45808632,8 count=259 plug?1
3, Don't run thread by stress
[50861.355246] [kworker/u66:19:32376] bio start,size:13544504,8 count=0 plug?0
[50861.355288] [kworker/u66:19:32376] bio start,size:13544512,8 count=0 plug?0
[50861.355322] [kworker/u66:19:32376] bio start,size:13544520,8 count=0 plug?0
[50861.355353] [kworker/u66:19:32376] bio start,size:13544528,8 count=0 plug?0
[50861.355392] [kworker/u66:19:32376] bio start,size:13544536,8 count=0 plug?0
[50861.355431] [kworker/u66:19:32376] bio start,size:13544544,8 count=0 plug?0
[50861.355468] [kworker/u66:19:32376] bio start,size:13544552,8 count=0 plug?0
[50861.355499] [kworker/u66:19:32376] bio start,size:13544560,8 count=0 plug?0
[50861.355532] [kworker/u66:19:32376] bio start,size:13544568,8 count=0 plug?0
[50861.355575] [kworker/u66:19:32376] bio start,size:13544576,8 count=0 plug?0
[50861.355618] [kworker/u66:19:32376] bio start,size:13544584,8 count=0 plug?0
[50861.355659] [kworker/u66:19:32376] bio start,size:13544592,8 count=0 plug?0
[50861.355740] [kworker/u66:0:32346] bio start,size:13544600,8 count=0 plug?1
[50861.355748] [kworker/u66:0:32346] bio start,size:13544608,8 count=1 plug?1
[50861.355962] [kworker/u66:2:32347] bio start,size:13544616,8 count=0 plug?0
[50861.356272] [kworker/u66:7:31962] bio start,size:13544624,8 count=0 plug?0
[50861.356446] [kworker/u66:7:31962] bio start,size:13544632,8 count=0 plug?0
[50861.356567] [kworker/u66:7:31962] bio start,size:13544640,8 count=0 plug?0
[50861.356707] [kworker/u66:19:32376] bio start,size:13544648,8 count=0 plug?0
[50861.356748] [kworker/u66:15:32355] bio start,size:13544656,8 count=0 plug?0
[50861.356825] [kworker/u66:17:31970] bio start,size:13544664,8 count=0 plug?0
Analysis of above 3 test results with different system load:
>From above test, we can see more and more continuous bios can be plugged
with system load increasing. When run "stress -c 64 &", 310 continuous
bios are plugged; When run "stress -c 32 &", 260 continuous bios are
plugged; When don't run stress, at most only 2 continuous bios are
plugged, in most cases, bio_list only contains one single bio.
How to explain above phenomenon:
We know, in submit_bio(), if the bio is a REQ_CGROUP_PUNT io, it will
queue a work to workqueue blkcg_punt_bio_wq. But when the workqueue is
scheduled, it depends on the system load. When system load is low, the
workqueue will be quickly scheduled, and the bio in bio_list will be
quickly processed in blkg_async_bio_workfn(), so there is less chance
that the same io submit thread can add multiple continuous bios to
bio_list before workqueue is scheduled to run. The analysis aligned with
above test "3".
When system load is high, there is some delay before the workqueue can
be scheduled to run, the higher the system load the greater the delay.
So there is more chance that the same io submit thread can add multiple
continuous bios to bio_list. Then when the workqueue is scheduled to run,
there are more continuous bios in bio_list, which will be processed in
blkg_async_bio_workfn(). The analysis aligned with above test "1" and "2".
According to test, we can get io performance improved with the patch,
especially when system load is higher. Another optimazition is to use
the plug only when bio_list contains at least 2 bios.
Signed-off-by: Xianting Tian <tian.xianting@h3c.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-09-09 23:15:06 -07:00
|
|
|
if (need_plug)
|
|
|
|
blk_finish_plug(&plug);
|
2019-06-27 13:39:52 -07:00
|
|
|
}
|
|
|
|
|
2023-03-26 17:49:53 -07:00
|
|
|
/*
|
|
|
|
* When a shared kthread issues a bio for a cgroup, doing so synchronously can
|
|
|
|
* lead to priority inversions as the kthread can be trapped waiting for that
|
|
|
|
* cgroup. Use this helper instead of submit_bio to punt the actual issuing to
|
|
|
|
* a dedicated per-blkcg work item to avoid such priority inversions.
|
|
|
|
*/
|
|
|
|
void blkcg_punt_bio_submit(struct bio *bio)
|
|
|
|
{
|
|
|
|
struct blkcg_gq *blkg = bio->bi_blkg;
|
|
|
|
|
|
|
|
if (blkg->parent) {
|
|
|
|
spin_lock(&blkg->async_bio_lock);
|
|
|
|
bio_list_add(&blkg->async_bios, bio);
|
|
|
|
spin_unlock(&blkg->async_bio_lock);
|
|
|
|
queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work);
|
|
|
|
} else {
|
|
|
|
/* never bounce for the root cgroup */
|
|
|
|
submit_bio(bio);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(blkcg_punt_bio_submit);
|
|
|
|
|
|
|
|
static int __init blkcg_punt_bio_init(void)
|
|
|
|
{
|
|
|
|
blkcg_punt_bio_wq = alloc_workqueue("blkcg_punt_bio",
|
|
|
|
WQ_MEM_RECLAIM | WQ_FREEZABLE |
|
|
|
|
WQ_UNBOUND | WQ_SYSFS, 0);
|
|
|
|
if (!blkcg_punt_bio_wq)
|
|
|
|
return -ENOMEM;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
subsys_initcall(blkcg_punt_bio_init);
|
|
|
|
#endif /* CONFIG_BLK_CGROUP_PUNT_BIO */
|
|
|
|
|
2022-04-19 21:27:17 -07:00
|
|
|
/**
|
|
|
|
* bio_blkcg_css - return the blkcg CSS associated with a bio
|
|
|
|
* @bio: target bio
|
|
|
|
*
|
|
|
|
* This returns the CSS for the blkcg associated with a bio, or %NULL if not
|
|
|
|
* associated. Callers are expected to either handle %NULL or know association
|
|
|
|
* has been done prior to calling this.
|
|
|
|
*/
|
|
|
|
struct cgroup_subsys_state *bio_blkcg_css(struct bio *bio)
|
|
|
|
{
|
|
|
|
if (!bio || !bio->bi_blkg)
|
|
|
|
return NULL;
|
|
|
|
return &bio->bi_blkg->blkcg->css;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(bio_blkcg_css);
|
|
|
|
|
2022-04-19 21:27:14 -07:00
|
|
|
/**
|
|
|
|
* blkcg_parent - get the parent of a blkcg
|
|
|
|
* @blkcg: blkcg of interest
|
|
|
|
*
|
|
|
|
* Return the parent blkcg of @blkcg. Can be called anytime.
|
|
|
|
*/
|
|
|
|
static inline struct blkcg *blkcg_parent(struct blkcg *blkcg)
|
|
|
|
{
|
|
|
|
return css_to_blkcg(blkcg->css.parent);
|
|
|
|
}
|
|
|
|
|
2012-03-05 14:15:14 -07:00
|
|
|
/**
|
|
|
|
* blkg_alloc - allocate a blkg
|
|
|
|
* @blkcg: block cgroup the new blkg is associated with
|
2022-09-21 11:05:01 -07:00
|
|
|
* @disk: gendisk the new blkg is associated with
|
2012-06-04 20:40:52 -07:00
|
|
|
* @gfp_mask: allocation mask to use
|
2012-03-05 14:15:14 -07:00
|
|
|
*
|
2024-01-14 12:10:56 -07:00
|
|
|
* Allocate a new blkg associating @blkcg and @disk.
|
2012-03-05 14:15:14 -07:00
|
|
|
*/
|
2022-09-21 11:05:01 -07:00
|
|
|
static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct gendisk *disk,
|
2012-06-04 20:40:52 -07:00
|
|
|
gfp_t gfp_mask)
|
2012-03-05 14:15:14 -07:00
|
|
|
{
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2019-11-07 12:18:03 -07:00
|
|
|
int i, cpu;
|
2012-03-05 14:15:14 -07:00
|
|
|
|
|
|
|
/* alloc and init base part */
|
2022-09-21 11:05:01 -07:00
|
|
|
blkg = kzalloc_node(sizeof(*blkg), gfp_mask, disk->queue->node);
|
2012-03-05 14:15:14 -07:00
|
|
|
if (!blkg)
|
|
|
|
return NULL;
|
2019-06-13 15:30:39 -07:00
|
|
|
if (percpu_ref_init(&blkg->refcnt, blkg_release, 0, gfp_mask))
|
2023-02-03 08:03:44 -07:00
|
|
|
goto out_free_blkg;
|
2019-11-07 12:18:03 -07:00
|
|
|
blkg->iostat_cpu = alloc_percpu_gfp(struct blkg_iostat_set, gfp_mask);
|
|
|
|
if (!blkg->iostat_cpu)
|
2023-02-03 08:03:44 -07:00
|
|
|
goto out_exit_refcnt;
|
2022-09-21 11:05:01 -07:00
|
|
|
if (!blk_get_queue(disk->queue))
|
2023-02-03 08:03:44 -07:00
|
|
|
goto out_free_iostat;
|
2022-03-18 06:01:43 -07:00
|
|
|
|
2022-09-21 11:05:01 -07:00
|
|
|
blkg->q = disk->queue;
|
2012-03-05 14:15:20 -07:00
|
|
|
INIT_LIST_HEAD(&blkg->q_node);
|
2023-03-26 17:49:53 -07:00
|
|
|
blkg->blkcg = blkcg;
|
2024-05-15 07:30:59 -07:00
|
|
|
blkg->iostat.blkg = blkg;
|
2023-03-26 17:49:53 -07:00
|
|
|
#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
|
2019-06-27 13:39:52 -07:00
|
|
|
spin_lock_init(&blkg->async_bio_lock);
|
|
|
|
bio_list_init(&blkg->async_bios);
|
|
|
|
INIT_WORK(&blkg->async_bio_work, blkg_async_bio_workfn);
|
2023-03-26 17:49:53 -07:00
|
|
|
#endif
|
2012-03-05 14:15:14 -07:00
|
|
|
|
2019-11-07 12:18:03 -07:00
|
|
|
u64_stats_init(&blkg->iostat.sync);
|
2022-11-04 17:59:01 -07:00
|
|
|
for_each_possible_cpu(cpu) {
|
2019-11-07 12:18:03 -07:00
|
|
|
u64_stats_init(&per_cpu_ptr(blkg->iostat_cpu, cpu)->sync);
|
2022-11-04 17:59:01 -07:00
|
|
|
per_cpu_ptr(blkg->iostat_cpu, cpu)->blkg = blkg;
|
|
|
|
}
|
2019-11-07 12:18:03 -07:00
|
|
|
|
2012-04-13 13:11:28 -07:00
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++) {
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
2012-03-05 14:15:20 -07:00
|
|
|
struct blkg_policy_data *pd;
|
2012-03-05 14:15:14 -07:00
|
|
|
|
2022-09-21 11:05:01 -07:00
|
|
|
if (!blkcg_policy_enabled(disk->queue, pol))
|
2012-03-05 14:15:20 -07:00
|
|
|
continue;
|
|
|
|
|
|
|
|
/* alloc per-policy data and attach it to blkg */
|
2023-02-03 08:03:58 -07:00
|
|
|
pd = pol->pd_alloc_fn(disk, blkcg, gfp_mask);
|
blkcg: implement per-blkg request allocation
Currently, request_queue has one request_list to allocate requests
from regardless of blkcg of the IO being issued. When the unified
request pool is used up, cfq proportional IO limits become meaningless
- whoever grabs the next request being freed wins the race regardless
of the configured weights.
This can be easily demonstrated by creating a blkio cgroup w/ very low
weight, put a program which can issue a lot of random direct IOs there
and running a sequential IO from a different cgroup. As soon as the
request pool is used up, the sequential IO bandwidth crashes.
This patch implements per-blkg request_list. Each blkg has its own
request_list and any IO allocates its request from the matching blkg
making blkcgs completely isolated in terms of request allocation.
* Root blkcg uses the request_list embedded in each request_queue,
which was renamed to @q->root_rl from @q->rq. While making blkcg rl
handling a bit harier, this enables avoiding most overhead for root
blkcg.
* Queue fullness is properly per request_list but bdi isn't blkcg
aware yet, so congestion state currently just follows the root
blkcg. As writeback isn't aware of blkcg yet, this works okay for
async congestion but readahead may get the wrong signals. It's
better than blkcg completely collapsing with shared request_list but
needs to be improved with future changes.
* After this change, each block cgroup gets a full request pool making
resource consumption of each cgroup higher. This makes allowing
non-root users to create cgroups less desirable; however, note that
allowing non-root users to directly manage cgroups is already
severely broken regardless of this patch - each block cgroup
consumes kernel memory and skews IO weight (IO weights are not
hierarchical).
v2: queue-sysfs.txt updated and patch description udpated as suggested
by Vivek.
v3: blk_get_rl() wasn't checking error return from
blkg_lookup_create() and may cause oops on lookup failure. Fix it
by falling back to root_rl on blkg lookup failures. This problem
was spotted by Rakesh Iyer <rni@google.com>.
v4: Updated to accomodate 458f27a982 "block: Avoid missed wakeup in
request waitqueue". blk_drain_queue() now wakes up waiters on all
blkg->rl on the target queue.
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2012-06-26 15:05:44 -07:00
|
|
|
if (!pd)
|
2023-02-03 08:03:44 -07:00
|
|
|
goto out_free_pds;
|
2012-03-05 14:15:20 -07:00
|
|
|
blkg->pd[i] = pd;
|
|
|
|
pd->blkg = blkg;
|
2013-01-09 09:05:12 -07:00
|
|
|
pd->plid = i;
|
2023-01-19 04:03:49 -07:00
|
|
|
pd->online = false;
|
2012-03-05 14:15:20 -07:00
|
|
|
}
|
|
|
|
|
2012-03-05 14:15:14 -07:00
|
|
|
return blkg;
|
blkcg: implement per-blkg request allocation
Currently, request_queue has one request_list to allocate requests
from regardless of blkcg of the IO being issued. When the unified
request pool is used up, cfq proportional IO limits become meaningless
- whoever grabs the next request being freed wins the race regardless
of the configured weights.
This can be easily demonstrated by creating a blkio cgroup w/ very low
weight, put a program which can issue a lot of random direct IOs there
and running a sequential IO from a different cgroup. As soon as the
request pool is used up, the sequential IO bandwidth crashes.
This patch implements per-blkg request_list. Each blkg has its own
request_list and any IO allocates its request from the matching blkg
making blkcgs completely isolated in terms of request allocation.
* Root blkcg uses the request_list embedded in each request_queue,
which was renamed to @q->root_rl from @q->rq. While making blkcg rl
handling a bit harier, this enables avoiding most overhead for root
blkcg.
* Queue fullness is properly per request_list but bdi isn't blkcg
aware yet, so congestion state currently just follows the root
blkcg. As writeback isn't aware of blkcg yet, this works okay for
async congestion but readahead may get the wrong signals. It's
better than blkcg completely collapsing with shared request_list but
needs to be improved with future changes.
* After this change, each block cgroup gets a full request pool making
resource consumption of each cgroup higher. This makes allowing
non-root users to create cgroups less desirable; however, note that
allowing non-root users to directly manage cgroups is already
severely broken regardless of this patch - each block cgroup
consumes kernel memory and skews IO weight (IO weights are not
hierarchical).
v2: queue-sysfs.txt updated and patch description udpated as suggested
by Vivek.
v3: blk_get_rl() wasn't checking error return from
blkg_lookup_create() and may cause oops on lookup failure. Fix it
by falling back to root_rl on blkg lookup failures. This problem
was spotted by Rakesh Iyer <rni@google.com>.
v4: Updated to accomodate 458f27a982 "block: Avoid missed wakeup in
request waitqueue". blk_drain_queue() now wakes up waiters on all
blkg->rl on the target queue.
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2012-06-26 15:05:44 -07:00
|
|
|
|
2023-02-03 08:03:44 -07:00
|
|
|
out_free_pds:
|
|
|
|
while (--i >= 0)
|
|
|
|
if (blkg->pd[i])
|
|
|
|
blkcg_policy[i]->pd_free_fn(blkg->pd[i]);
|
2023-02-14 11:33:08 -07:00
|
|
|
blk_put_queue(disk->queue);
|
2023-02-03 08:03:44 -07:00
|
|
|
out_free_iostat:
|
|
|
|
free_percpu(blkg->iostat_cpu);
|
|
|
|
out_exit_refcnt:
|
|
|
|
percpu_ref_exit(&blkg->refcnt);
|
|
|
|
out_free_blkg:
|
|
|
|
kfree(blkg);
|
blkcg: implement per-blkg request allocation
Currently, request_queue has one request_list to allocate requests
from regardless of blkcg of the IO being issued. When the unified
request pool is used up, cfq proportional IO limits become meaningless
- whoever grabs the next request being freed wins the race regardless
of the configured weights.
This can be easily demonstrated by creating a blkio cgroup w/ very low
weight, put a program which can issue a lot of random direct IOs there
and running a sequential IO from a different cgroup. As soon as the
request pool is used up, the sequential IO bandwidth crashes.
This patch implements per-blkg request_list. Each blkg has its own
request_list and any IO allocates its request from the matching blkg
making blkcgs completely isolated in terms of request allocation.
* Root blkcg uses the request_list embedded in each request_queue,
which was renamed to @q->root_rl from @q->rq. While making blkcg rl
handling a bit harier, this enables avoiding most overhead for root
blkcg.
* Queue fullness is properly per request_list but bdi isn't blkcg
aware yet, so congestion state currently just follows the root
blkcg. As writeback isn't aware of blkcg yet, this works okay for
async congestion but readahead may get the wrong signals. It's
better than blkcg completely collapsing with shared request_list but
needs to be improved with future changes.
* After this change, each block cgroup gets a full request pool making
resource consumption of each cgroup higher. This makes allowing
non-root users to create cgroups less desirable; however, note that
allowing non-root users to directly manage cgroups is already
severely broken regardless of this patch - each block cgroup
consumes kernel memory and skews IO weight (IO weights are not
hierarchical).
v2: queue-sysfs.txt updated and patch description udpated as suggested
by Vivek.
v3: blk_get_rl() wasn't checking error return from
blkg_lookup_create() and may cause oops on lookup failure. Fix it
by falling back to root_rl on blkg lookup failures. This problem
was spotted by Rakesh Iyer <rni@google.com>.
v4: Updated to accomodate 458f27a982 "block: Avoid missed wakeup in
request waitqueue". blk_drain_queue() now wakes up waiters on all
blkg->rl on the target queue.
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Vivek Goyal <vgoyal@redhat.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2012-06-26 15:05:44 -07:00
|
|
|
return NULL;
|
2012-03-05 14:15:14 -07:00
|
|
|
}
|
|
|
|
|
2012-06-04 20:40:52 -07:00
|
|
|
/*
|
2017-03-29 10:25:48 -07:00
|
|
|
* If @new_blkg is %NULL, this function tries to allocate a new one as
|
|
|
|
* necessary using %GFP_NOWAIT. @new_blkg is always consumed on return.
|
2012-06-04 20:40:52 -07:00
|
|
|
*/
|
2022-09-21 11:05:01 -07:00
|
|
|
static struct blkcg_gq *blkg_create(struct blkcg *blkcg, struct gendisk *disk,
|
2017-03-29 10:25:48 -07:00
|
|
|
struct blkcg_gq *new_blkg)
|
2011-05-19 12:38:28 -07:00
|
|
|
{
|
2017-03-29 10:25:48 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2013-01-09 09:05:12 -07:00
|
|
|
int i, ret;
|
2011-05-19 12:38:28 -07:00
|
|
|
|
2022-09-21 11:05:01 -07:00
|
|
|
lockdep_assert_held(&disk->queue->queue_lock);
|
2012-03-05 14:15:06 -07:00
|
|
|
|
2018-12-11 16:03:08 -07:00
|
|
|
/* request_queue is dying, do not create/recreate a blkg */
|
2022-09-21 11:05:01 -07:00
|
|
|
if (blk_queue_dying(disk->queue)) {
|
2018-12-11 16:03:08 -07:00
|
|
|
ret = -ENODEV;
|
|
|
|
goto err_free_blkg;
|
|
|
|
}
|
|
|
|
|
2012-03-05 14:15:11 -07:00
|
|
|
/* blkg holds a reference to blkcg */
|
2014-05-13 09:11:01 -07:00
|
|
|
if (!css_tryget_online(&blkcg->css)) {
|
2015-08-18 14:55:28 -07:00
|
|
|
ret = -ENODEV;
|
2013-01-09 09:05:10 -07:00
|
|
|
goto err_free_blkg;
|
2012-06-04 20:40:52 -07:00
|
|
|
}
|
2012-03-05 14:15:06 -07:00
|
|
|
|
2017-03-29 10:25:48 -07:00
|
|
|
/* allocate */
|
|
|
|
if (!new_blkg) {
|
2022-09-21 11:05:01 -07:00
|
|
|
new_blkg = blkg_alloc(blkcg, disk, GFP_NOWAIT | __GFP_NOWARN);
|
2017-03-29 10:25:48 -07:00
|
|
|
if (unlikely(!new_blkg)) {
|
|
|
|
ret = -ENOMEM;
|
2020-07-01 02:06:21 -07:00
|
|
|
goto err_put_css;
|
2012-06-04 20:40:52 -07:00
|
|
|
}
|
|
|
|
}
|
2017-03-29 10:25:48 -07:00
|
|
|
blkg = new_blkg;
|
2012-03-05 14:15:06 -07:00
|
|
|
|
2013-05-14 13:52:31 -07:00
|
|
|
/* link parent */
|
2013-01-09 09:05:10 -07:00
|
|
|
if (blkcg_parent(blkcg)) {
|
2022-09-21 11:05:01 -07:00
|
|
|
blkg->parent = blkg_lookup(blkcg_parent(blkcg), disk->queue);
|
2013-01-09 09:05:10 -07:00
|
|
|
if (WARN_ON_ONCE(!blkg->parent)) {
|
2015-08-18 14:55:28 -07:00
|
|
|
ret = -ENODEV;
|
2020-07-01 02:06:21 -07:00
|
|
|
goto err_put_css;
|
2013-01-09 09:05:10 -07:00
|
|
|
}
|
|
|
|
blkg_get(blkg->parent);
|
|
|
|
}
|
|
|
|
|
2013-05-14 13:52:31 -07:00
|
|
|
/* invoke per-policy init */
|
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++) {
|
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
|
|
|
|
|
|
|
if (blkg->pd[i] && pol->pd_init_fn)
|
2015-08-18 14:55:14 -07:00
|
|
|
pol->pd_init_fn(blkg->pd[i]);
|
2013-05-14 13:52:31 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* insert */
|
2012-03-05 14:15:06 -07:00
|
|
|
spin_lock(&blkcg->lock);
|
2022-09-21 11:05:01 -07:00
|
|
|
ret = radix_tree_insert(&blkcg->blkg_tree, disk->queue->id, blkg);
|
2012-04-19 16:29:24 -07:00
|
|
|
if (likely(!ret)) {
|
|
|
|
hlist_add_head_rcu(&blkg->blkcg_node, &blkcg->blkg_list);
|
2022-09-21 11:05:01 -07:00
|
|
|
list_add(&blkg->q_node, &disk->queue->blkg_list);
|
2013-01-09 09:05:12 -07:00
|
|
|
|
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++) {
|
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
|
|
|
|
2023-01-19 04:03:49 -07:00
|
|
|
if (blkg->pd[i]) {
|
|
|
|
if (pol->pd_online_fn)
|
|
|
|
pol->pd_online_fn(blkg->pd[i]);
|
|
|
|
blkg->pd[i]->online = true;
|
|
|
|
}
|
2013-01-09 09:05:12 -07:00
|
|
|
}
|
2012-04-19 16:29:24 -07:00
|
|
|
}
|
2013-01-09 09:05:12 -07:00
|
|
|
blkg->online = true;
|
2012-03-05 14:15:06 -07:00
|
|
|
spin_unlock(&blkcg->lock);
|
2012-04-19 16:29:23 -07:00
|
|
|
|
2015-05-22 14:13:19 -07:00
|
|
|
if (!ret)
|
2012-04-19 16:29:24 -07:00
|
|
|
return blkg;
|
2012-06-04 20:40:52 -07:00
|
|
|
|
2013-01-09 09:05:10 -07:00
|
|
|
/* @blkg failed fully initialized, use the usual release path */
|
|
|
|
blkg_put(blkg);
|
|
|
|
return ERR_PTR(ret);
|
|
|
|
|
2017-03-29 10:25:48 -07:00
|
|
|
err_put_css:
|
2012-04-19 16:29:23 -07:00
|
|
|
css_put(&blkcg->css);
|
2013-01-09 09:05:10 -07:00
|
|
|
err_free_blkg:
|
2023-02-06 08:02:01 -07:00
|
|
|
if (new_blkg)
|
|
|
|
blkg_free(new_blkg);
|
2013-01-09 09:05:10 -07:00
|
|
|
return ERR_PTR(ret);
|
2009-12-03 10:59:42 -07:00
|
|
|
}
|
2012-04-13 13:11:34 -07:00
|
|
|
|
2013-01-09 09:05:10 -07:00
|
|
|
/**
|
2020-06-27 00:31:52 -07:00
|
|
|
* blkg_lookup_create - lookup blkg, try to create one if not there
|
2013-01-09 09:05:10 -07:00
|
|
|
* @blkcg: blkcg of interest
|
2022-09-21 11:05:01 -07:00
|
|
|
* @disk: gendisk of interest
|
2013-01-09 09:05:10 -07:00
|
|
|
*
|
2022-09-21 11:05:01 -07:00
|
|
|
* Lookup blkg for the @blkcg - @disk pair. If it doesn't exist, try to
|
2013-01-09 09:05:10 -07:00
|
|
|
* create one. blkg creation is performed recursively from blkcg_root such
|
|
|
|
* that all non-root blkg's have access to the parent blkg. This function
|
2022-09-21 11:05:01 -07:00
|
|
|
* should be called under RCU read lock and takes @disk->queue->queue_lock.
|
2013-01-09 09:05:10 -07:00
|
|
|
*
|
2018-12-05 10:10:28 -07:00
|
|
|
* Returns the blkg or the closest blkg if blkg_create() fails as it walks
|
|
|
|
* down from root.
|
2013-01-09 09:05:10 -07:00
|
|
|
*/
|
2020-06-27 00:31:52 -07:00
|
|
|
static struct blkcg_gq *blkg_lookup_create(struct blkcg *blkcg,
|
2022-09-21 11:05:01 -07:00
|
|
|
struct gendisk *disk)
|
2012-04-13 13:11:34 -07:00
|
|
|
{
|
2022-09-21 11:05:01 -07:00
|
|
|
struct request_queue *q = disk->queue;
|
2013-01-09 09:05:10 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2020-06-27 00:31:52 -07:00
|
|
|
unsigned long flags;
|
2013-01-09 09:05:10 -07:00
|
|
|
|
|
|
|
WARN_ON_ONCE(!rcu_read_lock_held());
|
|
|
|
|
2020-06-27 00:31:52 -07:00
|
|
|
blkg = blkg_lookup(blkcg, q);
|
2013-01-09 09:05:10 -07:00
|
|
|
if (blkg)
|
|
|
|
return blkg;
|
|
|
|
|
2020-06-27 00:31:52 -07:00
|
|
|
spin_lock_irqsave(&q->queue_lock, flags);
|
2022-09-21 11:04:48 -07:00
|
|
|
blkg = blkg_lookup(blkcg, q);
|
|
|
|
if (blkg) {
|
2022-09-26 23:54:25 -07:00
|
|
|
if (blkcg != &blkcg_root &&
|
|
|
|
blkg != rcu_dereference(blkcg->blkg_hint))
|
|
|
|
rcu_assign_pointer(blkcg->blkg_hint, blkg);
|
2020-06-27 00:31:52 -07:00
|
|
|
goto found;
|
2022-09-21 11:04:48 -07:00
|
|
|
}
|
2020-06-27 00:31:52 -07:00
|
|
|
|
2013-01-09 09:05:10 -07:00
|
|
|
/*
|
|
|
|
* Create blkgs walking down from blkcg_root to @blkcg, so that all
|
2018-12-05 10:10:28 -07:00
|
|
|
* non-root blkgs have access to their parents. Returns the closest
|
|
|
|
* blkg to the intended blkg should blkg_create() fail.
|
2013-01-09 09:05:10 -07:00
|
|
|
*/
|
|
|
|
while (true) {
|
|
|
|
struct blkcg *pos = blkcg;
|
|
|
|
struct blkcg *parent = blkcg_parent(blkcg);
|
2018-12-05 10:10:28 -07:00
|
|
|
struct blkcg_gq *ret_blkg = q->root_blkg;
|
|
|
|
|
|
|
|
while (parent) {
|
2022-09-21 11:04:47 -07:00
|
|
|
blkg = blkg_lookup(parent, q);
|
2018-12-05 10:10:28 -07:00
|
|
|
if (blkg) {
|
|
|
|
/* remember closest blkg */
|
|
|
|
ret_blkg = blkg;
|
|
|
|
break;
|
|
|
|
}
|
2013-01-09 09:05:10 -07:00
|
|
|
pos = parent;
|
|
|
|
parent = blkcg_parent(parent);
|
|
|
|
}
|
|
|
|
|
2022-09-21 11:05:01 -07:00
|
|
|
blkg = blkg_create(pos, disk, NULL);
|
2020-06-27 00:31:52 -07:00
|
|
|
if (IS_ERR(blkg)) {
|
|
|
|
blkg = ret_blkg;
|
|
|
|
break;
|
|
|
|
}
|
2018-12-05 10:10:28 -07:00
|
|
|
if (pos == blkcg)
|
2020-06-27 00:31:52 -07:00
|
|
|
break;
|
2018-12-05 10:10:27 -07:00
|
|
|
}
|
|
|
|
|
2020-06-27 00:31:52 -07:00
|
|
|
found:
|
|
|
|
spin_unlock_irqrestore(&q->queue_lock, flags);
|
2018-12-05 10:10:27 -07:00
|
|
|
return blkg;
|
|
|
|
}
|
|
|
|
|
2012-04-16 13:57:25 -07:00
|
|
|
static void blkg_destroy(struct blkcg_gq *blkg)
|
2012-03-05 14:15:19 -07:00
|
|
|
{
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg *blkcg = blkg->blkcg;
|
2018-08-31 13:22:42 -07:00
|
|
|
int i;
|
2012-03-05 14:15:19 -07:00
|
|
|
|
2018-11-15 12:17:28 -07:00
|
|
|
lockdep_assert_held(&blkg->q->queue_lock);
|
2012-03-05 14:15:21 -07:00
|
|
|
lockdep_assert_held(&blkcg->lock);
|
2012-03-05 14:15:19 -07:00
|
|
|
|
2023-01-19 04:03:50 -07:00
|
|
|
/*
|
|
|
|
* blkg stays on the queue list until blkg_free_workfn(), see details in
|
|
|
|
* blkg_free_workfn(), hence this function can be called from
|
|
|
|
* blkcg_destroy_blkgs() first and again from blkg_destroy_all() before
|
|
|
|
* blkg_free_workfn().
|
|
|
|
*/
|
|
|
|
if (hlist_unhashed(&blkg->blkcg_node))
|
|
|
|
return;
|
2012-04-19 16:29:24 -07:00
|
|
|
|
2018-08-31 13:22:42 -07:00
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++) {
|
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
|
|
|
|
2023-01-19 04:03:49 -07:00
|
|
|
if (blkg->pd[i] && blkg->pd[i]->online) {
|
2023-02-02 06:49:13 -07:00
|
|
|
blkg->pd[i]->online = false;
|
2023-01-19 04:03:49 -07:00
|
|
|
if (pol->pd_offline_fn)
|
|
|
|
pol->pd_offline_fn(blkg->pd[i]);
|
|
|
|
}
|
2018-08-31 13:22:42 -07:00
|
|
|
}
|
|
|
|
|
2013-01-09 09:05:12 -07:00
|
|
|
blkg->online = false;
|
|
|
|
|
2012-04-19 16:29:24 -07:00
|
|
|
radix_tree_delete(&blkcg->blkg_tree, blkg->q->id);
|
2012-03-05 14:15:21 -07:00
|
|
|
hlist_del_init_rcu(&blkg->blkcg_node);
|
2012-03-05 14:15:19 -07:00
|
|
|
|
2012-04-19 16:29:24 -07:00
|
|
|
/*
|
|
|
|
* Both setting lookup hint to and clearing it from @blkg are done
|
|
|
|
* under queue_lock. If it's not pointing to @blkg now, it never
|
|
|
|
* will. Hint assignment itself can race safely.
|
|
|
|
*/
|
2014-02-17 14:35:57 -07:00
|
|
|
if (rcu_access_pointer(blkcg->blkg_hint) == blkg)
|
2012-04-19 16:29:24 -07:00
|
|
|
rcu_assign_pointer(blkcg->blkg_hint, NULL);
|
|
|
|
|
2012-03-05 14:15:19 -07:00
|
|
|
/*
|
|
|
|
* Put the reference taken at the time of creation so that when all
|
|
|
|
* queues are gone, group can be destroyed.
|
|
|
|
*/
|
2018-12-05 10:10:38 -07:00
|
|
|
percpu_ref_kill(&blkg->refcnt);
|
2012-03-05 14:15:19 -07:00
|
|
|
}
|
|
|
|
|
2022-09-21 11:04:59 -07:00
|
|
|
static void blkg_destroy_all(struct gendisk *disk)
|
2012-03-05 14:15:00 -07:00
|
|
|
{
|
2022-09-21 11:04:59 -07:00
|
|
|
struct request_queue *q = disk->queue;
|
2024-01-04 11:00:30 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2021-07-06 18:56:49 -07:00
|
|
|
int count = BLKG_DESTROY_BATCH_SIZE;
|
2023-11-16 19:35:24 -07:00
|
|
|
int i;
|
2012-03-05 14:15:00 -07:00
|
|
|
|
2021-07-06 18:56:49 -07:00
|
|
|
restart:
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
2024-01-04 11:00:30 -07:00
|
|
|
list_for_each_entry(blkg, &q->blkg_list, q_node) {
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg *blkcg = blkg->blkcg;
|
2012-03-05 14:15:00 -07:00
|
|
|
|
2023-04-27 21:51:49 -07:00
|
|
|
if (hlist_unhashed(&blkg->blkcg_node))
|
|
|
|
continue;
|
|
|
|
|
2012-03-05 14:15:21 -07:00
|
|
|
spin_lock(&blkcg->lock);
|
|
|
|
blkg_destroy(blkg);
|
|
|
|
spin_unlock(&blkcg->lock);
|
2021-07-06 18:56:49 -07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* in order to avoid holding the spin lock for too long, release
|
|
|
|
* it when a batch of blkgs are destroyed.
|
|
|
|
*/
|
|
|
|
if (!(--count)) {
|
|
|
|
count = BLKG_DESTROY_BATCH_SIZE;
|
|
|
|
spin_unlock_irq(&q->queue_lock);
|
|
|
|
cond_resched();
|
|
|
|
goto restart;
|
|
|
|
}
|
2012-03-05 14:15:00 -07:00
|
|
|
}
|
2015-09-05 12:47:36 -07:00
|
|
|
|
2023-11-16 19:35:24 -07:00
|
|
|
/*
|
|
|
|
* Mark policy deactivated since policy offline has been done, and
|
|
|
|
* the free is scheduled, so future blkcg_deactivate_policy() can
|
|
|
|
* be bypassed
|
|
|
|
*/
|
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++) {
|
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
|
|
|
|
|
|
|
if (pol)
|
|
|
|
__clear_bit(pol->plid, q->blkcg_pols);
|
|
|
|
}
|
|
|
|
|
2015-09-05 12:47:36 -07:00
|
|
|
q->root_blkg = NULL;
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2012-03-05 14:15:00 -07:00
|
|
|
}
|
|
|
|
|
2024-05-14 18:31:56 -07:00
|
|
|
static void blkg_iostat_set(struct blkg_iostat *dst, struct blkg_iostat *src)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < BLKG_IOSTAT_NR; i++) {
|
|
|
|
dst->bytes[i] = src->bytes[i];
|
|
|
|
dst->ios[i] = src->ios[i];
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __blkg_clear_stat(struct blkg_iostat_set *bis)
|
|
|
|
{
|
|
|
|
struct blkg_iostat cur = {0};
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
flags = u64_stats_update_begin_irqsave(&bis->sync);
|
|
|
|
blkg_iostat_set(&bis->cur, &cur);
|
|
|
|
blkg_iostat_set(&bis->last, &cur);
|
|
|
|
u64_stats_update_end_irqrestore(&bis->sync, flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void blkg_clear_stat(struct blkcg_gq *blkg)
|
|
|
|
{
|
|
|
|
int cpu;
|
|
|
|
|
|
|
|
for_each_possible_cpu(cpu) {
|
|
|
|
struct blkg_iostat_set *s = per_cpu_ptr(blkg->iostat_cpu, cpu);
|
|
|
|
|
|
|
|
__blkg_clear_stat(s);
|
|
|
|
}
|
|
|
|
__blkg_clear_stat(&blkg->iostat);
|
|
|
|
}
|
|
|
|
|
2013-08-08 17:11:24 -07:00
|
|
|
static int blkcg_reset_stats(struct cgroup_subsys_state *css,
|
|
|
|
struct cftype *cftype, u64 val)
|
2010-04-01 15:01:24 -07:00
|
|
|
{
|
2013-08-08 17:11:24 -07:00
|
|
|
struct blkcg *blkcg = css_to_blkcg(css);
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2024-05-14 18:31:56 -07:00
|
|
|
int i;
|
2010-04-01 15:01:24 -07:00
|
|
|
|
2015-07-09 13:39:47 -07:00
|
|
|
mutex_lock(&blkcg_pol_mutex);
|
2010-04-01 15:01:24 -07:00
|
|
|
spin_lock_irq(&blkcg->lock);
|
2012-03-08 11:53:58 -07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Note that stat reset is racy - it doesn't synchronize against
|
|
|
|
* stat updates. This is a debug feature which shouldn't exist
|
|
|
|
* anyway. If you get hit by a race, retry.
|
|
|
|
*/
|
hlist: drop the node parameter from iterators
I'm not sure why, but the hlist for each entry iterators were conceived
list_for_each_entry(pos, head, member)
The hlist ones were greedy and wanted an extra parameter:
hlist_for_each_entry(tpos, pos, head, member)
Why did they need an extra pos parameter? I'm not quite sure. Not only
they don't really need it, it also prevents the iterator from looking
exactly like the list iterator, which is unfortunate.
Besides the semantic patch, there was some manual work required:
- Fix up the actual hlist iterators in linux/list.h
- Fix up the declaration of other iterators based on the hlist ones.
- A very small amount of places were using the 'node' parameter, this
was modified to use 'obj->member' instead.
- Coccinelle didn't handle the hlist_for_each_entry_safe iterator
properly, so those had to be fixed up manually.
The semantic patch which is mostly the work of Peter Senna Tschudin is here:
@@
iterator name hlist_for_each_entry, hlist_for_each_entry_continue, hlist_for_each_entry_from, hlist_for_each_entry_rcu, hlist_for_each_entry_rcu_bh, hlist_for_each_entry_continue_rcu_bh, for_each_busy_worker, ax25_uid_for_each, ax25_for_each, inet_bind_bucket_for_each, sctp_for_each_hentry, sk_for_each, sk_for_each_rcu, sk_for_each_from, sk_for_each_safe, sk_for_each_bound, hlist_for_each_entry_safe, hlist_for_each_entry_continue_rcu, nr_neigh_for_each, nr_neigh_for_each_safe, nr_node_for_each, nr_node_for_each_safe, for_each_gfn_indirect_valid_sp, for_each_gfn_sp, for_each_host;
type T;
expression a,c,d,e;
identifier b;
statement S;
@@
-T b;
<+... when != b
(
hlist_for_each_entry(a,
- b,
c, d) S
|
hlist_for_each_entry_continue(a,
- b,
c) S
|
hlist_for_each_entry_from(a,
- b,
c) S
|
hlist_for_each_entry_rcu(a,
- b,
c, d) S
|
hlist_for_each_entry_rcu_bh(a,
- b,
c, d) S
|
hlist_for_each_entry_continue_rcu_bh(a,
- b,
c) S
|
for_each_busy_worker(a, c,
- b,
d) S
|
ax25_uid_for_each(a,
- b,
c) S
|
ax25_for_each(a,
- b,
c) S
|
inet_bind_bucket_for_each(a,
- b,
c) S
|
sctp_for_each_hentry(a,
- b,
c) S
|
sk_for_each(a,
- b,
c) S
|
sk_for_each_rcu(a,
- b,
c) S
|
sk_for_each_from
-(a, b)
+(a)
S
+ sk_for_each_from(a) S
|
sk_for_each_safe(a,
- b,
c, d) S
|
sk_for_each_bound(a,
- b,
c) S
|
hlist_for_each_entry_safe(a,
- b,
c, d, e) S
|
hlist_for_each_entry_continue_rcu(a,
- b,
c) S
|
nr_neigh_for_each(a,
- b,
c) S
|
nr_neigh_for_each_safe(a,
- b,
c, d) S
|
nr_node_for_each(a,
- b,
c) S
|
nr_node_for_each_safe(a,
- b,
c, d) S
|
- for_each_gfn_sp(a, c, d, b) S
+ for_each_gfn_sp(a, c, d) S
|
- for_each_gfn_indirect_valid_sp(a, c, d, b) S
+ for_each_gfn_indirect_valid_sp(a, c, d) S
|
for_each_host(a,
- b,
c) S
|
for_each_host_safe(a,
- b,
c, d) S
|
for_each_mesh_entry(a,
- b,
c, d) S
)
...+>
[akpm@linux-foundation.org: drop bogus change from net/ipv4/raw.c]
[akpm@linux-foundation.org: drop bogus hunk from net/ipv6/raw.c]
[akpm@linux-foundation.org: checkpatch fixes]
[akpm@linux-foundation.org: fix warnings]
[akpm@linux-foudnation.org: redo intrusive kvm changes]
Tested-by: Peter Senna Tschudin <peter.senna@gmail.com>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Sasha Levin <sasha.levin@oracle.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Gleb Natapov <gleb@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-02-27 18:06:00 -07:00
|
|
|
hlist_for_each_entry(blkg, &blkcg->blkg_list, blkcg_node) {
|
2024-05-14 18:31:56 -07:00
|
|
|
blkg_clear_stat(blkg);
|
2012-04-13 13:11:28 -07:00
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++) {
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
2012-03-05 14:15:16 -07:00
|
|
|
|
2015-08-18 14:55:14 -07:00
|
|
|
if (blkg->pd[i] && pol->pd_reset_stats_fn)
|
|
|
|
pol->pd_reset_stats_fn(blkg->pd[i]);
|
2012-04-13 13:11:26 -07:00
|
|
|
}
|
2010-04-01 15:01:24 -07:00
|
|
|
}
|
2011-05-19 12:38:30 -07:00
|
|
|
|
2010-04-01 15:01:24 -07:00
|
|
|
spin_unlock_irq(&blkcg->lock);
|
2012-04-13 13:11:26 -07:00
|
|
|
mutex_unlock(&blkcg_pol_mutex);
|
2010-04-01 15:01:24 -07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-08-18 14:55:33 -07:00
|
|
|
const char *blkg_dev_name(struct blkcg_gq *blkg)
|
2010-04-01 15:01:24 -07:00
|
|
|
{
|
2023-02-14 11:33:08 -07:00
|
|
|
if (!blkg->q->disk)
|
2021-08-09 07:17:43 -07:00
|
|
|
return NULL;
|
2021-08-16 06:46:24 -07:00
|
|
|
return bdi_dev_name(blkg->q->disk->bdi);
|
2010-04-01 15:01:24 -07:00
|
|
|
}
|
|
|
|
|
2012-04-01 14:38:42 -07:00
|
|
|
/**
|
|
|
|
* blkcg_print_blkgs - helper for printing per-blkg data
|
|
|
|
* @sf: seq_file to print to
|
|
|
|
* @blkcg: blkcg of interest
|
|
|
|
* @prfill: fill function to print out a blkg
|
|
|
|
* @pol: policy in question
|
|
|
|
* @data: data to be passed to @prfill
|
|
|
|
* @show_total: to print out sum of prfill return values or not
|
|
|
|
*
|
|
|
|
* This function invokes @prfill on each blkg of @blkcg if pd for the
|
|
|
|
* policy specified by @pol exists. @prfill is invoked with @sf, the
|
2013-01-09 09:05:13 -07:00
|
|
|
* policy data and @data and the matching queue lock held. If @show_total
|
|
|
|
* is %true, the sum of the return values from @prfill is printed with
|
|
|
|
* "Total" label at the end.
|
2012-04-01 14:38:42 -07:00
|
|
|
*
|
|
|
|
* This is to be used to construct print functions for
|
|
|
|
* cftype->read_seq_string method.
|
|
|
|
*/
|
2012-04-16 13:57:25 -07:00
|
|
|
void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg,
|
2012-04-16 13:57:26 -07:00
|
|
|
u64 (*prfill)(struct seq_file *,
|
|
|
|
struct blkg_policy_data *, int),
|
2012-04-16 13:57:25 -07:00
|
|
|
const struct blkcg_policy *pol, int data,
|
2012-04-13 13:11:27 -07:00
|
|
|
bool show_total)
|
2011-05-19 12:38:28 -07:00
|
|
|
{
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2012-04-01 14:38:42 -07:00
|
|
|
u64 total = 0;
|
2011-05-19 12:38:28 -07:00
|
|
|
|
2013-01-09 09:05:13 -07:00
|
|
|
rcu_read_lock();
|
2013-02-28 13:52:24 -07:00
|
|
|
hlist_for_each_entry_rcu(blkg, &blkcg->blkg_list, blkcg_node) {
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_lock_irq(&blkg->q->queue_lock);
|
2012-04-13 13:11:33 -07:00
|
|
|
if (blkcg_policy_enabled(blkg->q, pol))
|
2012-04-16 13:57:26 -07:00
|
|
|
total += prfill(sf, blkg->pd[pol->plid], data);
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&blkg->q->queue_lock);
|
2013-01-09 09:05:13 -07:00
|
|
|
}
|
|
|
|
rcu_read_unlock();
|
2012-04-01 14:38:42 -07:00
|
|
|
|
|
|
|
if (show_total)
|
|
|
|
seq_printf(sf, "Total %llu\n", (unsigned long long)total);
|
|
|
|
}
|
2012-04-01 14:38:43 -07:00
|
|
|
EXPORT_SYMBOL_GPL(blkcg_print_blkgs);
|
2012-04-01 14:38:42 -07:00
|
|
|
|
|
|
|
/**
|
|
|
|
* __blkg_prfill_u64 - prfill helper for a single u64 value
|
|
|
|
* @sf: seq_file to print to
|
2012-04-16 13:57:26 -07:00
|
|
|
* @pd: policy private data of interest
|
2012-04-01 14:38:42 -07:00
|
|
|
* @v: value to print
|
|
|
|
*
|
2022-12-06 02:33:07 -07:00
|
|
|
* Print @v to @sf for the device associated with @pd.
|
2012-04-01 14:38:42 -07:00
|
|
|
*/
|
2012-04-16 13:57:26 -07:00
|
|
|
u64 __blkg_prfill_u64(struct seq_file *sf, struct blkg_policy_data *pd, u64 v)
|
2012-04-01 14:38:42 -07:00
|
|
|
{
|
2012-04-16 13:57:26 -07:00
|
|
|
const char *dname = blkg_dev_name(pd->blkg);
|
2012-04-01 14:38:42 -07:00
|
|
|
|
|
|
|
if (!dname)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
seq_printf(sf, "%s %llu\n", dname, (unsigned long long)v);
|
|
|
|
return v;
|
|
|
|
}
|
2012-04-01 14:38:43 -07:00
|
|
|
EXPORT_SYMBOL_GPL(__blkg_prfill_u64);
|
2012-04-01 14:38:42 -07:00
|
|
|
|
2019-08-28 15:05:53 -07:00
|
|
|
/**
|
2023-04-12 17:06:47 -07:00
|
|
|
* blkg_conf_init - initialize a blkg_conf_ctx
|
|
|
|
* @ctx: blkg_conf_ctx to initialize
|
|
|
|
* @input: input string
|
2019-08-28 15:05:53 -07:00
|
|
|
*
|
2023-04-12 17:06:47 -07:00
|
|
|
* Initialize @ctx which can be used to parse blkg config input string @input.
|
|
|
|
* Once initialized, @ctx can be used with blkg_conf_open_bdev() and
|
|
|
|
* blkg_conf_prep(), and must be cleaned up with blkg_conf_exit().
|
|
|
|
*/
|
|
|
|
void blkg_conf_init(struct blkg_conf_ctx *ctx, char *input)
|
|
|
|
{
|
|
|
|
*ctx = (struct blkg_conf_ctx){ .input = input };
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(blkg_conf_init);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* blkg_conf_open_bdev - parse and open bdev for per-blkg config update
|
|
|
|
* @ctx: blkg_conf_ctx initialized with blkg_conf_init()
|
2019-08-28 15:05:53 -07:00
|
|
|
*
|
2023-04-12 17:06:47 -07:00
|
|
|
* Parse the device node prefix part, MAJ:MIN, of per-blkg config update from
|
|
|
|
* @ctx->input and get and store the matching bdev in @ctx->bdev. @ctx->body is
|
|
|
|
* set to point past the device node prefix.
|
|
|
|
*
|
|
|
|
* This function may be called multiple times on @ctx and the extra calls become
|
|
|
|
* NOOPs. blkg_conf_prep() implicitly calls this function. Use this function
|
|
|
|
* explicitly if bdev access is needed without resolving the blkcg / policy part
|
|
|
|
* of @ctx->input. Returns -errno on error.
|
2019-08-28 15:05:53 -07:00
|
|
|
*/
|
2023-04-12 17:06:47 -07:00
|
|
|
int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx)
|
2019-08-28 15:05:53 -07:00
|
|
|
{
|
2023-04-12 17:06:47 -07:00
|
|
|
char *input = ctx->input;
|
2019-08-28 15:05:53 -07:00
|
|
|
unsigned int major, minor;
|
2020-11-26 01:23:26 -07:00
|
|
|
struct block_device *bdev;
|
|
|
|
int key_len;
|
2019-08-28 15:05:53 -07:00
|
|
|
|
2023-04-12 17:06:47 -07:00
|
|
|
if (ctx->bdev)
|
|
|
|
return 0;
|
|
|
|
|
2019-08-28 15:05:53 -07:00
|
|
|
if (sscanf(input, "%u:%u%n", &major, &minor, &key_len) != 2)
|
2023-04-12 17:06:47 -07:00
|
|
|
return -EINVAL;
|
2019-08-28 15:05:53 -07:00
|
|
|
|
|
|
|
input += key_len;
|
|
|
|
if (!isspace(*input))
|
2023-04-12 17:06:47 -07:00
|
|
|
return -EINVAL;
|
2019-08-28 15:05:53 -07:00
|
|
|
input = skip_spaces(input);
|
|
|
|
|
2020-11-26 01:23:26 -07:00
|
|
|
bdev = blkdev_get_no_open(MKDEV(major, minor));
|
|
|
|
if (!bdev)
|
2023-04-12 17:06:47 -07:00
|
|
|
return -ENODEV;
|
2020-11-26 01:23:26 -07:00
|
|
|
if (bdev_is_partition(bdev)) {
|
|
|
|
blkdev_put_no_open(bdev);
|
2023-04-12 17:06:47 -07:00
|
|
|
return -ENODEV;
|
2019-08-28 15:05:53 -07:00
|
|
|
}
|
|
|
|
|
block/rq_qos: protect rq_qos apis with a new lock
commit 50e34d78815e ("block: disable the elevator int del_gendisk")
move rq_qos_exit() from disk_release() to del_gendisk(), this will
introduce some problems:
1) If rq_qos_add() is triggered by enabling iocost/iolatency through
cgroupfs, then it can concurrent with del_gendisk(), it's not safe to
write 'q->rq_qos' concurrently.
2) Activate cgroup policy that is relied on rq_qos will call
rq_qos_add() and blkcg_activate_policy(), and if rq_qos_exit() is
called in the middle, null-ptr-dereference will be triggered in
blkcg_activate_policy().
3) blkg_conf_open_bdev() can call blkdev_get_no_open() first to find the
disk, then if rq_qos_exit() from del_gendisk() is done before
rq_qos_add(), then memory will be leaked.
This patch add a new disk level mutex 'rq_qos_mutex':
1) The lock will protect rq_qos_exit() directly.
2) For wbt that doesn't relied on blk-cgroup, rq_qos_add() can only be
called from disk initialization for now because wbt can't be
destructed until rq_qos_exit(), so it's safe not to protect wbt for
now. Hoever, in case that rq_qos dynamically destruction is supported
in the furture, this patch also protect rq_qos_add() from wbt_init()
directly, this is enough because blk-sysfs already synchronize
writers with disk removal.
3) For iocost and iolatency, in order to synchronize disk removal and
cgroup configuration, the lock is held after blkdev_get_no_open()
from blkg_conf_open_bdev(), and is released in blkg_conf_exit().
In order to fix the above memory leak, disk_live() is checked after
holding the new lock.
Fixes: 50e34d78815e ("block: disable the elevator int del_gendisk")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/20230414084008.2085155-1-yukuai1@huaweicloud.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-04-14 01:40:08 -07:00
|
|
|
mutex_lock(&bdev->bd_queue->rq_qos_mutex);
|
|
|
|
if (!disk_live(bdev->bd_disk)) {
|
|
|
|
blkdev_put_no_open(bdev);
|
|
|
|
mutex_unlock(&bdev->bd_queue->rq_qos_mutex);
|
|
|
|
return -ENODEV;
|
|
|
|
}
|
|
|
|
|
2023-04-12 17:06:47 -07:00
|
|
|
ctx->body = input;
|
|
|
|
ctx->bdev = bdev;
|
|
|
|
return 0;
|
2019-08-28 15:05:53 -07:00
|
|
|
}
|
|
|
|
|
2012-04-01 14:38:43 -07:00
|
|
|
/**
|
|
|
|
* blkg_conf_prep - parse and prepare for per-blkg config update
|
|
|
|
* @blkcg: target block cgroup
|
2012-04-13 13:11:29 -07:00
|
|
|
* @pol: target policy
|
2023-04-12 17:06:47 -07:00
|
|
|
* @ctx: blkg_conf_ctx initialized with blkg_conf_init()
|
|
|
|
*
|
|
|
|
* Parse per-blkg config update from @ctx->input and initialize @ctx
|
|
|
|
* accordingly. On success, @ctx->body points to the part of @ctx->input
|
|
|
|
* following MAJ:MIN, @ctx->bdev points to the target block device and
|
|
|
|
* @ctx->blkg to the blkg being configured.
|
2012-04-01 14:38:43 -07:00
|
|
|
*
|
2023-04-12 17:06:47 -07:00
|
|
|
* blkg_conf_open_bdev() may be called on @ctx beforehand. On success, this
|
|
|
|
* function returns with queue lock held and must be followed by
|
|
|
|
* blkg_conf_exit().
|
2012-04-01 14:38:43 -07:00
|
|
|
*/
|
2012-04-16 13:57:25 -07:00
|
|
|
int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
|
2023-04-12 17:06:47 -07:00
|
|
|
struct blkg_conf_ctx *ctx)
|
2023-04-12 17:06:46 -07:00
|
|
|
__acquires(&bdev->bd_queue->queue_lock)
|
2010-04-13 01:05:49 -07:00
|
|
|
{
|
2022-09-21 11:05:01 -07:00
|
|
|
struct gendisk *disk;
|
2017-03-29 10:27:19 -07:00
|
|
|
struct request_queue *q;
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2019-08-28 15:05:53 -07:00
|
|
|
int ret;
|
2015-08-18 14:55:31 -07:00
|
|
|
|
2023-04-12 17:06:47 -07:00
|
|
|
ret = blkg_conf_open_bdev(ctx);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
disk = ctx->bdev->bd_disk;
|
2022-09-21 11:05:01 -07:00
|
|
|
q = disk->queue;
|
2012-04-13 13:11:29 -07:00
|
|
|
|
2021-10-19 18:40:36 -07:00
|
|
|
/*
|
|
|
|
* blkcg_deactivate_policy() requires queue to be frozen, we can grab
|
|
|
|
* q_usage_counter to prevent concurrent with blkcg_deactivate_policy().
|
|
|
|
*/
|
|
|
|
ret = blk_queue_enter(q, 0);
|
|
|
|
if (ret)
|
2021-11-01 19:07:05 -07:00
|
|
|
goto fail;
|
2021-10-19 18:40:36 -07:00
|
|
|
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
2012-03-05 14:15:07 -07:00
|
|
|
|
2022-09-21 11:04:49 -07:00
|
|
|
if (!blkcg_policy_enabled(q, pol)) {
|
|
|
|
ret = -EOPNOTSUPP;
|
2017-03-29 10:27:19 -07:00
|
|
|
goto fail_unlock;
|
|
|
|
}
|
|
|
|
|
2022-09-21 11:04:49 -07:00
|
|
|
blkg = blkg_lookup(blkcg, q);
|
2022-09-26 23:54:25 -07:00
|
|
|
if (blkg)
|
2017-03-29 10:27:19 -07:00
|
|
|
goto success;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create blkgs walking down from blkcg_root to @blkcg, so that all
|
|
|
|
* non-root blkgs have access to their parents.
|
|
|
|
*/
|
|
|
|
while (true) {
|
|
|
|
struct blkcg *pos = blkcg;
|
|
|
|
struct blkcg *parent;
|
|
|
|
struct blkcg_gq *new_blkg;
|
|
|
|
|
|
|
|
parent = blkcg_parent(blkcg);
|
2022-09-21 11:04:47 -07:00
|
|
|
while (parent && !blkg_lookup(parent, q)) {
|
2017-03-29 10:27:19 -07:00
|
|
|
pos = parent;
|
|
|
|
parent = blkcg_parent(parent);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Drop locks to do new blkg allocation with GFP_KERNEL. */
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2017-03-29 10:27:19 -07:00
|
|
|
|
2022-09-21 11:05:01 -07:00
|
|
|
new_blkg = blkg_alloc(pos, disk, GFP_KERNEL);
|
2017-03-29 10:27:19 -07:00
|
|
|
if (unlikely(!new_blkg)) {
|
|
|
|
ret = -ENOMEM;
|
2021-11-01 19:07:05 -07:00
|
|
|
goto fail_exit_queue;
|
2010-09-15 14:06:36 -07:00
|
|
|
}
|
2012-04-01 14:38:43 -07:00
|
|
|
|
2020-10-22 13:58:42 -07:00
|
|
|
if (radix_tree_preload(GFP_KERNEL)) {
|
|
|
|
blkg_free(new_blkg);
|
|
|
|
ret = -ENOMEM;
|
2021-11-01 19:07:05 -07:00
|
|
|
goto fail_exit_queue;
|
2020-10-22 13:58:42 -07:00
|
|
|
}
|
|
|
|
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
2017-03-29 10:27:19 -07:00
|
|
|
|
2022-09-21 11:04:49 -07:00
|
|
|
if (!blkcg_policy_enabled(q, pol)) {
|
2020-10-22 13:58:41 -07:00
|
|
|
blkg_free(new_blkg);
|
2022-09-21 11:04:49 -07:00
|
|
|
ret = -EOPNOTSUPP;
|
2020-10-22 13:58:42 -07:00
|
|
|
goto fail_preloaded;
|
2017-03-29 10:27:19 -07:00
|
|
|
}
|
|
|
|
|
2022-09-21 11:04:49 -07:00
|
|
|
blkg = blkg_lookup(pos, q);
|
2017-03-29 10:27:19 -07:00
|
|
|
if (blkg) {
|
|
|
|
blkg_free(new_blkg);
|
|
|
|
} else {
|
2022-09-21 11:05:01 -07:00
|
|
|
blkg = blkg_create(pos, disk, new_blkg);
|
2019-06-05 07:24:27 -07:00
|
|
|
if (IS_ERR(blkg)) {
|
2017-03-29 10:27:19 -07:00
|
|
|
ret = PTR_ERR(blkg);
|
2020-10-22 13:58:42 -07:00
|
|
|
goto fail_preloaded;
|
2017-03-29 10:27:19 -07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-10-22 13:58:42 -07:00
|
|
|
radix_tree_preload_end();
|
|
|
|
|
2017-03-29 10:27:19 -07:00
|
|
|
if (pos == blkcg)
|
|
|
|
goto success;
|
|
|
|
}
|
|
|
|
success:
|
2021-10-19 18:40:36 -07:00
|
|
|
blk_queue_exit(q);
|
2012-04-01 14:38:43 -07:00
|
|
|
ctx->blkg = blkg;
|
2012-04-01 14:38:43 -07:00
|
|
|
return 0;
|
2017-03-29 10:27:19 -07:00
|
|
|
|
2020-10-22 13:58:42 -07:00
|
|
|
fail_preloaded:
|
|
|
|
radix_tree_preload_end();
|
2017-03-29 10:27:19 -07:00
|
|
|
fail_unlock:
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2021-11-01 19:07:05 -07:00
|
|
|
fail_exit_queue:
|
|
|
|
blk_queue_exit(q);
|
2017-03-29 10:27:19 -07:00
|
|
|
fail:
|
|
|
|
/*
|
|
|
|
* If queue was bypassing, we should retry. Do so after a
|
|
|
|
* short msleep(). It isn't strictly necessary but queue
|
|
|
|
* can be bypassing for some time and it's always nice to
|
|
|
|
* avoid busy looping.
|
|
|
|
*/
|
|
|
|
if (ret == -EBUSY) {
|
|
|
|
msleep(10);
|
|
|
|
ret = restart_syscall();
|
|
|
|
}
|
|
|
|
return ret;
|
2010-04-13 01:05:49 -07:00
|
|
|
}
|
2019-09-14 10:31:50 -07:00
|
|
|
EXPORT_SYMBOL_GPL(blkg_conf_prep);
|
2010-04-13 01:05:49 -07:00
|
|
|
|
2012-04-01 14:38:43 -07:00
|
|
|
/**
|
2023-04-12 17:06:47 -07:00
|
|
|
* blkg_conf_exit - clean up per-blkg config update
|
|
|
|
* @ctx: blkg_conf_ctx initialized with blkg_conf_init()
|
2012-04-01 14:38:43 -07:00
|
|
|
*
|
2023-04-12 17:06:47 -07:00
|
|
|
* Clean up after per-blkg config update. This function must be called on all
|
|
|
|
* blkg_conf_ctx's initialized with blkg_conf_init().
|
2012-04-01 14:38:43 -07:00
|
|
|
*/
|
2023-04-12 17:06:47 -07:00
|
|
|
void blkg_conf_exit(struct blkg_conf_ctx *ctx)
|
2023-04-12 17:06:46 -07:00
|
|
|
__releases(&ctx->bdev->bd_queue->queue_lock)
|
block/rq_qos: protect rq_qos apis with a new lock
commit 50e34d78815e ("block: disable the elevator int del_gendisk")
move rq_qos_exit() from disk_release() to del_gendisk(), this will
introduce some problems:
1) If rq_qos_add() is triggered by enabling iocost/iolatency through
cgroupfs, then it can concurrent with del_gendisk(), it's not safe to
write 'q->rq_qos' concurrently.
2) Activate cgroup policy that is relied on rq_qos will call
rq_qos_add() and blkcg_activate_policy(), and if rq_qos_exit() is
called in the middle, null-ptr-dereference will be triggered in
blkcg_activate_policy().
3) blkg_conf_open_bdev() can call blkdev_get_no_open() first to find the
disk, then if rq_qos_exit() from del_gendisk() is done before
rq_qos_add(), then memory will be leaked.
This patch add a new disk level mutex 'rq_qos_mutex':
1) The lock will protect rq_qos_exit() directly.
2) For wbt that doesn't relied on blk-cgroup, rq_qos_add() can only be
called from disk initialization for now because wbt can't be
destructed until rq_qos_exit(), so it's safe not to protect wbt for
now. Hoever, in case that rq_qos dynamically destruction is supported
in the furture, this patch also protect rq_qos_add() from wbt_init()
directly, this is enough because blk-sysfs already synchronize
writers with disk removal.
3) For iocost and iolatency, in order to synchronize disk removal and
cgroup configuration, the lock is held after blkdev_get_no_open()
from blkg_conf_open_bdev(), and is released in blkg_conf_exit().
In order to fix the above memory leak, disk_live() is checked after
holding the new lock.
Fixes: 50e34d78815e ("block: disable the elevator int del_gendisk")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/20230414084008.2085155-1-yukuai1@huaweicloud.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-04-14 01:40:08 -07:00
|
|
|
__releases(&ctx->bdev->bd_queue->rq_qos_mutex)
|
2010-04-13 01:05:49 -07:00
|
|
|
{
|
2023-04-12 17:06:47 -07:00
|
|
|
if (ctx->blkg) {
|
|
|
|
spin_unlock_irq(&bdev_get_queue(ctx->bdev)->queue_lock);
|
|
|
|
ctx->blkg = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (ctx->bdev) {
|
block/rq_qos: protect rq_qos apis with a new lock
commit 50e34d78815e ("block: disable the elevator int del_gendisk")
move rq_qos_exit() from disk_release() to del_gendisk(), this will
introduce some problems:
1) If rq_qos_add() is triggered by enabling iocost/iolatency through
cgroupfs, then it can concurrent with del_gendisk(), it's not safe to
write 'q->rq_qos' concurrently.
2) Activate cgroup policy that is relied on rq_qos will call
rq_qos_add() and blkcg_activate_policy(), and if rq_qos_exit() is
called in the middle, null-ptr-dereference will be triggered in
blkcg_activate_policy().
3) blkg_conf_open_bdev() can call blkdev_get_no_open() first to find the
disk, then if rq_qos_exit() from del_gendisk() is done before
rq_qos_add(), then memory will be leaked.
This patch add a new disk level mutex 'rq_qos_mutex':
1) The lock will protect rq_qos_exit() directly.
2) For wbt that doesn't relied on blk-cgroup, rq_qos_add() can only be
called from disk initialization for now because wbt can't be
destructed until rq_qos_exit(), so it's safe not to protect wbt for
now. Hoever, in case that rq_qos dynamically destruction is supported
in the furture, this patch also protect rq_qos_add() from wbt_init()
directly, this is enough because blk-sysfs already synchronize
writers with disk removal.
3) For iocost and iolatency, in order to synchronize disk removal and
cgroup configuration, the lock is held after blkdev_get_no_open()
from blkg_conf_open_bdev(), and is released in blkg_conf_exit().
In order to fix the above memory leak, disk_live() is checked after
holding the new lock.
Fixes: 50e34d78815e ("block: disable the elevator int del_gendisk")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/20230414084008.2085155-1-yukuai1@huaweicloud.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-04-14 01:40:08 -07:00
|
|
|
mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex);
|
2023-04-12 17:06:47 -07:00
|
|
|
blkdev_put_no_open(ctx->bdev);
|
|
|
|
ctx->body = NULL;
|
|
|
|
ctx->bdev = NULL;
|
|
|
|
}
|
2010-04-13 01:05:49 -07:00
|
|
|
}
|
2023-04-12 17:06:47 -07:00
|
|
|
EXPORT_SYMBOL_GPL(blkg_conf_exit);
|
2010-04-13 01:05:49 -07:00
|
|
|
|
2020-06-01 13:11:43 -07:00
|
|
|
static void blkg_iostat_add(struct blkg_iostat *dst, struct blkg_iostat *src)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < BLKG_IOSTAT_NR; i++) {
|
|
|
|
dst->bytes[i] += src->bytes[i];
|
|
|
|
dst->ios[i] += src->ios[i];
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void blkg_iostat_sub(struct blkg_iostat *dst, struct blkg_iostat *src)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < BLKG_IOSTAT_NR; i++) {
|
|
|
|
dst->bytes[i] -= src->bytes[i];
|
|
|
|
dst->ios[i] -= src->ios[i];
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-06-29 00:09:16 -07:00
|
|
|
static void blkcg_iostat_update(struct blkcg_gq *blkg, struct blkg_iostat *cur,
|
|
|
|
struct blkg_iostat *last)
|
|
|
|
{
|
|
|
|
struct blkg_iostat delta;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
/* propagate percpu delta to global */
|
|
|
|
flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
|
|
|
|
blkg_iostat_set(&delta, cur);
|
|
|
|
blkg_iostat_sub(&delta, last);
|
|
|
|
blkg_iostat_add(&blkg->iostat.cur, &delta);
|
|
|
|
blkg_iostat_add(last, &delta);
|
|
|
|
u64_stats_update_end_irqrestore(&blkg->iostat.sync, flags);
|
|
|
|
}
|
|
|
|
|
2023-06-09 16:42:49 -07:00
|
|
|
static void __blkcg_rstat_flush(struct blkcg *blkcg, int cpu)
|
2020-06-01 13:11:43 -07:00
|
|
|
{
|
2022-11-04 17:59:01 -07:00
|
|
|
struct llist_head *lhead = per_cpu_ptr(blkcg->lhead, cpu);
|
|
|
|
struct llist_node *lnode;
|
|
|
|
struct blkg_iostat_set *bisc, *next_bisc;
|
2023-06-22 01:42:49 -07:00
|
|
|
unsigned long flags;
|
2020-06-01 13:11:43 -07:00
|
|
|
|
|
|
|
rcu_read_lock();
|
|
|
|
|
2022-11-04 17:59:01 -07:00
|
|
|
lnode = llist_del_all(lhead);
|
|
|
|
if (!lnode)
|
|
|
|
goto out;
|
|
|
|
|
2023-06-09 16:42:49 -07:00
|
|
|
/*
|
|
|
|
* For covering concurrent parent blkg update from blkg_release().
|
|
|
|
*
|
|
|
|
* When flushing from cgroup, cgroup_rstat_lock is always held, so
|
|
|
|
* this lock won't cause contention most of time.
|
|
|
|
*/
|
2023-06-22 01:42:49 -07:00
|
|
|
raw_spin_lock_irqsave(&blkg_stat_lock, flags);
|
2023-06-09 16:42:49 -07:00
|
|
|
|
2022-11-04 17:59:01 -07:00
|
|
|
/*
|
|
|
|
* Iterate only the iostat_cpu's queued in the lockless list.
|
|
|
|
*/
|
|
|
|
llist_for_each_entry_safe(bisc, next_bisc, lnode, lnode) {
|
|
|
|
struct blkcg_gq *blkg = bisc->blkg;
|
2020-06-01 13:11:43 -07:00
|
|
|
struct blkcg_gq *parent = blkg->parent;
|
2022-06-29 00:09:16 -07:00
|
|
|
struct blkg_iostat cur;
|
2020-06-01 13:11:43 -07:00
|
|
|
unsigned int seq;
|
|
|
|
|
2024-05-14 18:31:57 -07:00
|
|
|
/*
|
|
|
|
* Order assignment of `next_bisc` from `bisc->lnode.next` in
|
|
|
|
* llist_for_each_entry_safe and clearing `bisc->lqueued` for
|
|
|
|
* avoiding to assign `next_bisc` with new next pointer added
|
|
|
|
* in blk_cgroup_bio_start() in case of re-ordering.
|
|
|
|
*
|
|
|
|
* The pair barrier is implied in llist_add() in blk_cgroup_bio_start().
|
|
|
|
*/
|
|
|
|
smp_mb();
|
|
|
|
|
2022-11-04 17:59:01 -07:00
|
|
|
WRITE_ONCE(bisc->lqueued, false);
|
2024-05-15 07:30:59 -07:00
|
|
|
if (bisc == &blkg->iostat)
|
|
|
|
goto propagate_up; /* propagate up to parent only */
|
2022-11-04 17:59:01 -07:00
|
|
|
|
2020-06-01 13:11:43 -07:00
|
|
|
/* fetch the current per-cpu values */
|
|
|
|
do {
|
|
|
|
seq = u64_stats_fetch_begin(&bisc->sync);
|
|
|
|
blkg_iostat_set(&cur, &bisc->cur);
|
|
|
|
} while (u64_stats_fetch_retry(&bisc->sync, seq));
|
|
|
|
|
2022-06-29 00:09:16 -07:00
|
|
|
blkcg_iostat_update(blkg, &cur, &bisc->last);
|
2020-06-01 13:11:43 -07:00
|
|
|
|
2024-05-15 07:30:59 -07:00
|
|
|
propagate_up:
|
cgroup: rstat: punt root-level optimization to individual controllers
Current users of the rstat code can source root-level statistics from
the native counters of their respective subsystem, allowing them to
forego aggregation at the root level. This optimization is currently
implemented inside the generic rstat code, which doesn't track the root
cgroup and doesn't invoke the subsystem flush callbacks on it.
However, the memory controller cannot do this optimization, because
cgroup1 breaks out memory specifically for the local level, including at
the root level. In preparation for the memory controller switching to
rstat, move the optimization from rstat core to the controllers.
Afterwards, rstat will always track the root cgroup for changes and
invoke the subsystem callbacks on it; and it's up to the subsystem to
special-case and skip aggregation of the root cgroup if it can source
this information through other, cheaper means.
This is the case for the io controller and the cgroup base stats. In
their respective flush callbacks, check whether the parent is the root
cgroup, and if so, skip the unnecessary upward propagation.
The extra cost of tracking the root cgroup is negligible: on stat
changes, we actually remove a branch that checks for the root. The
queueing for a flush touches only per-cpu data, and only the first stat
change since a flush requires a (per-cpu) lock.
Link: https://lkml.kernel.org/r/20210209163304.77088-6-hannes@cmpxchg.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Roman Gushchin <guro@fb.com>
Cc: Shakeel Butt <shakeelb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-04-29 22:56:23 -07:00
|
|
|
/* propagate global delta to parent (unless that's root) */
|
2024-05-15 07:30:59 -07:00
|
|
|
if (parent && parent->parent) {
|
2022-06-29 00:09:16 -07:00
|
|
|
blkcg_iostat_update(parent, &blkg->iostat.cur,
|
|
|
|
&blkg->iostat.last);
|
2024-05-15 07:30:59 -07:00
|
|
|
/*
|
|
|
|
* Queue parent->iostat to its blkcg's lockless
|
|
|
|
* list to propagate up to the grandparent if the
|
|
|
|
* iostat hasn't been queued yet.
|
|
|
|
*/
|
|
|
|
if (!parent->iostat.lqueued) {
|
|
|
|
struct llist_head *plhead;
|
|
|
|
|
|
|
|
plhead = per_cpu_ptr(parent->blkcg->lhead, cpu);
|
|
|
|
llist_add(&parent->iostat.lnode, plhead);
|
|
|
|
parent->iostat.lqueued = true;
|
|
|
|
}
|
|
|
|
}
|
2020-06-01 13:11:43 -07:00
|
|
|
}
|
2023-06-22 01:42:49 -07:00
|
|
|
raw_spin_unlock_irqrestore(&blkg_stat_lock, flags);
|
2022-11-04 17:59:01 -07:00
|
|
|
out:
|
2020-06-01 13:11:43 -07:00
|
|
|
rcu_read_unlock();
|
|
|
|
}
|
|
|
|
|
2023-06-09 16:42:49 -07:00
|
|
|
static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu)
|
|
|
|
{
|
|
|
|
/* Root-level stats are sourced from system-wide IO stats */
|
|
|
|
if (cgroup_parent(css->cgroup))
|
|
|
|
__blkcg_rstat_flush(css_to_blkcg(css), cpu);
|
|
|
|
}
|
|
|
|
|
2020-06-01 13:12:05 -07:00
|
|
|
/*
|
cgroup: rstat: punt root-level optimization to individual controllers
Current users of the rstat code can source root-level statistics from
the native counters of their respective subsystem, allowing them to
forego aggregation at the root level. This optimization is currently
implemented inside the generic rstat code, which doesn't track the root
cgroup and doesn't invoke the subsystem flush callbacks on it.
However, the memory controller cannot do this optimization, because
cgroup1 breaks out memory specifically for the local level, including at
the root level. In preparation for the memory controller switching to
rstat, move the optimization from rstat core to the controllers.
Afterwards, rstat will always track the root cgroup for changes and
invoke the subsystem callbacks on it; and it's up to the subsystem to
special-case and skip aggregation of the root cgroup if it can source
this information through other, cheaper means.
This is the case for the io controller and the cgroup base stats. In
their respective flush callbacks, check whether the parent is the root
cgroup, and if so, skip the unnecessary upward propagation.
The extra cost of tracking the root cgroup is negligible: on stat
changes, we actually remove a branch that checks for the root. The
queueing for a flush touches only per-cpu data, and only the first stat
change since a flush requires a (per-cpu) lock.
Link: https://lkml.kernel.org/r/20210209163304.77088-6-hannes@cmpxchg.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Roman Gushchin <guro@fb.com>
Cc: Shakeel Butt <shakeelb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-04-29 22:56:23 -07:00
|
|
|
* We source root cgroup stats from the system-wide stats to avoid
|
|
|
|
* tracking the same information twice and incurring overhead when no
|
|
|
|
* cgroups are defined. For that reason, cgroup_rstat_flush in
|
|
|
|
* blkcg_print_stat does not actually fill out the iostat in the root
|
|
|
|
* cgroup's blkcg_gq.
|
2020-06-01 13:12:05 -07:00
|
|
|
*
|
|
|
|
* However, we would like to re-use the printing code between the root and
|
|
|
|
* non-root cgroups to the extent possible. For that reason, we simulate
|
|
|
|
* flushing the root cgroup's stats by explicitly filling in the iostat
|
|
|
|
* with disk level statistics.
|
|
|
|
*/
|
|
|
|
static void blkcg_fill_root_iostats(void)
|
|
|
|
{
|
|
|
|
struct class_dev_iter iter;
|
|
|
|
struct device *dev;
|
|
|
|
|
|
|
|
class_dev_iter_init(&iter, &block_class, NULL, &disk_type);
|
|
|
|
while ((dev = class_dev_iter_next(&iter))) {
|
2020-11-27 08:43:51 -07:00
|
|
|
struct block_device *bdev = dev_to_bdev(dev);
|
2022-09-21 11:04:46 -07:00
|
|
|
struct blkcg_gq *blkg = bdev->bd_disk->queue->root_blkg;
|
2020-06-01 13:12:05 -07:00
|
|
|
struct blkg_iostat tmp;
|
|
|
|
int cpu;
|
2022-02-13 01:59:02 -07:00
|
|
|
unsigned long flags;
|
2020-06-01 13:12:05 -07:00
|
|
|
|
|
|
|
memset(&tmp, 0, sizeof(tmp));
|
|
|
|
for_each_possible_cpu(cpu) {
|
|
|
|
struct disk_stats *cpu_dkstats;
|
|
|
|
|
2020-11-27 08:43:51 -07:00
|
|
|
cpu_dkstats = per_cpu_ptr(bdev->bd_stats, cpu);
|
2020-06-01 13:12:05 -07:00
|
|
|
tmp.ios[BLKG_IOSTAT_READ] +=
|
|
|
|
cpu_dkstats->ios[STAT_READ];
|
|
|
|
tmp.ios[BLKG_IOSTAT_WRITE] +=
|
|
|
|
cpu_dkstats->ios[STAT_WRITE];
|
|
|
|
tmp.ios[BLKG_IOSTAT_DISCARD] +=
|
|
|
|
cpu_dkstats->ios[STAT_DISCARD];
|
|
|
|
// convert sectors to bytes
|
|
|
|
tmp.bytes[BLKG_IOSTAT_READ] +=
|
|
|
|
cpu_dkstats->sectors[STAT_READ] << 9;
|
|
|
|
tmp.bytes[BLKG_IOSTAT_WRITE] +=
|
|
|
|
cpu_dkstats->sectors[STAT_WRITE] << 9;
|
|
|
|
tmp.bytes[BLKG_IOSTAT_DISCARD] +=
|
|
|
|
cpu_dkstats->sectors[STAT_DISCARD] << 9;
|
|
|
|
}
|
2022-02-13 01:59:02 -07:00
|
|
|
|
|
|
|
flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
|
|
|
|
blkg_iostat_set(&blkg->iostat.cur, &tmp);
|
|
|
|
u64_stats_update_end_irqrestore(&blkg->iostat.sync, flags);
|
2020-06-01 13:12:05 -07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
static void blkcg_print_one_stat(struct blkcg_gq *blkg, struct seq_file *s)
|
2015-08-18 14:55:34 -07:00
|
|
|
{
|
2021-08-10 08:26:22 -07:00
|
|
|
struct blkg_iostat_set *bis = &blkg->iostat;
|
|
|
|
u64 rbytes, wbytes, rios, wios, dbytes, dios;
|
|
|
|
const char *dname;
|
|
|
|
unsigned seq;
|
|
|
|
int i;
|
2020-06-01 13:12:05 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
if (!blkg->online)
|
|
|
|
return;
|
2015-08-18 14:55:34 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
dname = blkg_dev_name(blkg);
|
|
|
|
if (!dname)
|
|
|
|
return;
|
2015-08-18 14:55:34 -07:00
|
|
|
|
2021-08-10 08:26:23 -07:00
|
|
|
seq_printf(s, "%s ", dname);
|
2019-11-05 09:09:51 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
do {
|
|
|
|
seq = u64_stats_fetch_begin(&bis->sync);
|
2019-11-05 09:09:51 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
rbytes = bis->cur.bytes[BLKG_IOSTAT_READ];
|
|
|
|
wbytes = bis->cur.bytes[BLKG_IOSTAT_WRITE];
|
|
|
|
dbytes = bis->cur.bytes[BLKG_IOSTAT_DISCARD];
|
|
|
|
rios = bis->cur.ios[BLKG_IOSTAT_READ];
|
|
|
|
wios = bis->cur.ios[BLKG_IOSTAT_WRITE];
|
|
|
|
dios = bis->cur.ios[BLKG_IOSTAT_DISCARD];
|
|
|
|
} while (u64_stats_fetch_retry(&bis->sync, seq));
|
2015-08-18 14:55:34 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
if (rbytes || wbytes || rios || wios) {
|
2021-08-10 08:26:23 -07:00
|
|
|
seq_printf(s, "rbytes=%llu wbytes=%llu rios=%llu wios=%llu dbytes=%llu dios=%llu",
|
2021-08-10 08:26:22 -07:00
|
|
|
rbytes, wbytes, rios, wios,
|
|
|
|
dbytes, dios);
|
|
|
|
}
|
2018-07-03 08:14:52 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
if (blkcg_debug_stats && atomic_read(&blkg->use_delay)) {
|
2021-08-10 08:26:23 -07:00
|
|
|
seq_printf(s, " use_delay=%d delay_nsec=%llu",
|
2021-08-10 08:26:22 -07:00
|
|
|
atomic_read(&blkg->use_delay),
|
|
|
|
atomic64_read(&blkg->delay_nsec));
|
|
|
|
}
|
2015-08-18 14:55:34 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++) {
|
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
2015-08-18 14:55:34 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
if (!blkg->pd[i] || !pol->pd_stat_fn)
|
|
|
|
continue;
|
2018-07-03 08:14:52 -07:00
|
|
|
|
2022-01-11 01:31:59 -07:00
|
|
|
pol->pd_stat_fn(blkg->pd[i], s);
|
2021-08-10 08:26:22 -07:00
|
|
|
}
|
2018-07-03 08:14:55 -07:00
|
|
|
|
2022-01-11 01:31:59 -07:00
|
|
|
seq_puts(s, "\n");
|
2021-08-10 08:26:22 -07:00
|
|
|
}
|
2018-07-03 08:14:52 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
static int blkcg_print_stat(struct seq_file *sf, void *v)
|
|
|
|
{
|
|
|
|
struct blkcg *blkcg = css_to_blkcg(seq_css(sf));
|
|
|
|
struct blkcg_gq *blkg;
|
2018-07-03 08:14:52 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
if (!seq_css(sf)->parent)
|
|
|
|
blkcg_fill_root_iostats();
|
|
|
|
else
|
|
|
|
cgroup_rstat_flush(blkcg->css.cgroup);
|
2019-07-16 07:58:31 -07:00
|
|
|
|
2021-08-10 08:26:22 -07:00
|
|
|
rcu_read_lock();
|
|
|
|
hlist_for_each_entry_rcu(blkg, &blkcg->blkg_list, blkcg_node) {
|
|
|
|
spin_lock_irq(&blkg->q->queue_lock);
|
|
|
|
blkcg_print_one_stat(blkg, sf);
|
2019-11-05 09:09:51 -07:00
|
|
|
spin_unlock_irq(&blkg->q->queue_lock);
|
2015-08-18 14:55:34 -07:00
|
|
|
}
|
|
|
|
rcu_read_unlock();
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-06-14 08:04:32 -07:00
|
|
|
static struct cftype blkcg_files[] = {
|
2015-08-18 14:55:34 -07:00
|
|
|
{
|
|
|
|
.name = "stat",
|
|
|
|
.seq_show = blkcg_print_stat,
|
|
|
|
},
|
|
|
|
{ } /* terminate */
|
|
|
|
};
|
|
|
|
|
2016-06-14 08:04:32 -07:00
|
|
|
static struct cftype blkcg_legacy_files[] = {
|
2010-04-08 23:31:19 -07:00
|
|
|
{
|
|
|
|
.name = "reset_stats",
|
2012-04-16 13:57:25 -07:00
|
|
|
.write_u64 = blkcg_reset_stats,
|
2009-12-03 10:59:49 -07:00
|
|
|
},
|
2012-04-01 12:09:55 -07:00
|
|
|
{ } /* terminate */
|
2009-12-03 10:59:42 -07:00
|
|
|
};
|
|
|
|
|
2022-04-19 21:27:15 -07:00
|
|
|
#ifdef CONFIG_CGROUP_WRITEBACK
|
|
|
|
struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css)
|
|
|
|
{
|
|
|
|
return &css_to_blkcg(css)->cgwb_list;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2018-08-31 13:22:43 -07:00
|
|
|
/*
|
|
|
|
* blkcg destruction is a three-stage process.
|
|
|
|
*
|
|
|
|
* 1. Destruction starts. The blkcg_css_offline() callback is invoked
|
|
|
|
* which offlines writeback. Here we tie the next stage of blkg destruction
|
|
|
|
* to the completion of writeback associated with the blkcg. This lets us
|
|
|
|
* avoid punting potentially large amounts of outstanding writeback to root
|
|
|
|
* while maintaining any ongoing policies. The next stage is triggered when
|
|
|
|
* the nr_cgwbs count goes to zero.
|
|
|
|
*
|
|
|
|
* 2. When the nr_cgwbs count goes to zero, blkcg_destroy_blkgs() is called
|
|
|
|
* and handles the destruction of blkgs. Here the css reference held by
|
|
|
|
* the blkg is put back eventually allowing blkcg_css_free() to be called.
|
|
|
|
* This work may occur in cgwb_release_workfn() on the cgwb_release
|
|
|
|
* workqueue. Any submitted ios that fail to get the blkg ref will be
|
|
|
|
* punted to the root_blkg.
|
|
|
|
*
|
|
|
|
* 3. Once the blkcg ref count goes to zero, blkcg_css_free() is called.
|
|
|
|
* This finally frees the blkcg.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/**
|
|
|
|
* blkcg_destroy_blkgs - responsible for shooting down blkgs
|
|
|
|
* @blkcg: blkcg of interest
|
|
|
|
*
|
|
|
|
* blkgs should be removed while holding both q and blkcg locks. As blkcg lock
|
|
|
|
* is nested inside q lock, this function performs reverse double lock dancing.
|
|
|
|
* Destroying the blkgs releases the reference held on the blkcg's css allowing
|
|
|
|
* blkcg_css_free to eventually be called.
|
|
|
|
*
|
|
|
|
* This is the blkcg counterpart of ioc_release_fn().
|
|
|
|
*/
|
2022-04-19 21:27:14 -07:00
|
|
|
static void blkcg_destroy_blkgs(struct blkcg *blkcg)
|
2018-08-31 13:22:43 -07:00
|
|
|
{
|
2021-01-27 22:58:15 -07:00
|
|
|
might_sleep();
|
|
|
|
|
2012-03-05 14:15:21 -07:00
|
|
|
spin_lock_irq(&blkcg->lock);
|
2012-03-05 14:15:11 -07:00
|
|
|
|
blk-throttle: fix race between blkcg_bio_issue_check() and cgroup_rmdir()
We've triggered a WARNING in blk_throtl_bio() when throttling writeback
io, which complains blkg->refcnt is already 0 when calling blkg_get(),
and then kernel crashes with invalid page request.
After investigating this issue, we've found it is caused by a race
between blkcg_bio_issue_check() and cgroup_rmdir(), which is described
below:
writeback kworker cgroup_rmdir
cgroup_destroy_locked
kill_css
css_killed_ref_fn
css_killed_work_fn
offline_css
blkcg_css_offline
blkcg_bio_issue_check
rcu_read_lock
blkg_lookup
spin_trylock(q->queue_lock)
blkg_destroy
spin_unlock(q->queue_lock)
blk_throtl_bio
spin_lock_irq(q->queue_lock)
...
spin_unlock_irq(q->queue_lock)
rcu_read_unlock
Since rcu can only prevent blkg from releasing when it is being used,
the blkg->refcnt can be decreased to 0 during blkg_destroy() and schedule
blkg release.
Then trying to blkg_get() in blk_throtl_bio() will complains the WARNING.
And then the corresponding blkg_put() will schedule blkg release again,
which result in double free.
This race is introduced by commit ae1188963611 ("blkcg: consolidate blkg
creation in blkcg_bio_issue_check()"). Before this commit, it will
lookup first and then try to lookup/create again with queue_lock. Since
revive this logic is a bit drastic, so fix it by only offlining pd during
blkcg_css_offline(), and move the rest destruction (especially
blkg_put()) into blkcg_css_free(), which should be the right way as
discussed.
Fixes: ae1188963611 ("blkcg: consolidate blkg creation in blkcg_bio_issue_check()")
Reported-by: Jiufei Xue <jiufei.xue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15 23:51:27 -07:00
|
|
|
while (!hlist_empty(&blkcg->blkg_list)) {
|
|
|
|
struct blkcg_gq *blkg = hlist_entry(blkcg->blkg_list.first,
|
2018-08-31 13:22:42 -07:00
|
|
|
struct blkcg_gq, blkcg_node);
|
blk-throttle: fix race between blkcg_bio_issue_check() and cgroup_rmdir()
We've triggered a WARNING in blk_throtl_bio() when throttling writeback
io, which complains blkg->refcnt is already 0 when calling blkg_get(),
and then kernel crashes with invalid page request.
After investigating this issue, we've found it is caused by a race
between blkcg_bio_issue_check() and cgroup_rmdir(), which is described
below:
writeback kworker cgroup_rmdir
cgroup_destroy_locked
kill_css
css_killed_ref_fn
css_killed_work_fn
offline_css
blkcg_css_offline
blkcg_bio_issue_check
rcu_read_lock
blkg_lookup
spin_trylock(q->queue_lock)
blkg_destroy
spin_unlock(q->queue_lock)
blk_throtl_bio
spin_lock_irq(q->queue_lock)
...
spin_unlock_irq(q->queue_lock)
rcu_read_unlock
Since rcu can only prevent blkg from releasing when it is being used,
the blkg->refcnt can be decreased to 0 during blkg_destroy() and schedule
blkg release.
Then trying to blkg_get() in blk_throtl_bio() will complains the WARNING.
And then the corresponding blkg_put() will schedule blkg release again,
which result in double free.
This race is introduced by commit ae1188963611 ("blkcg: consolidate blkg
creation in blkcg_bio_issue_check()"). Before this commit, it will
lookup first and then try to lookup/create again with queue_lock. Since
revive this logic is a bit drastic, so fix it by only offlining pd during
blkcg_css_offline(), and move the rest destruction (especially
blkg_put()) into blkcg_css_free(), which should be the right way as
discussed.
Fixes: ae1188963611 ("blkcg: consolidate blkg creation in blkcg_bio_issue_check()")
Reported-by: Jiufei Xue <jiufei.xue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15 23:51:27 -07:00
|
|
|
struct request_queue *q = blkg->q;
|
|
|
|
|
2021-01-27 22:58:15 -07:00
|
|
|
if (need_resched() || !spin_trylock(&q->queue_lock)) {
|
|
|
|
/*
|
|
|
|
* Given that the system can accumulate a huge number
|
|
|
|
* of blkgs in pathological cases, check to see if we
|
|
|
|
* need to rescheduling to avoid softlockup.
|
|
|
|
*/
|
blk-throttle: fix race between blkcg_bio_issue_check() and cgroup_rmdir()
We've triggered a WARNING in blk_throtl_bio() when throttling writeback
io, which complains blkg->refcnt is already 0 when calling blkg_get(),
and then kernel crashes with invalid page request.
After investigating this issue, we've found it is caused by a race
between blkcg_bio_issue_check() and cgroup_rmdir(), which is described
below:
writeback kworker cgroup_rmdir
cgroup_destroy_locked
kill_css
css_killed_ref_fn
css_killed_work_fn
offline_css
blkcg_css_offline
blkcg_bio_issue_check
rcu_read_lock
blkg_lookup
spin_trylock(q->queue_lock)
blkg_destroy
spin_unlock(q->queue_lock)
blk_throtl_bio
spin_lock_irq(q->queue_lock)
...
spin_unlock_irq(q->queue_lock)
rcu_read_unlock
Since rcu can only prevent blkg from releasing when it is being used,
the blkg->refcnt can be decreased to 0 during blkg_destroy() and schedule
blkg release.
Then trying to blkg_get() in blk_throtl_bio() will complains the WARNING.
And then the corresponding blkg_put() will schedule blkg release again,
which result in double free.
This race is introduced by commit ae1188963611 ("blkcg: consolidate blkg
creation in blkcg_bio_issue_check()"). Before this commit, it will
lookup first and then try to lookup/create again with queue_lock. Since
revive this logic is a bit drastic, so fix it by only offlining pd during
blkcg_css_offline(), and move the rest destruction (especially
blkg_put()) into blkcg_css_free(), which should be the right way as
discussed.
Fixes: ae1188963611 ("blkcg: consolidate blkg creation in blkcg_bio_issue_check()")
Reported-by: Jiufei Xue <jiufei.xue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15 23:51:27 -07:00
|
|
|
spin_unlock_irq(&blkcg->lock);
|
2021-01-27 22:58:15 -07:00
|
|
|
cond_resched();
|
blk-throttle: fix race between blkcg_bio_issue_check() and cgroup_rmdir()
We've triggered a WARNING in blk_throtl_bio() when throttling writeback
io, which complains blkg->refcnt is already 0 when calling blkg_get(),
and then kernel crashes with invalid page request.
After investigating this issue, we've found it is caused by a race
between blkcg_bio_issue_check() and cgroup_rmdir(), which is described
below:
writeback kworker cgroup_rmdir
cgroup_destroy_locked
kill_css
css_killed_ref_fn
css_killed_work_fn
offline_css
blkcg_css_offline
blkcg_bio_issue_check
rcu_read_lock
blkg_lookup
spin_trylock(q->queue_lock)
blkg_destroy
spin_unlock(q->queue_lock)
blk_throtl_bio
spin_lock_irq(q->queue_lock)
...
spin_unlock_irq(q->queue_lock)
rcu_read_unlock
Since rcu can only prevent blkg from releasing when it is being used,
the blkg->refcnt can be decreased to 0 during blkg_destroy() and schedule
blkg release.
Then trying to blkg_get() in blk_throtl_bio() will complains the WARNING.
And then the corresponding blkg_put() will schedule blkg release again,
which result in double free.
This race is introduced by commit ae1188963611 ("blkcg: consolidate blkg
creation in blkcg_bio_issue_check()"). Before this commit, it will
lookup first and then try to lookup/create again with queue_lock. Since
revive this logic is a bit drastic, so fix it by only offlining pd during
blkcg_css_offline(), and move the rest destruction (especially
blkg_put()) into blkcg_css_free(), which should be the right way as
discussed.
Fixes: ae1188963611 ("blkcg: consolidate blkg creation in blkcg_bio_issue_check()")
Reported-by: Jiufei Xue <jiufei.xue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15 23:51:27 -07:00
|
|
|
spin_lock_irq(&blkcg->lock);
|
2021-01-27 22:58:15 -07:00
|
|
|
continue;
|
blk-throttle: fix race between blkcg_bio_issue_check() and cgroup_rmdir()
We've triggered a WARNING in blk_throtl_bio() when throttling writeback
io, which complains blkg->refcnt is already 0 when calling blkg_get(),
and then kernel crashes with invalid page request.
After investigating this issue, we've found it is caused by a race
between blkcg_bio_issue_check() and cgroup_rmdir(), which is described
below:
writeback kworker cgroup_rmdir
cgroup_destroy_locked
kill_css
css_killed_ref_fn
css_killed_work_fn
offline_css
blkcg_css_offline
blkcg_bio_issue_check
rcu_read_lock
blkg_lookup
spin_trylock(q->queue_lock)
blkg_destroy
spin_unlock(q->queue_lock)
blk_throtl_bio
spin_lock_irq(q->queue_lock)
...
spin_unlock_irq(q->queue_lock)
rcu_read_unlock
Since rcu can only prevent blkg from releasing when it is being used,
the blkg->refcnt can be decreased to 0 during blkg_destroy() and schedule
blkg release.
Then trying to blkg_get() in blk_throtl_bio() will complains the WARNING.
And then the corresponding blkg_put() will schedule blkg release again,
which result in double free.
This race is introduced by commit ae1188963611 ("blkcg: consolidate blkg
creation in blkcg_bio_issue_check()"). Before this commit, it will
lookup first and then try to lookup/create again with queue_lock. Since
revive this logic is a bit drastic, so fix it by only offlining pd during
blkcg_css_offline(), and move the rest destruction (especially
blkg_put()) into blkcg_css_free(), which should be the right way as
discussed.
Fixes: ae1188963611 ("blkcg: consolidate blkg creation in blkcg_bio_issue_check()")
Reported-by: Jiufei Xue <jiufei.xue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15 23:51:27 -07:00
|
|
|
}
|
2021-01-27 22:58:15 -07:00
|
|
|
|
|
|
|
blkg_destroy(blkg);
|
|
|
|
spin_unlock(&q->queue_lock);
|
blk-throttle: fix race between blkcg_bio_issue_check() and cgroup_rmdir()
We've triggered a WARNING in blk_throtl_bio() when throttling writeback
io, which complains blkg->refcnt is already 0 when calling blkg_get(),
and then kernel crashes with invalid page request.
After investigating this issue, we've found it is caused by a race
between blkcg_bio_issue_check() and cgroup_rmdir(), which is described
below:
writeback kworker cgroup_rmdir
cgroup_destroy_locked
kill_css
css_killed_ref_fn
css_killed_work_fn
offline_css
blkcg_css_offline
blkcg_bio_issue_check
rcu_read_lock
blkg_lookup
spin_trylock(q->queue_lock)
blkg_destroy
spin_unlock(q->queue_lock)
blk_throtl_bio
spin_lock_irq(q->queue_lock)
...
spin_unlock_irq(q->queue_lock)
rcu_read_unlock
Since rcu can only prevent blkg from releasing when it is being used,
the blkg->refcnt can be decreased to 0 during blkg_destroy() and schedule
blkg release.
Then trying to blkg_get() in blk_throtl_bio() will complains the WARNING.
And then the corresponding blkg_put() will schedule blkg release again,
which result in double free.
This race is introduced by commit ae1188963611 ("blkcg: consolidate blkg
creation in blkcg_bio_issue_check()"). Before this commit, it will
lookup first and then try to lookup/create again with queue_lock. Since
revive this logic is a bit drastic, so fix it by only offlining pd during
blkcg_css_offline(), and move the rest destruction (especially
blkg_put()) into blkcg_css_free(), which should be the right way as
discussed.
Fixes: ae1188963611 ("blkcg: consolidate blkg creation in blkcg_bio_issue_check()")
Reported-by: Jiufei Xue <jiufei.xue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15 23:51:27 -07:00
|
|
|
}
|
2018-08-31 13:22:42 -07:00
|
|
|
|
blk-throttle: fix race between blkcg_bio_issue_check() and cgroup_rmdir()
We've triggered a WARNING in blk_throtl_bio() when throttling writeback
io, which complains blkg->refcnt is already 0 when calling blkg_get(),
and then kernel crashes with invalid page request.
After investigating this issue, we've found it is caused by a race
between blkcg_bio_issue_check() and cgroup_rmdir(), which is described
below:
writeback kworker cgroup_rmdir
cgroup_destroy_locked
kill_css
css_killed_ref_fn
css_killed_work_fn
offline_css
blkcg_css_offline
blkcg_bio_issue_check
rcu_read_lock
blkg_lookup
spin_trylock(q->queue_lock)
blkg_destroy
spin_unlock(q->queue_lock)
blk_throtl_bio
spin_lock_irq(q->queue_lock)
...
spin_unlock_irq(q->queue_lock)
rcu_read_unlock
Since rcu can only prevent blkg from releasing when it is being used,
the blkg->refcnt can be decreased to 0 during blkg_destroy() and schedule
blkg release.
Then trying to blkg_get() in blk_throtl_bio() will complains the WARNING.
And then the corresponding blkg_put() will schedule blkg release again,
which result in double free.
This race is introduced by commit ae1188963611 ("blkcg: consolidate blkg
creation in blkcg_bio_issue_check()"). Before this commit, it will
lookup first and then try to lookup/create again with queue_lock. Since
revive this logic is a bit drastic, so fix it by only offlining pd during
blkcg_css_offline(), and move the rest destruction (especially
blkg_put()) into blkcg_css_free(), which should be the right way as
discussed.
Fixes: ae1188963611 ("blkcg: consolidate blkg creation in blkcg_bio_issue_check()")
Reported-by: Jiufei Xue <jiufei.xue@linux.alibaba.com>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-15 23:51:27 -07:00
|
|
|
spin_unlock_irq(&blkcg->lock);
|
|
|
|
}
|
|
|
|
|
2022-04-19 21:27:14 -07:00
|
|
|
/**
|
|
|
|
* blkcg_pin_online - pin online state
|
|
|
|
* @blkcg_css: blkcg of interest
|
|
|
|
*
|
|
|
|
* While pinned, a blkcg is kept online. This is primarily used to
|
|
|
|
* impedance-match blkg and cgwb lifetimes so that blkg doesn't go offline
|
|
|
|
* while an associated cgwb is still active.
|
|
|
|
*/
|
|
|
|
void blkcg_pin_online(struct cgroup_subsys_state *blkcg_css)
|
|
|
|
{
|
|
|
|
refcount_inc(&css_to_blkcg(blkcg_css)->online_pin);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* blkcg_unpin_online - unpin online state
|
|
|
|
* @blkcg_css: blkcg of interest
|
|
|
|
*
|
|
|
|
* This is primarily used to impedance-match blkg and cgwb lifetimes so
|
|
|
|
* that blkg doesn't go offline while an associated cgwb is still active.
|
|
|
|
* When this count goes to zero, all active cgwbs have finished so the
|
|
|
|
* blkcg can continue destruction by calling blkcg_destroy_blkgs().
|
|
|
|
*/
|
|
|
|
void blkcg_unpin_online(struct cgroup_subsys_state *blkcg_css)
|
|
|
|
{
|
|
|
|
struct blkcg *blkcg = css_to_blkcg(blkcg_css);
|
|
|
|
|
|
|
|
do {
|
|
|
|
if (!refcount_dec_and_test(&blkcg->online_pin))
|
|
|
|
break;
|
|
|
|
blkcg_destroy_blkgs(blkcg);
|
|
|
|
blkcg = blkcg_parent(blkcg);
|
|
|
|
} while (blkcg);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* blkcg_css_offline - cgroup css_offline callback
|
|
|
|
* @css: css of interest
|
|
|
|
*
|
|
|
|
* This function is called when @css is about to go away. Here the cgwbs are
|
|
|
|
* offlined first and only once writeback associated with the blkcg has
|
|
|
|
* finished do we start step 2 (see above).
|
|
|
|
*/
|
|
|
|
static void blkcg_css_offline(struct cgroup_subsys_state *css)
|
|
|
|
{
|
|
|
|
/* this prevents anyone from attaching or migrating to this blkcg */
|
2022-04-19 21:27:15 -07:00
|
|
|
wb_blkcg_offline(css);
|
2022-04-19 21:27:14 -07:00
|
|
|
|
|
|
|
/* put the base online pin allowing step 2 to be triggered */
|
|
|
|
blkcg_unpin_online(css);
|
|
|
|
}
|
|
|
|
|
2013-08-08 17:11:23 -07:00
|
|
|
static void blkcg_css_free(struct cgroup_subsys_state *css)
|
2012-03-05 14:15:11 -07:00
|
|
|
{
|
2013-08-08 17:11:23 -07:00
|
|
|
struct blkcg *blkcg = css_to_blkcg(css);
|
2015-08-18 14:55:08 -07:00
|
|
|
int i;
|
2012-03-05 14:15:11 -07:00
|
|
|
|
2015-07-09 13:39:49 -07:00
|
|
|
mutex_lock(&blkcg_pol_mutex);
|
2015-08-18 14:55:16 -07:00
|
|
|
|
2015-07-09 13:39:49 -07:00
|
|
|
list_del(&blkcg->all_blkcgs_node);
|
|
|
|
|
2015-08-18 14:55:08 -07:00
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++)
|
2015-08-18 14:55:16 -07:00
|
|
|
if (blkcg->cpd[i])
|
|
|
|
blkcg_policy[i]->cpd_free_fn(blkcg->cpd[i]);
|
|
|
|
|
|
|
|
mutex_unlock(&blkcg_pol_mutex);
|
|
|
|
|
2022-11-04 17:59:01 -07:00
|
|
|
free_percpu(blkcg->lhead);
|
2015-08-18 14:55:08 -07:00
|
|
|
kfree(blkcg);
|
2009-12-03 10:59:42 -07:00
|
|
|
}
|
|
|
|
|
2013-08-08 17:11:23 -07:00
|
|
|
static struct cgroup_subsys_state *
|
|
|
|
blkcg_css_alloc(struct cgroup_subsys_state *parent_css)
|
2009-12-03 10:59:42 -07:00
|
|
|
{
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg *blkcg;
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
int i;
|
2009-12-03 10:59:42 -07:00
|
|
|
|
2015-07-09 13:39:49 -07:00
|
|
|
mutex_lock(&blkcg_pol_mutex);
|
|
|
|
|
2013-08-08 17:11:23 -07:00
|
|
|
if (!parent_css) {
|
2012-04-16 13:57:25 -07:00
|
|
|
blkcg = &blkcg_root;
|
2015-08-18 14:55:08 -07:00
|
|
|
} else {
|
|
|
|
blkcg = kzalloc(sizeof(*blkcg), GFP_KERNEL);
|
2022-11-04 17:59:00 -07:00
|
|
|
if (!blkcg)
|
2017-08-25 08:49:32 -07:00
|
|
|
goto unlock;
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
}
|
|
|
|
|
2022-11-04 17:59:01 -07:00
|
|
|
if (init_blkcg_llists(blkcg))
|
|
|
|
goto free_blkcg;
|
|
|
|
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
for (i = 0; i < BLKCG_MAX_POLS ; i++) {
|
|
|
|
struct blkcg_policy *pol = blkcg_policy[i];
|
|
|
|
struct blkcg_policy_data *cpd;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the policy hasn't been attached yet, wait for it
|
|
|
|
* to be attached before doing anything else. Otherwise,
|
|
|
|
* check if the policy requires any specific per-cgroup
|
|
|
|
* data: if it does, allocate and initialize it.
|
|
|
|
*/
|
2015-08-18 14:55:16 -07:00
|
|
|
if (!pol || !pol->cpd_alloc_fn)
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
continue;
|
|
|
|
|
2015-08-18 14:55:16 -07:00
|
|
|
cpd = pol->cpd_alloc_fn(GFP_KERNEL);
|
2022-11-04 17:59:00 -07:00
|
|
|
if (!cpd)
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
goto free_pd_blkcg;
|
2022-11-04 17:59:00 -07:00
|
|
|
|
2015-08-18 14:55:15 -07:00
|
|
|
blkcg->cpd[i] = cpd;
|
|
|
|
cpd->blkcg = blkcg;
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
cpd->plid = i;
|
|
|
|
}
|
2009-12-03 10:59:42 -07:00
|
|
|
|
|
|
|
spin_lock_init(&blkcg->lock);
|
2019-07-24 10:37:22 -07:00
|
|
|
refcount_set(&blkcg->online_pin, 1);
|
2016-11-21 16:03:32 -07:00
|
|
|
INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_NOWAIT | __GFP_NOWARN);
|
2009-12-03 10:59:42 -07:00
|
|
|
INIT_HLIST_HEAD(&blkcg->blkg_list);
|
2015-05-22 14:13:37 -07:00
|
|
|
#ifdef CONFIG_CGROUP_WRITEBACK
|
|
|
|
INIT_LIST_HEAD(&blkcg->cgwb_list);
|
|
|
|
#endif
|
2015-07-09 13:39:49 -07:00
|
|
|
list_add_tail(&blkcg->all_blkcgs_node, &all_blkcgs);
|
|
|
|
|
|
|
|
mutex_unlock(&blkcg_pol_mutex);
|
2009-12-03 10:59:42 -07:00
|
|
|
return &blkcg->css;
|
block, cgroup: implement policy-specific per-blkcg data
The block IO (blkio) controller enables the block layer to provide service
guarantees in a hierarchical fashion. Specifically, service guarantees
are provided by registered request-accounting policies. As of now, a
proportional-share and a throttling policy are available. They are
implemented, respectively, by the CFQ I/O scheduler and the blk-throttle
subsystem. Unfortunately, as for adding new policies, the current
implementation of the block IO controller is only halfway ready to allow
new policies to be plugged in. This commit provides a solution to make
the block IO controller fully ready to handle new policies.
In what follows, we first describe briefly the current state, and then
list the changes made by this commit.
The throttling policy does not need any per-cgroup information to perform
its task. In contrast, the proportional share policy uses, for each cgroup,
both the weight assigned by the user to the cgroup, and a set of dynamically-
computed weights, one for each device.
The first, user-defined weight is stored in the blkcg data structure: the
block IO controller allocates a private blkcg data structure for each
cgroup in the blkio cgroups hierarchy (regardless of which policy is active).
In other words, the block IO controller internally mirrors the blkio cgroups
with private blkcg data structures.
On the other hand, for each cgroup and device, the corresponding dynamically-
computed weight is maintained in the following, different way. For each device,
the block IO controller keeps a private blkcg_gq structure for each cgroup in
blkio. In other words, block IO also keeps one private mirror copy of the blkio
cgroups hierarchy for each device, made of blkcg_gq structures.
Each blkcg_gq structure keeps per-policy information in a generic array of
dynamically-allocated 'dedicated' data structures, one for each registered
policy (so currently the array contains two elements). To be inserted into the
generic array, each dedicated data structure embeds a generic blkg_policy_data
structure. Consider now the array contained in the blkcg_gq structure
corresponding to a given pair of cgroup and device: one of the elements
of the array contains the dedicated data structure for the proportional-share
policy, and this dedicated data structure contains the dynamically-computed
weight for that pair of cgroup and device.
The generic strategy adopted for storing per-policy data in blkcg_gq structures
is already capable of handling new policies, whereas the one adopted with blkcg
structures is not, because per-policy data are hard-coded in the blkcg
structures themselves (currently only data related to the proportional-
share policy).
This commit addresses the above issues through the following changes:
. It generalizes blkcg structures so that per-policy data are stored in the same
way as in blkcg_gq structures.
Specifically, it lets also the blkcg structure store per-policy data in a
generic array of dynamically-allocated dedicated data structures. We will
refer to these data structures as blkcg dedicated data structures, to
distinguish them from the dedicated data structures inserted in the generic
arrays kept by blkcg_gq structures.
To allow blkcg dedicated data structures to be inserted in the generic array
inside a blkcg structure, this commit also introduces a new blkcg_policy_data
structure, which is the equivalent of blkg_policy_data for blkcg dedicated
data structures.
. It adds to the blkcg_policy structure, i.e., to the descriptor of a policy, a
cpd_size field and a cpd_init field, to be initialized by the policy with,
respectively, the size of the blkcg dedicated data structures, and the
address of a constructor function for blkcg dedicated data structures.
. It moves the CFQ-specific fields embedded in the blkcg data structure (i.e.,
the fields related to the proportional-share policy), into a new blkcg
dedicated data structure called cfq_group_data.
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Jens Axboe <axboe@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-06-05 14:38:42 -07:00
|
|
|
|
|
|
|
free_pd_blkcg:
|
|
|
|
for (i--; i >= 0; i--)
|
2015-08-18 14:55:16 -07:00
|
|
|
if (blkcg->cpd[i])
|
|
|
|
blkcg_policy[i]->cpd_free_fn(blkcg->cpd[i]);
|
2022-11-04 17:59:01 -07:00
|
|
|
free_percpu(blkcg->lhead);
|
|
|
|
free_blkcg:
|
2017-08-25 08:49:32 -07:00
|
|
|
if (blkcg != &blkcg_root)
|
|
|
|
kfree(blkcg);
|
|
|
|
unlock:
|
2015-07-09 13:39:49 -07:00
|
|
|
mutex_unlock(&blkcg_pol_mutex);
|
2022-11-04 17:59:00 -07:00
|
|
|
return ERR_PTR(-ENOMEM);
|
2009-12-03 10:59:42 -07:00
|
|
|
}
|
|
|
|
|
2019-07-24 10:37:55 -07:00
|
|
|
static int blkcg_css_online(struct cgroup_subsys_state *css)
|
|
|
|
{
|
2022-04-19 21:27:14 -07:00
|
|
|
struct blkcg *parent = blkcg_parent(css_to_blkcg(css));
|
2019-07-24 10:37:55 -07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* blkcg_pin_online() is used to delay blkcg offline so that blkgs
|
|
|
|
* don't go offline while cgwbs are still active on them. Pin the
|
|
|
|
* parent so that offline always happens towards the root.
|
|
|
|
*/
|
|
|
|
if (parent)
|
2022-11-14 11:19:30 -07:00
|
|
|
blkcg_pin_online(&parent->css);
|
2019-07-24 10:37:55 -07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2024-04-07 05:59:10 -07:00
|
|
|
void blkg_init_queue(struct request_queue *q)
|
|
|
|
{
|
|
|
|
INIT_LIST_HEAD(&q->blkg_list);
|
|
|
|
mutex_init(&q->blkcg_mutex);
|
|
|
|
}
|
|
|
|
|
2022-09-21 11:04:50 -07:00
|
|
|
int blkcg_init_disk(struct gendisk *disk)
|
2012-03-05 14:15:12 -07:00
|
|
|
{
|
2022-09-21 11:04:50 -07:00
|
|
|
struct request_queue *q = disk->queue;
|
2017-03-29 10:25:48 -07:00
|
|
|
struct blkcg_gq *new_blkg, *blkg;
|
|
|
|
bool preloaded;
|
2015-05-22 14:13:19 -07:00
|
|
|
|
2022-09-21 11:05:01 -07:00
|
|
|
new_blkg = blkg_alloc(&blkcg_root, disk, GFP_KERNEL);
|
2017-03-29 10:25:48 -07:00
|
|
|
if (!new_blkg)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
preloaded = !radix_tree_preload(GFP_KERNEL);
|
|
|
|
|
2018-04-18 21:04:26 -07:00
|
|
|
/* Make sure the root blkg exists. */
|
2022-05-16 10:39:30 -07:00
|
|
|
/* spin_lock_irq can serve as RCU read-side critical section. */
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
2022-09-21 11:05:01 -07:00
|
|
|
blkg = blkg_create(&blkcg_root, disk, new_blkg);
|
2018-04-18 21:06:09 -07:00
|
|
|
if (IS_ERR(blkg))
|
|
|
|
goto err_unlock;
|
|
|
|
q->root_blkg = blkg;
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2015-05-22 14:13:19 -07:00
|
|
|
|
2017-03-29 10:25:48 -07:00
|
|
|
if (preloaded)
|
|
|
|
radix_tree_preload_end();
|
|
|
|
|
2018-11-14 09:02:12 -07:00
|
|
|
return 0;
|
2018-04-18 21:06:09 -07:00
|
|
|
|
|
|
|
err_unlock:
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2018-04-18 21:06:09 -07:00
|
|
|
if (preloaded)
|
|
|
|
radix_tree_preload_end();
|
|
|
|
return PTR_ERR(blkg);
|
2012-03-05 14:15:12 -07:00
|
|
|
}
|
|
|
|
|
2022-09-21 11:04:50 -07:00
|
|
|
void blkcg_exit_disk(struct gendisk *disk)
|
2012-03-05 14:15:12 -07:00
|
|
|
{
|
2022-09-21 11:04:59 -07:00
|
|
|
blkg_destroy_all(disk);
|
2022-09-21 11:04:56 -07:00
|
|
|
blk_throtl_exit(disk);
|
2012-03-05 14:15:12 -07:00
|
|
|
}
|
|
|
|
|
2018-07-03 08:14:55 -07:00
|
|
|
static void blkcg_exit(struct task_struct *tsk)
|
|
|
|
{
|
2023-02-03 08:03:48 -07:00
|
|
|
if (tsk->throttle_disk)
|
|
|
|
put_disk(tsk->throttle_disk);
|
|
|
|
tsk->throttle_disk = NULL;
|
2018-07-03 08:14:55 -07:00
|
|
|
}
|
|
|
|
|
blkcg: rename subsystem name from blkio to io
blkio interface has become messy over time and is currently the
largest. In addition to the inconsistent naming scheme, it has
multiple stat files which report more or less the same thing, a number
of debug stat files which expose internal details which shouldn't have
been part of the public interface in the first place, recursive and
non-recursive stats and leaf and non-leaf knobs.
Both recursive vs. non-recursive and leaf vs. non-leaf distinctions
don't make any sense on the unified hierarchy as only leaf cgroups can
contain processes. cgroups is going through a major interface
revision with the unified hierarchy involving significant fundamental
usage changes and given that a significant portion of the interface
doesn't make sense anymore, it's a good time to reorganize the
interface.
As the first step, this patch renames the external visible subsystem
name from "blkio" to "io". This is more concise, matches the other
two major subsystem names, "cpu" and "memory", and better suited as
blkcg will be involved in anything writeback related too whether an
actual block device is involved or not.
As the subsystem legacy_name is set to "blkio", the only userland
visible change outside the unified hierarchy is that blkcg is reported
as "io" instead of "blkio" in the subsystem initialized message during
boot. On the unified hierarchy, blkcg now appears as "io".
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: cgroups@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-08-18 14:55:29 -07:00
|
|
|
struct cgroup_subsys io_cgrp_subsys = {
|
2012-11-19 09:13:38 -07:00
|
|
|
.css_alloc = blkcg_css_alloc,
|
2019-07-24 10:37:55 -07:00
|
|
|
.css_online = blkcg_css_online,
|
2012-11-19 09:13:38 -07:00
|
|
|
.css_offline = blkcg_css_offline,
|
|
|
|
.css_free = blkcg_css_free,
|
2019-11-07 12:18:03 -07:00
|
|
|
.css_rstat_flush = blkcg_rstat_flush,
|
2015-08-18 14:55:34 -07:00
|
|
|
.dfl_cftypes = blkcg_files,
|
2015-08-18 14:55:30 -07:00
|
|
|
.legacy_cftypes = blkcg_legacy_files,
|
blkcg: rename subsystem name from blkio to io
blkio interface has become messy over time and is currently the
largest. In addition to the inconsistent naming scheme, it has
multiple stat files which report more or less the same thing, a number
of debug stat files which expose internal details which shouldn't have
been part of the public interface in the first place, recursive and
non-recursive stats and leaf and non-leaf knobs.
Both recursive vs. non-recursive and leaf vs. non-leaf distinctions
don't make any sense on the unified hierarchy as only leaf cgroups can
contain processes. cgroups is going through a major interface
revision with the unified hierarchy involving significant fundamental
usage changes and given that a significant portion of the interface
doesn't make sense anymore, it's a good time to reorganize the
interface.
As the first step, this patch renames the external visible subsystem
name from "blkio" to "io". This is more concise, matches the other
two major subsystem names, "cpu" and "memory", and better suited as
blkcg will be involved in anything writeback related too whether an
actual block device is involved or not.
As the subsystem legacy_name is set to "blkio", the only userland
visible change outside the unified hierarchy is that blkcg is reported
as "io" instead of "blkio" in the subsystem initialized message during
boot. On the unified hierarchy, blkcg now appears as "io".
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: cgroups@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-08-18 14:55:29 -07:00
|
|
|
.legacy_name = "blkio",
|
2018-07-03 08:14:55 -07:00
|
|
|
.exit = blkcg_exit,
|
2014-07-08 15:02:57 -07:00
|
|
|
#ifdef CONFIG_MEMCG
|
|
|
|
/*
|
|
|
|
* This ensures that, if available, memcg is automatically enabled
|
|
|
|
* together on the default hierarchy so that the owner cgroup can
|
|
|
|
* be retrieved from writeback pages.
|
|
|
|
*/
|
|
|
|
.depends_on = 1 << memory_cgrp_id,
|
|
|
|
#endif
|
2012-04-01 12:09:55 -07:00
|
|
|
};
|
blkcg: rename subsystem name from blkio to io
blkio interface has become messy over time and is currently the
largest. In addition to the inconsistent naming scheme, it has
multiple stat files which report more or less the same thing, a number
of debug stat files which expose internal details which shouldn't have
been part of the public interface in the first place, recursive and
non-recursive stats and leaf and non-leaf knobs.
Both recursive vs. non-recursive and leaf vs. non-leaf distinctions
don't make any sense on the unified hierarchy as only leaf cgroups can
contain processes. cgroups is going through a major interface
revision with the unified hierarchy involving significant fundamental
usage changes and given that a significant portion of the interface
doesn't make sense anymore, it's a good time to reorganize the
interface.
As the first step, this patch renames the external visible subsystem
name from "blkio" to "io". This is more concise, matches the other
two major subsystem names, "cpu" and "memory", and better suited as
blkcg will be involved in anything writeback related too whether an
actual block device is involved or not.
As the subsystem legacy_name is set to "blkio", the only userland
visible change outside the unified hierarchy is that blkcg is reported
as "io" instead of "blkio" in the subsystem initialized message during
boot. On the unified hierarchy, blkcg now appears as "io".
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: cgroups@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-08-18 14:55:29 -07:00
|
|
|
EXPORT_SYMBOL_GPL(io_cgrp_subsys);
|
2012-04-01 12:09:55 -07:00
|
|
|
|
2012-04-13 13:11:33 -07:00
|
|
|
/**
|
2023-02-03 08:03:57 -07:00
|
|
|
* blkcg_activate_policy - activate a blkcg policy on a gendisk
|
|
|
|
* @disk: gendisk of interest
|
2012-04-13 13:11:33 -07:00
|
|
|
* @pol: blkcg policy to activate
|
|
|
|
*
|
2023-02-03 08:03:57 -07:00
|
|
|
* Activate @pol on @disk. Requires %GFP_KERNEL context. @disk goes through
|
2012-04-13 13:11:33 -07:00
|
|
|
* bypass mode to populate its blkgs with policy_data for @pol.
|
|
|
|
*
|
2023-02-03 08:03:57 -07:00
|
|
|
* Activation happens with @disk bypassed, so nobody would be accessing blkgs
|
2012-04-13 13:11:33 -07:00
|
|
|
* from IO path. Update of each blkg is protected by both queue and blkcg
|
|
|
|
* locks so that holding either lock and testing blkcg_policy_enabled() is
|
|
|
|
* always enough for dereferencing policy data.
|
|
|
|
*
|
|
|
|
* The caller is responsible for synchronizing [de]activations and policy
|
|
|
|
* [un]registerations. Returns 0 on success, -errno on failure.
|
|
|
|
*/
|
2023-02-03 08:03:57 -07:00
|
|
|
int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
|
2012-04-13 13:11:33 -07:00
|
|
|
{
|
2023-02-03 08:03:57 -07:00
|
|
|
struct request_queue *q = disk->queue;
|
2015-08-18 14:55:09 -07:00
|
|
|
struct blkg_policy_data *pd_prealloc = NULL;
|
2019-10-15 09:03:47 -07:00
|
|
|
struct blkcg_gq *blkg, *pinned_blkg = NULL;
|
2015-08-18 14:55:09 -07:00
|
|
|
int ret;
|
2012-04-13 13:11:33 -07:00
|
|
|
|
|
|
|
if (blkcg_policy_enabled(q, pol))
|
|
|
|
return 0;
|
|
|
|
|
2024-07-19 00:15:04 -07:00
|
|
|
/*
|
|
|
|
* Policy is allowed to be registered without pd_alloc_fn/pd_free_fn,
|
|
|
|
* for example, ioprio. Such policy will work on blkcg level, not disk
|
|
|
|
* level, and don't need to be activated.
|
|
|
|
*/
|
|
|
|
if (WARN_ON_ONCE(!pol->pd_alloc_fn || !pol->pd_free_fn))
|
|
|
|
return -EINVAL;
|
|
|
|
|
2018-11-15 12:22:51 -07:00
|
|
|
if (queue_is_mq(q))
|
2017-01-17 06:03:22 -07:00
|
|
|
blk_mq_freeze_queue(q);
|
2019-10-15 09:03:47 -07:00
|
|
|
retry:
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
2012-04-13 13:11:33 -07:00
|
|
|
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
/* blkg_list is pushed at the head, reverse walk to initialize parents first */
|
2019-06-13 15:30:40 -07:00
|
|
|
list_for_each_entry_reverse(blkg, &q->blkg_list, q_node) {
|
2015-08-18 14:55:09 -07:00
|
|
|
struct blkg_policy_data *pd;
|
|
|
|
|
|
|
|
if (blkg->pd[pol->plid])
|
|
|
|
continue;
|
2012-04-13 13:11:33 -07:00
|
|
|
|
2019-10-15 09:03:47 -07:00
|
|
|
/* If prealloc matches, use it; otherwise try GFP_NOWAIT */
|
|
|
|
if (blkg == pinned_blkg) {
|
|
|
|
pd = pd_prealloc;
|
|
|
|
pd_prealloc = NULL;
|
|
|
|
} else {
|
2023-02-03 08:03:58 -07:00
|
|
|
pd = pol->pd_alloc_fn(disk, blkg->blkcg,
|
|
|
|
GFP_NOWAIT | __GFP_NOWARN);
|
2019-10-15 09:03:47 -07:00
|
|
|
}
|
|
|
|
|
2015-08-18 14:55:09 -07:00
|
|
|
if (!pd) {
|
2019-10-15 09:03:47 -07:00
|
|
|
/*
|
|
|
|
* GFP_NOWAIT failed. Free the existing one and
|
|
|
|
* prealloc for @blkg w/ GFP_KERNEL.
|
|
|
|
*/
|
|
|
|
if (pinned_blkg)
|
|
|
|
blkg_put(pinned_blkg);
|
|
|
|
blkg_get(blkg);
|
|
|
|
pinned_blkg = blkg;
|
|
|
|
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2019-10-15 09:03:47 -07:00
|
|
|
|
|
|
|
if (pd_prealloc)
|
|
|
|
pol->pd_free_fn(pd_prealloc);
|
2023-02-03 08:03:58 -07:00
|
|
|
pd_prealloc = pol->pd_alloc_fn(disk, blkg->blkcg,
|
|
|
|
GFP_KERNEL);
|
2019-10-15 09:03:47 -07:00
|
|
|
if (pd_prealloc)
|
|
|
|
goto retry;
|
|
|
|
else
|
|
|
|
goto enomem;
|
2015-08-18 14:55:09 -07:00
|
|
|
}
|
2012-04-13 13:11:33 -07:00
|
|
|
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
spin_lock(&blkg->blkcg->lock);
|
|
|
|
|
2012-04-13 13:11:33 -07:00
|
|
|
pd->blkg = blkg;
|
2013-01-09 09:05:12 -07:00
|
|
|
pd->plid = pol->plid;
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
blkg->pd[pol->plid] = pd;
|
2012-04-13 13:11:33 -07:00
|
|
|
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
if (pol->pd_init_fn)
|
|
|
|
pol->pd_init_fn(pd);
|
2019-10-15 09:03:47 -07:00
|
|
|
|
2023-01-19 04:03:49 -07:00
|
|
|
if (pol->pd_online_fn)
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
pol->pd_online_fn(pd);
|
|
|
|
pd->online = true;
|
|
|
|
|
|
|
|
spin_unlock(&blkg->blkcg->lock);
|
2023-01-19 04:03:49 -07:00
|
|
|
}
|
2023-01-03 04:28:33 -07:00
|
|
|
|
2012-04-13 13:11:33 -07:00
|
|
|
__set_bit(pol->plid, q->blkcg_pols);
|
|
|
|
ret = 0;
|
2015-08-18 14:55:09 -07:00
|
|
|
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2019-10-15 09:03:47 -07:00
|
|
|
out:
|
2018-11-15 12:22:51 -07:00
|
|
|
if (queue_is_mq(q))
|
2017-01-17 06:03:22 -07:00
|
|
|
blk_mq_unfreeze_queue(q);
|
2019-10-15 09:03:47 -07:00
|
|
|
if (pinned_blkg)
|
|
|
|
blkg_put(pinned_blkg);
|
2015-08-18 14:55:11 -07:00
|
|
|
if (pd_prealloc)
|
|
|
|
pol->pd_free_fn(pd_prealloc);
|
2012-04-13 13:11:33 -07:00
|
|
|
return ret;
|
2019-10-15 09:03:47 -07:00
|
|
|
|
|
|
|
enomem:
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
/* alloc failed, take down everything */
|
2019-10-15 09:03:47 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
|
|
|
list_for_each_entry(blkg, &q->blkg_list, q_node) {
|
2021-09-13 21:26:05 -07:00
|
|
|
struct blkcg *blkcg = blkg->blkcg;
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
struct blkg_policy_data *pd;
|
2021-09-13 21:26:05 -07:00
|
|
|
|
|
|
|
spin_lock(&blkcg->lock);
|
blk-cgroup: Fix NULL deref caused by blkg_policy_data being installed before init
blk-iocost sometimes causes the following crash:
BUG: kernel NULL pointer dereference, address: 00000000000000e0
...
RIP: 0010:_raw_spin_lock+0x17/0x30
Code: be 01 02 00 00 e8 79 38 39 ff 31 d2 89 d0 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 65 ff 05 48 d0 34 7e b9 01 00 00 00 31 c0 <f0> 0f b1 0f 75 02 5d c3 89 c6 e8 ea 04 00 00 5d c3 0f 1f 84 00 00
RSP: 0018:ffffc900023b3d40 EFLAGS: 00010046
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 0000000000000001
RDX: ffffc900023b3d20 RSI: ffffc900023b3cf0 RDI: 00000000000000e0
RBP: ffffc900023b3d40 R08: ffffc900023b3c10 R09: 0000000000000003
R10: 0000000000000064 R11: 000000000000000a R12: ffff888102337000
R13: fffffffffffffff2 R14: ffff88810af408c8 R15: ffff8881070c3600
FS: 00007faaaf364fc0(0000) GS:ffff88842fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000e0 CR3: 00000001097b1000 CR4: 0000000000350ea0
Call Trace:
<TASK>
ioc_weight_write+0x13d/0x410
cgroup_file_write+0x7a/0x130
kernfs_fop_write_iter+0xf5/0x170
vfs_write+0x298/0x370
ksys_write+0x5f/0xb0
__x64_sys_write+0x1b/0x20
do_syscall_64+0x3d/0x80
entry_SYSCALL_64_after_hwframe+0x46/0xb0
This happens because iocg->ioc is NULL. The field is initialized by
ioc_pd_init() and never cleared. The NULL deref is caused by
blkcg_activate_policy() installing blkg_policy_data before initializing it.
blkcg_activate_policy() was doing the following:
1. Allocate pd's for all existing blkg's and install them in blkg->pd[].
2. Initialize all pd's.
3. Online all pd's.
blkcg_activate_policy() only grabs the queue_lock and may release and
re-acquire the lock as allocation may need to sleep. ioc_weight_write()
grabs blkcg->lock and iterates all its blkg's. The two can race and if
ioc_weight_write() runs during #1 or between #1 and #2, it can encounter a
pd which is not initialized yet, leading to crash.
The crash can be reproduced with the following script:
#!/bin/bash
echo +io > /sys/fs/cgroup/cgroup.subtree_control
systemd-run --unit touch-sda --scope dd if=/dev/sda of=/dev/null bs=1M count=1 iflag=direct
echo 100 > /sys/fs/cgroup/system.slice/io.weight
bash -c "echo '8:0 enable=1' > /sys/fs/cgroup/io.cost.qos" &
sleep .2
echo 100 > /sys/fs/cgroup/system.slice/io.weight
with the following patch applied:
> diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
> index fc49be622e05..38d671d5e10c 100644
> --- a/block/blk-cgroup.c
> +++ b/block/blk-cgroup.c
> @@ -1553,6 +1553,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
> pd->online = false;
> }
>
> + if (system_state == SYSTEM_RUNNING) {
> + spin_unlock_irq(&q->queue_lock);
> + ssleep(1);
> + spin_lock_irq(&q->queue_lock);
> + }
> +
> /* all allocated, init in the same order */
> if (pol->pd_init_fn)
> list_for_each_entry_reverse(blkg, &q->blkg_list, q_node)
I don't see a reason why all pd's should be allocated, initialized and
onlined together. The only ordering requirement is that parent blkgs to be
initialized and onlined before children, which is guaranteed from the
walking order. Let's fix the bug by allocating, initializing and onlining pd
for each blkg and holding blkcg->lock over initialization and onlining. This
ensures that an installed blkg is always fully initialized and onlined
removing the the race window.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Breno Leitao <leitao@debian.org>
Fixes: 9d179b865449 ("blkcg: Fix multiple bugs in blkcg_activate_policy()")
Link: https://lore.kernel.org/r/ZN0p5_W-Q9mAHBVY@slm.duckdns.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-08-16 12:56:23 -07:00
|
|
|
pd = blkg->pd[pol->plid];
|
|
|
|
if (pd) {
|
|
|
|
if (pd->online && pol->pd_offline_fn)
|
|
|
|
pol->pd_offline_fn(pd);
|
|
|
|
pd->online = false;
|
|
|
|
pol->pd_free_fn(pd);
|
2019-10-15 09:03:47 -07:00
|
|
|
blkg->pd[pol->plid] = NULL;
|
|
|
|
}
|
2021-09-13 21:26:05 -07:00
|
|
|
spin_unlock(&blkcg->lock);
|
2019-10-15 09:03:47 -07:00
|
|
|
}
|
|
|
|
spin_unlock_irq(&q->queue_lock);
|
|
|
|
ret = -ENOMEM;
|
|
|
|
goto out;
|
2012-04-13 13:11:33 -07:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(blkcg_activate_policy);
|
|
|
|
|
|
|
|
/**
|
2023-02-03 08:03:57 -07:00
|
|
|
* blkcg_deactivate_policy - deactivate a blkcg policy on a gendisk
|
|
|
|
* @disk: gendisk of interest
|
2012-04-13 13:11:33 -07:00
|
|
|
* @pol: blkcg policy to deactivate
|
|
|
|
*
|
2023-02-03 08:03:57 -07:00
|
|
|
* Deactivate @pol on @disk. Follows the same synchronization rules as
|
2012-04-13 13:11:33 -07:00
|
|
|
* blkcg_activate_policy().
|
|
|
|
*/
|
2023-02-03 08:03:57 -07:00
|
|
|
void blkcg_deactivate_policy(struct gendisk *disk,
|
2012-04-16 13:57:25 -07:00
|
|
|
const struct blkcg_policy *pol)
|
2012-04-13 13:11:33 -07:00
|
|
|
{
|
2023-02-03 08:03:57 -07:00
|
|
|
struct request_queue *q = disk->queue;
|
2012-04-16 13:57:25 -07:00
|
|
|
struct blkcg_gq *blkg;
|
2012-04-13 13:11:33 -07:00
|
|
|
|
|
|
|
if (!blkcg_policy_enabled(q, pol))
|
|
|
|
return;
|
|
|
|
|
2018-11-15 12:22:51 -07:00
|
|
|
if (queue_is_mq(q))
|
2017-01-17 06:03:22 -07:00
|
|
|
blk_mq_freeze_queue(q);
|
|
|
|
|
2023-02-14 11:33:04 -07:00
|
|
|
mutex_lock(&q->blkcg_mutex);
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_lock_irq(&q->queue_lock);
|
2012-04-13 13:11:33 -07:00
|
|
|
|
|
|
|
__clear_bit(pol->plid, q->blkcg_pols);
|
|
|
|
|
|
|
|
list_for_each_entry(blkg, &q->blkg_list, q_node) {
|
2021-09-13 21:26:05 -07:00
|
|
|
struct blkcg *blkcg = blkg->blkcg;
|
|
|
|
|
|
|
|
spin_lock(&blkcg->lock);
|
2015-08-18 14:55:11 -07:00
|
|
|
if (blkg->pd[pol->plid]) {
|
2023-01-19 04:03:49 -07:00
|
|
|
if (blkg->pd[pol->plid]->online && pol->pd_offline_fn)
|
2015-08-18 14:55:14 -07:00
|
|
|
pol->pd_offline_fn(blkg->pd[pol->plid]);
|
2015-08-18 14:55:11 -07:00
|
|
|
pol->pd_free_fn(blkg->pd[pol->plid]);
|
|
|
|
blkg->pd[pol->plid] = NULL;
|
|
|
|
}
|
2021-09-13 21:26:05 -07:00
|
|
|
spin_unlock(&blkcg->lock);
|
2012-04-13 13:11:33 -07:00
|
|
|
}
|
|
|
|
|
2018-11-15 12:17:28 -07:00
|
|
|
spin_unlock_irq(&q->queue_lock);
|
2023-02-14 11:33:04 -07:00
|
|
|
mutex_unlock(&q->blkcg_mutex);
|
2017-01-17 06:03:22 -07:00
|
|
|
|
2018-11-15 12:22:51 -07:00
|
|
|
if (queue_is_mq(q))
|
2017-01-17 06:03:22 -07:00
|
|
|
blk_mq_unfreeze_queue(q);
|
2012-04-13 13:11:33 -07:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(blkcg_deactivate_policy);
|
|
|
|
|
2022-06-29 00:09:17 -07:00
|
|
|
static void blkcg_free_all_cpd(struct blkcg_policy *pol)
|
|
|
|
{
|
|
|
|
struct blkcg *blkcg;
|
|
|
|
|
|
|
|
list_for_each_entry(blkcg, &all_blkcgs, all_blkcgs_node) {
|
|
|
|
if (blkcg->cpd[pol->plid]) {
|
|
|
|
pol->cpd_free_fn(blkcg->cpd[pol->plid]);
|
|
|
|
blkcg->cpd[pol->plid] = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-04-13 13:11:28 -07:00
|
|
|
/**
|
2012-04-16 13:57:25 -07:00
|
|
|
* blkcg_policy_register - register a blkcg policy
|
|
|
|
* @pol: blkcg policy to register
|
2012-04-13 13:11:28 -07:00
|
|
|
*
|
2012-04-16 13:57:25 -07:00
|
|
|
* Register @pol with blkcg core. Might sleep and @pol may be modified on
|
|
|
|
* successful registration. Returns 0 on success and -errno on failure.
|
2012-04-13 13:11:28 -07:00
|
|
|
*/
|
2014-06-22 15:31:56 -07:00
|
|
|
int blkcg_policy_register(struct blkcg_policy *pol)
|
2009-12-04 08:36:42 -07:00
|
|
|
{
|
2015-07-09 13:39:50 -07:00
|
|
|
struct blkcg *blkcg;
|
2012-04-13 13:11:28 -07:00
|
|
|
int i, ret;
|
2012-03-05 14:15:20 -07:00
|
|
|
|
2015-07-09 13:39:47 -07:00
|
|
|
mutex_lock(&blkcg_pol_register_mutex);
|
2012-04-13 13:11:26 -07:00
|
|
|
mutex_lock(&blkcg_pol_mutex);
|
|
|
|
|
2012-04-13 13:11:28 -07:00
|
|
|
/* find an empty slot */
|
|
|
|
ret = -ENOSPC;
|
|
|
|
for (i = 0; i < BLKCG_MAX_POLS; i++)
|
2012-04-16 13:57:25 -07:00
|
|
|
if (!blkcg_policy[i])
|
2012-04-13 13:11:28 -07:00
|
|
|
break;
|
2018-09-11 09:59:53 -07:00
|
|
|
if (i >= BLKCG_MAX_POLS) {
|
|
|
|
pr_warn("blkcg_policy_register: BLKCG_MAX_POLS too small\n");
|
2015-07-09 13:39:47 -07:00
|
|
|
goto err_unlock;
|
2018-09-11 09:59:53 -07:00
|
|
|
}
|
2012-03-05 14:15:04 -07:00
|
|
|
|
2024-07-19 00:15:04 -07:00
|
|
|
/*
|
|
|
|
* Make sure cpd/pd_alloc_fn and cpd/pd_free_fn in pairs, and policy
|
|
|
|
* without pd_alloc_fn/pd_free_fn can't be activated.
|
|
|
|
*/
|
2017-10-17 08:56:21 -07:00
|
|
|
if ((!pol->cpd_alloc_fn ^ !pol->cpd_free_fn) ||
|
2024-07-19 00:15:04 -07:00
|
|
|
(!pol->pd_alloc_fn ^ !pol->pd_free_fn))
|
2017-10-17 08:56:21 -07:00
|
|
|
goto err_unlock;
|
|
|
|
|
2015-07-09 13:39:50 -07:00
|
|
|
/* register @pol */
|
2012-04-16 13:57:25 -07:00
|
|
|
pol->plid = i;
|
2015-07-09 13:39:50 -07:00
|
|
|
blkcg_policy[pol->plid] = pol;
|
|
|
|
|
|
|
|
/* allocate and install cpd's */
|
2015-08-18 14:55:16 -07:00
|
|
|
if (pol->cpd_alloc_fn) {
|
2015-07-09 13:39:50 -07:00
|
|
|
list_for_each_entry(blkcg, &all_blkcgs, all_blkcgs_node) {
|
|
|
|
struct blkcg_policy_data *cpd;
|
|
|
|
|
2015-08-18 14:55:16 -07:00
|
|
|
cpd = pol->cpd_alloc_fn(GFP_KERNEL);
|
2016-09-29 08:33:30 -07:00
|
|
|
if (!cpd)
|
2015-07-09 13:39:50 -07:00
|
|
|
goto err_free_cpds;
|
|
|
|
|
2015-08-18 14:55:15 -07:00
|
|
|
blkcg->cpd[pol->plid] = cpd;
|
|
|
|
cpd->blkcg = blkcg;
|
2015-07-09 13:39:50 -07:00
|
|
|
cpd->plid = pol->plid;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-07-09 13:39:47 -07:00
|
|
|
mutex_unlock(&blkcg_pol_mutex);
|
2012-04-13 13:11:28 -07:00
|
|
|
|
|
|
|
/* everything is in place, add intf files for the new policy */
|
2015-08-18 14:55:34 -07:00
|
|
|
if (pol->dfl_cftypes)
|
|
|
|
WARN_ON(cgroup_add_dfl_cftypes(&io_cgrp_subsys,
|
|
|
|
pol->dfl_cftypes));
|
2015-08-18 14:55:30 -07:00
|
|
|
if (pol->legacy_cftypes)
|
blkcg: rename subsystem name from blkio to io
blkio interface has become messy over time and is currently the
largest. In addition to the inconsistent naming scheme, it has
multiple stat files which report more or less the same thing, a number
of debug stat files which expose internal details which shouldn't have
been part of the public interface in the first place, recursive and
non-recursive stats and leaf and non-leaf knobs.
Both recursive vs. non-recursive and leaf vs. non-leaf distinctions
don't make any sense on the unified hierarchy as only leaf cgroups can
contain processes. cgroups is going through a major interface
revision with the unified hierarchy involving significant fundamental
usage changes and given that a significant portion of the interface
doesn't make sense anymore, it's a good time to reorganize the
interface.
As the first step, this patch renames the external visible subsystem
name from "blkio" to "io". This is more concise, matches the other
two major subsystem names, "cpu" and "memory", and better suited as
blkcg will be involved in anything writeback related too whether an
actual block device is involved or not.
As the subsystem legacy_name is set to "blkio", the only userland
visible change outside the unified hierarchy is that blkcg is reported
as "io" instead of "blkio" in the subsystem initialized message during
boot. On the unified hierarchy, blkcg now appears as "io".
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: cgroups@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-08-18 14:55:29 -07:00
|
|
|
WARN_ON(cgroup_add_legacy_cftypes(&io_cgrp_subsys,
|
2015-08-18 14:55:30 -07:00
|
|
|
pol->legacy_cftypes));
|
2015-07-09 13:39:47 -07:00
|
|
|
mutex_unlock(&blkcg_pol_register_mutex);
|
|
|
|
return 0;
|
|
|
|
|
2015-07-09 13:39:50 -07:00
|
|
|
err_free_cpds:
|
2022-06-29 00:09:17 -07:00
|
|
|
if (pol->cpd_free_fn)
|
|
|
|
blkcg_free_all_cpd(pol);
|
|
|
|
|
2015-07-09 13:39:50 -07:00
|
|
|
blkcg_policy[pol->plid] = NULL;
|
2015-07-09 13:39:47 -07:00
|
|
|
err_unlock:
|
2012-04-13 13:11:26 -07:00
|
|
|
mutex_unlock(&blkcg_pol_mutex);
|
2015-07-09 13:39:47 -07:00
|
|
|
mutex_unlock(&blkcg_pol_register_mutex);
|
2012-04-13 13:11:28 -07:00
|
|
|
return ret;
|
2009-12-04 08:36:42 -07:00
|
|
|
}
|
2012-04-16 13:57:25 -07:00
|
|
|
EXPORT_SYMBOL_GPL(blkcg_policy_register);
|
2009-12-04 08:36:42 -07:00
|
|
|
|
2012-04-13 13:11:28 -07:00
|
|
|
/**
|
2012-04-16 13:57:25 -07:00
|
|
|
* blkcg_policy_unregister - unregister a blkcg policy
|
|
|
|
* @pol: blkcg policy to unregister
|
2012-04-13 13:11:28 -07:00
|
|
|
*
|
2012-04-16 13:57:25 -07:00
|
|
|
* Undo blkcg_policy_register(@pol). Might sleep.
|
2012-04-13 13:11:28 -07:00
|
|
|
*/
|
2012-04-16 13:57:25 -07:00
|
|
|
void blkcg_policy_unregister(struct blkcg_policy *pol)
|
2009-12-04 08:36:42 -07:00
|
|
|
{
|
2015-07-09 13:39:47 -07:00
|
|
|
mutex_lock(&blkcg_pol_register_mutex);
|
2012-04-13 13:11:26 -07:00
|
|
|
|
2012-04-16 13:57:25 -07:00
|
|
|
if (WARN_ON(blkcg_policy[pol->plid] != pol))
|
2012-04-13 13:11:28 -07:00
|
|
|
goto out_unlock;
|
|
|
|
|
|
|
|
/* kill the intf files first */
|
2015-08-18 14:55:34 -07:00
|
|
|
if (pol->dfl_cftypes)
|
|
|
|
cgroup_rm_cftypes(pol->dfl_cftypes);
|
2015-08-18 14:55:30 -07:00
|
|
|
if (pol->legacy_cftypes)
|
|
|
|
cgroup_rm_cftypes(pol->legacy_cftypes);
|
2012-04-01 14:38:43 -07:00
|
|
|
|
2015-07-09 13:39:50 -07:00
|
|
|
/* remove cpds and unregister */
|
2015-07-09 13:39:47 -07:00
|
|
|
mutex_lock(&blkcg_pol_mutex);
|
2015-07-09 13:39:50 -07:00
|
|
|
|
2022-06-29 00:09:17 -07:00
|
|
|
if (pol->cpd_free_fn)
|
|
|
|
blkcg_free_all_cpd(pol);
|
|
|
|
|
2012-04-16 13:57:25 -07:00
|
|
|
blkcg_policy[pol->plid] = NULL;
|
2015-07-09 13:39:50 -07:00
|
|
|
|
2012-04-13 13:11:26 -07:00
|
|
|
mutex_unlock(&blkcg_pol_mutex);
|
2015-07-09 13:39:47 -07:00
|
|
|
out_unlock:
|
|
|
|
mutex_unlock(&blkcg_pol_register_mutex);
|
2009-12-04 08:36:42 -07:00
|
|
|
}
|
2012-04-16 13:57:25 -07:00
|
|
|
EXPORT_SYMBOL_GPL(blkcg_policy_unregister);
|
2018-07-03 08:14:52 -07:00
|
|
|
|
2018-07-03 08:14:55 -07:00
|
|
|
/*
|
|
|
|
* Scale the accumulated delay based on how long it has been since we updated
|
|
|
|
* the delay. We only call this when we are adding delay, in case it's been a
|
|
|
|
* while since we added delay, and when we are checking to see if we need to
|
|
|
|
* delay a task, to account for any delays that may have occurred.
|
|
|
|
*/
|
|
|
|
static void blkcg_scale_delay(struct blkcg_gq *blkg, u64 now)
|
|
|
|
{
|
|
|
|
u64 old = atomic64_read(&blkg->delay_start);
|
|
|
|
|
2020-04-13 09:27:55 -07:00
|
|
|
/* negative use_delay means no scaling, see blkcg_set_delay() */
|
|
|
|
if (atomic_read(&blkg->use_delay) < 0)
|
|
|
|
return;
|
|
|
|
|
2018-07-03 08:14:55 -07:00
|
|
|
/*
|
|
|
|
* We only want to scale down every second. The idea here is that we
|
|
|
|
* want to delay people for min(delay_nsec, NSEC_PER_SEC) in a certain
|
|
|
|
* time window. We only want to throttle tasks for recent delay that
|
|
|
|
* has occurred, in 1 second time windows since that's the maximum
|
|
|
|
* things can be throttled. We save the current delay window in
|
|
|
|
* blkg->last_delay so we know what amount is still left to be charged
|
|
|
|
* to the blkg from this point onward. blkg->last_use keeps track of
|
|
|
|
* the use_delay counter. The idea is if we're unthrottling the blkg we
|
|
|
|
* are ok with whatever is happening now, and we can take away more of
|
|
|
|
* the accumulated delay as we've already throttled enough that
|
|
|
|
* everybody is happy with their IO latencies.
|
|
|
|
*/
|
|
|
|
if (time_before64(old + NSEC_PER_SEC, now) &&
|
2022-07-12 08:44:55 -07:00
|
|
|
atomic64_try_cmpxchg(&blkg->delay_start, &old, now)) {
|
2018-07-03 08:14:55 -07:00
|
|
|
u64 cur = atomic64_read(&blkg->delay_nsec);
|
|
|
|
u64 sub = min_t(u64, blkg->last_delay, now - old);
|
|
|
|
int cur_use = atomic_read(&blkg->use_delay);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We've been unthrottled, subtract a larger chunk of our
|
|
|
|
* accumulated delay.
|
|
|
|
*/
|
|
|
|
if (cur_use < blkg->last_use)
|
|
|
|
sub = max_t(u64, sub, blkg->last_delay >> 1);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This shouldn't happen, but handle it anyway. Our delay_nsec
|
|
|
|
* should only ever be growing except here where we subtract out
|
|
|
|
* min(last_delay, 1 second), but lord knows bugs happen and I'd
|
|
|
|
* rather not end up with negative numbers.
|
|
|
|
*/
|
|
|
|
if (unlikely(cur < sub)) {
|
|
|
|
atomic64_set(&blkg->delay_nsec, 0);
|
|
|
|
blkg->last_delay = 0;
|
|
|
|
} else {
|
|
|
|
atomic64_sub(sub, &blkg->delay_nsec);
|
|
|
|
blkg->last_delay = cur - sub;
|
|
|
|
}
|
|
|
|
blkg->last_use = cur_use;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is called when we want to actually walk up the hierarchy and check to
|
|
|
|
* see if we need to throttle, and then actually throttle if there is some
|
|
|
|
* accumulated delay. This should only be called upon return to user space so
|
|
|
|
* we're not holding some lock that would induce a priority inversion.
|
|
|
|
*/
|
|
|
|
static void blkcg_maybe_throttle_blkg(struct blkcg_gq *blkg, bool use_memdelay)
|
|
|
|
{
|
2019-07-09 14:41:29 -07:00
|
|
|
unsigned long pflags;
|
2020-09-01 11:52:52 -07:00
|
|
|
bool clamp;
|
2024-01-15 14:45:07 -07:00
|
|
|
u64 now = blk_time_get_ns();
|
2018-07-03 08:14:55 -07:00
|
|
|
u64 exp;
|
|
|
|
u64 delay_nsec = 0;
|
|
|
|
int tok;
|
|
|
|
|
|
|
|
while (blkg->parent) {
|
2020-09-01 11:52:52 -07:00
|
|
|
int use_delay = atomic_read(&blkg->use_delay);
|
|
|
|
|
|
|
|
if (use_delay) {
|
|
|
|
u64 this_delay;
|
|
|
|
|
2018-07-03 08:14:55 -07:00
|
|
|
blkcg_scale_delay(blkg, now);
|
2020-09-01 11:52:52 -07:00
|
|
|
this_delay = atomic64_read(&blkg->delay_nsec);
|
|
|
|
if (this_delay > delay_nsec) {
|
|
|
|
delay_nsec = this_delay;
|
|
|
|
clamp = use_delay > 0;
|
|
|
|
}
|
2018-07-03 08:14:55 -07:00
|
|
|
}
|
|
|
|
blkg = blkg->parent;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!delay_nsec)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Let's not sleep for all eternity if we've amassed a huge delay.
|
|
|
|
* Swapping or metadata IO can accumulate 10's of seconds worth of
|
|
|
|
* delay, and we want userspace to be able to do _something_ so cap the
|
2020-09-01 11:52:52 -07:00
|
|
|
* delays at 0.25s. If there's 10's of seconds worth of delay then the
|
|
|
|
* tasks will be delayed for 0.25 second for every syscall. If
|
|
|
|
* blkcg_set_delay() was used as indicated by negative use_delay, the
|
|
|
|
* caller is responsible for regulating the range.
|
2018-07-03 08:14:55 -07:00
|
|
|
*/
|
2020-09-01 11:52:52 -07:00
|
|
|
if (clamp)
|
|
|
|
delay_nsec = min_t(u64, delay_nsec, 250 * NSEC_PER_MSEC);
|
2018-07-03 08:14:55 -07:00
|
|
|
|
2019-07-09 14:41:29 -07:00
|
|
|
if (use_memdelay)
|
|
|
|
psi_memstall_enter(&pflags);
|
2018-07-03 08:14:55 -07:00
|
|
|
|
|
|
|
exp = ktime_add_ns(now, delay_nsec);
|
|
|
|
tok = io_schedule_prepare();
|
|
|
|
do {
|
|
|
|
__set_current_state(TASK_KILLABLE);
|
|
|
|
if (!schedule_hrtimeout(&exp, HRTIMER_MODE_ABS))
|
|
|
|
break;
|
|
|
|
} while (!fatal_signal_pending(current));
|
|
|
|
io_schedule_finish(tok);
|
2019-07-09 14:41:29 -07:00
|
|
|
|
|
|
|
if (use_memdelay)
|
|
|
|
psi_memstall_leave(&pflags);
|
2018-07-03 08:14:55 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* blkcg_maybe_throttle_current - throttle the current task if it has been marked
|
|
|
|
*
|
|
|
|
* This is only called if we've been marked with set_notify_resume(). Obviously
|
|
|
|
* we can be set_notify_resume() for reasons other than blkcg throttling, so we
|
2023-02-03 08:03:48 -07:00
|
|
|
* check to see if current->throttle_disk is set and if not this doesn't do
|
2018-07-03 08:14:55 -07:00
|
|
|
* anything. This should only ever be called by the resume code, it's not meant
|
|
|
|
* to be called by people willy-nilly as it will actually do the work to
|
|
|
|
* throttle the task if it is setup for throttling.
|
|
|
|
*/
|
|
|
|
void blkcg_maybe_throttle_current(void)
|
|
|
|
{
|
2023-02-03 08:03:48 -07:00
|
|
|
struct gendisk *disk = current->throttle_disk;
|
2018-07-03 08:14:55 -07:00
|
|
|
struct blkcg *blkcg;
|
|
|
|
struct blkcg_gq *blkg;
|
|
|
|
bool use_memdelay = current->use_memdelay;
|
|
|
|
|
2023-02-03 08:03:48 -07:00
|
|
|
if (!disk)
|
2018-07-03 08:14:55 -07:00
|
|
|
return;
|
|
|
|
|
2023-02-03 08:03:48 -07:00
|
|
|
current->throttle_disk = NULL;
|
2018-07-03 08:14:55 -07:00
|
|
|
current->use_memdelay = false;
|
|
|
|
|
|
|
|
rcu_read_lock();
|
2022-04-19 21:27:22 -07:00
|
|
|
blkcg = css_to_blkcg(blkcg_css());
|
2018-07-03 08:14:55 -07:00
|
|
|
if (!blkcg)
|
|
|
|
goto out;
|
2023-02-14 11:33:07 -07:00
|
|
|
blkg = blkg_lookup(blkcg, disk->queue);
|
2018-07-03 08:14:55 -07:00
|
|
|
if (!blkg)
|
|
|
|
goto out;
|
2018-12-05 10:10:39 -07:00
|
|
|
if (!blkg_tryget(blkg))
|
2018-07-03 08:14:55 -07:00
|
|
|
goto out;
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
|
|
|
blkcg_maybe_throttle_blkg(blkg, use_memdelay);
|
|
|
|
blkg_put(blkg);
|
2023-02-03 08:03:48 -07:00
|
|
|
put_disk(disk);
|
2018-07-03 08:14:55 -07:00
|
|
|
return;
|
|
|
|
out:
|
|
|
|
rcu_read_unlock();
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* blkcg_schedule_throttle - this task needs to check for throttling
|
2022-12-01 18:17:13 -07:00
|
|
|
* @disk: disk to throttle
|
2019-03-20 13:18:45 -07:00
|
|
|
* @use_memdelay: do we charge this to memory delay for PSI
|
2018-07-03 08:14:55 -07:00
|
|
|
*
|
|
|
|
* This is called by the IO controller when we know there's delay accumulated
|
|
|
|
* for the blkg for this task. We do not pass the blkg because there are places
|
|
|
|
* we call this that may not have that information, the swapping code for
|
2022-09-21 11:05:00 -07:00
|
|
|
* instance will only have a block_device at that point. This set's the
|
2018-07-03 08:14:55 -07:00
|
|
|
* notify_resume for the task to check and see if it requires throttling before
|
|
|
|
* returning to user space.
|
|
|
|
*
|
|
|
|
* We will only schedule once per syscall. You can call this over and over
|
|
|
|
* again and it will only do the check once upon return to user space, and only
|
|
|
|
* throttle once. If the task needs to be throttled again it'll need to be
|
|
|
|
* re-set at the next time we see the task.
|
|
|
|
*/
|
2022-09-21 11:05:00 -07:00
|
|
|
void blkcg_schedule_throttle(struct gendisk *disk, bool use_memdelay)
|
2018-07-03 08:14:55 -07:00
|
|
|
{
|
|
|
|
if (unlikely(current->flags & PF_KTHREAD))
|
|
|
|
return;
|
|
|
|
|
2023-02-03 08:03:48 -07:00
|
|
|
if (current->throttle_disk != disk) {
|
|
|
|
if (test_bit(GD_DEAD, &disk->state))
|
2021-01-24 22:05:28 -07:00
|
|
|
return;
|
2023-02-03 08:03:48 -07:00
|
|
|
get_device(disk_to_dev(disk));
|
2021-01-24 22:05:28 -07:00
|
|
|
|
2023-02-03 08:03:48 -07:00
|
|
|
if (current->throttle_disk)
|
|
|
|
put_disk(current->throttle_disk);
|
|
|
|
current->throttle_disk = disk;
|
2021-01-24 22:05:28 -07:00
|
|
|
}
|
2018-07-03 08:14:55 -07:00
|
|
|
|
|
|
|
if (use_memdelay)
|
|
|
|
current->use_memdelay = use_memdelay;
|
|
|
|
set_notify_resume(current);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* blkcg_add_delay - add delay to this blkg
|
2019-03-20 13:18:45 -07:00
|
|
|
* @blkg: blkg of interest
|
|
|
|
* @now: the current time in nanoseconds
|
|
|
|
* @delta: how many nanoseconds of delay to add
|
2018-07-03 08:14:55 -07:00
|
|
|
*
|
|
|
|
* Charge @delta to the blkg's current delay accumulation. This is used to
|
|
|
|
* throttle tasks if an IO controller thinks we need more throttling.
|
|
|
|
*/
|
|
|
|
void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta)
|
|
|
|
{
|
2020-04-13 09:27:55 -07:00
|
|
|
if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0))
|
|
|
|
return;
|
2018-07-03 08:14:55 -07:00
|
|
|
blkcg_scale_delay(blkg, now);
|
|
|
|
atomic64_add(delta, &blkg->delay_nsec);
|
|
|
|
}
|
|
|
|
|
2020-06-27 00:31:51 -07:00
|
|
|
/**
|
|
|
|
* blkg_tryget_closest - try and get a blkg ref on the closet blkg
|
2020-06-27 00:31:54 -07:00
|
|
|
* @bio: target bio
|
|
|
|
* @css: target css
|
2020-06-27 00:31:51 -07:00
|
|
|
*
|
2020-06-27 00:31:54 -07:00
|
|
|
* As the failure mode here is to walk up the blkg tree, this ensure that the
|
|
|
|
* blkg->parent pointers are always valid. This returns the blkg that it ended
|
|
|
|
* up taking a reference on or %NULL if no reference was taken.
|
2020-06-27 00:31:51 -07:00
|
|
|
*/
|
2020-06-27 00:31:54 -07:00
|
|
|
static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,
|
|
|
|
struct cgroup_subsys_state *css)
|
2020-06-27 00:31:51 -07:00
|
|
|
{
|
2020-06-27 00:31:54 -07:00
|
|
|
struct blkcg_gq *blkg, *ret_blkg = NULL;
|
2020-06-27 00:31:51 -07:00
|
|
|
|
2020-06-27 00:31:54 -07:00
|
|
|
rcu_read_lock();
|
2022-09-21 11:05:01 -07:00
|
|
|
blkg = blkg_lookup_create(css_to_blkcg(css), bio->bi_bdev->bd_disk);
|
2020-06-27 00:31:51 -07:00
|
|
|
while (blkg) {
|
|
|
|
if (blkg_tryget(blkg)) {
|
|
|
|
ret_blkg = blkg;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
blkg = blkg->parent;
|
|
|
|
}
|
2020-06-27 00:31:54 -07:00
|
|
|
rcu_read_unlock();
|
2020-06-27 00:31:51 -07:00
|
|
|
|
|
|
|
return ret_blkg;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* bio_associate_blkg_from_css - associate a bio with a specified css
|
|
|
|
* @bio: target bio
|
|
|
|
* @css: target css
|
|
|
|
*
|
|
|
|
* Associate @bio with the blkg found by combining the css's blkg and the
|
|
|
|
* request_queue of the @bio. An association failure is handled by walking up
|
|
|
|
* the blkg tree. Therefore, the blkg associated can be anything between @blkg
|
|
|
|
* and q->root_blkg. This situation only happens when a cgroup is dying and
|
|
|
|
* then the remaining bios will spill to the closest alive blkg.
|
|
|
|
*
|
|
|
|
* A reference will be taken on the blkg and will be released when @bio is
|
|
|
|
* freed.
|
|
|
|
*/
|
|
|
|
void bio_associate_blkg_from_css(struct bio *bio,
|
|
|
|
struct cgroup_subsys_state *css)
|
|
|
|
{
|
|
|
|
if (bio->bi_blkg)
|
|
|
|
blkg_put(bio->bi_blkg);
|
|
|
|
|
2020-06-27 00:31:53 -07:00
|
|
|
if (css && css->parent) {
|
2020-06-27 00:31:54 -07:00
|
|
|
bio->bi_blkg = blkg_tryget_closest(bio, css);
|
2020-06-27 00:31:53 -07:00
|
|
|
} else {
|
2021-10-14 07:03:30 -07:00
|
|
|
blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg);
|
|
|
|
bio->bi_blkg = bdev_get_queue(bio->bi_bdev)->root_blkg;
|
2020-06-27 00:31:53 -07:00
|
|
|
}
|
2020-06-27 00:31:51 -07:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* bio_associate_blkg - associate a bio with a blkg
|
|
|
|
* @bio: target bio
|
|
|
|
*
|
|
|
|
* Associate @bio with the blkg found from the bio's css and request_queue.
|
|
|
|
* If one is not found, bio_lookup_blkg() creates the blkg. If a blkg is
|
|
|
|
* already associated, the css is reused and association redone as the
|
|
|
|
* request_queue may have changed.
|
|
|
|
*/
|
|
|
|
void bio_associate_blkg(struct bio *bio)
|
|
|
|
{
|
|
|
|
struct cgroup_subsys_state *css;
|
|
|
|
|
2023-12-18 08:27:22 -07:00
|
|
|
if (blk_op_is_passthrough(bio->bi_opf))
|
|
|
|
return;
|
|
|
|
|
2020-06-27 00:31:51 -07:00
|
|
|
rcu_read_lock();
|
|
|
|
|
|
|
|
if (bio->bi_blkg)
|
2022-04-19 21:27:17 -07:00
|
|
|
css = bio_blkcg_css(bio);
|
2020-06-27 00:31:51 -07:00
|
|
|
else
|
|
|
|
css = blkcg_css();
|
|
|
|
|
|
|
|
bio_associate_blkg_from_css(bio, css);
|
|
|
|
|
|
|
|
rcu_read_unlock();
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(bio_associate_blkg);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* bio_clone_blkg_association - clone blkg association from src to dst bio
|
|
|
|
* @dst: destination bio
|
|
|
|
* @src: source bio
|
|
|
|
*/
|
|
|
|
void bio_clone_blkg_association(struct bio *dst, struct bio *src)
|
|
|
|
{
|
2022-06-02 01:12:42 -07:00
|
|
|
if (src->bi_blkg)
|
|
|
|
bio_associate_blkg_from_css(dst, bio_blkcg_css(src));
|
2020-06-27 00:31:51 -07:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(bio_clone_blkg_association);
|
|
|
|
|
2020-06-27 00:31:58 -07:00
|
|
|
static int blk_cgroup_io_type(struct bio *bio)
|
|
|
|
{
|
|
|
|
if (op_is_discard(bio->bi_opf))
|
|
|
|
return BLKG_IOSTAT_DISCARD;
|
|
|
|
if (op_is_write(bio->bi_opf))
|
|
|
|
return BLKG_IOSTAT_WRITE;
|
|
|
|
return BLKG_IOSTAT_READ;
|
|
|
|
}
|
|
|
|
|
|
|
|
void blk_cgroup_bio_start(struct bio *bio)
|
|
|
|
{
|
2022-11-04 17:59:01 -07:00
|
|
|
struct blkcg *blkcg = bio->bi_blkg->blkcg;
|
2020-06-27 00:31:58 -07:00
|
|
|
int rwd = blk_cgroup_io_type(bio), cpu;
|
|
|
|
struct blkg_iostat_set *bis;
|
2021-10-14 16:20:22 -07:00
|
|
|
unsigned long flags;
|
2020-06-27 00:31:58 -07:00
|
|
|
|
2023-05-07 10:06:31 -07:00
|
|
|
if (!cgroup_subsys_on_dfl(io_cgrp_subsys))
|
|
|
|
return;
|
|
|
|
|
2023-02-01 19:18:04 -07:00
|
|
|
/* Root-level stats are sourced from system-wide IO stats */
|
|
|
|
if (!cgroup_parent(blkcg->css.cgroup))
|
|
|
|
return;
|
|
|
|
|
2020-06-27 00:31:58 -07:00
|
|
|
cpu = get_cpu();
|
|
|
|
bis = per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu);
|
2021-10-14 16:20:22 -07:00
|
|
|
flags = u64_stats_update_begin_irqsave(&bis->sync);
|
2020-06-27 00:31:58 -07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the bio is flagged with BIO_CGROUP_ACCT it means this is a split
|
|
|
|
* bio and we would have already accounted for the size of the bio.
|
|
|
|
*/
|
|
|
|
if (!bio_flagged(bio, BIO_CGROUP_ACCT)) {
|
|
|
|
bio_set_flag(bio, BIO_CGROUP_ACCT);
|
2020-06-30 08:54:41 -07:00
|
|
|
bis->cur.bytes[rwd] += bio->bi_iter.bi_size;
|
2020-06-27 00:31:58 -07:00
|
|
|
}
|
|
|
|
bis->cur.ios[rwd]++;
|
|
|
|
|
2022-11-04 17:59:01 -07:00
|
|
|
/*
|
|
|
|
* If the iostat_cpu isn't in a lockless list, put it into the
|
|
|
|
* list to indicate that a stat update is pending.
|
|
|
|
*/
|
|
|
|
if (!READ_ONCE(bis->lqueued)) {
|
|
|
|
struct llist_head *lhead = this_cpu_ptr(blkcg->lhead);
|
|
|
|
|
|
|
|
llist_add(&bis->lnode, lhead);
|
|
|
|
WRITE_ONCE(bis->lqueued, true);
|
|
|
|
}
|
|
|
|
|
2021-10-14 16:20:22 -07:00
|
|
|
u64_stats_update_end_irqrestore(&bis->sync, flags);
|
2023-05-07 10:06:31 -07:00
|
|
|
cgroup_rstat_updated(blkcg->css.cgroup, cpu);
|
2020-06-27 00:31:58 -07:00
|
|
|
put_cpu();
|
|
|
|
}
|
|
|
|
|
2022-04-19 21:27:13 -07:00
|
|
|
bool blk_cgroup_congested(void)
|
|
|
|
{
|
2024-07-16 06:30:58 -07:00
|
|
|
struct blkcg *blkcg;
|
2022-04-19 21:27:13 -07:00
|
|
|
bool ret = false;
|
|
|
|
|
|
|
|
rcu_read_lock();
|
2024-07-16 06:30:58 -07:00
|
|
|
for (blkcg = css_to_blkcg(blkcg_css()); blkcg;
|
|
|
|
blkcg = blkcg_parent(blkcg)) {
|
|
|
|
if (atomic_read(&blkcg->congestion_count)) {
|
2022-04-19 21:27:13 -07:00
|
|
|
ret = true;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
rcu_read_unlock();
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-07-03 08:14:52 -07:00
|
|
|
module_param(blkcg_debug_stats, bool, 0644);
|
|
|
|
MODULE_PARM_DESC(blkcg_debug_stats, "True if you want debug stats, false if not");
|