kevin/linux
1
Fork 0
Code

io_uring/uring_cmd: defer SQE copying until it's needed

Browse Source 
The previous commit turned on async data for uring_cmd, and did the
basic conversion of setting everything up on the prep side. However, for
a lot of use cases, -EIOCBQUEUED will get returned on issue, as the
operation got successfully queued. For that case, a persistent SQE isn't
needed, as it's just used for issue.

Unless execution goes async immediately, defer copying the double SQE
until it's necessary.

This greatly reduces the overhead of such commands, as evidenced by
a perf diff from before and after this change:

    10.60%     -8.58%  [kernel.vmlinux]  [k] io_uring_cmd_prep

where the prep side drops from 10.60% to ~2%, which is more expected.
Performance also rises from ~113M IOPS to ~122M IOPS, bringing us back
to where it was before the async command prep.

Tested-by: Anuj Gupta <anuj20.g@samsung.com>
Reviewed-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
This commit is contained in:
Jens Axboe 2024-03-20 15:23:47 -06:00
parent d10f19dff5
commit 5eff57fa9f
1 changed files with 19 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
io_uring/uring_cmd.c
View File

@ -182,12 +182,18 @@ static int io_uring_cmd_prep_setup(struct io_kiocb *req,
struct uring_cache *cache; struct uring_cache *cache;
cache = io_uring_async_get(req); cache = io_uring_async_get(req);
if (cache) { if (unlikely(!cache))
memcpy(cache->sqes, sqe, uring_sqe_size(req->ctx)); return -ENOMEM;
ioucmd->sqe = req->async_data;
if (!(req->flags & REQ_F_FORCE_ASYNC)) {
/* defer memcpy until we need it */
ioucmd->sqe = sqe;
return 0; return 0;
} }
return -ENOMEM;
memcpy(req->async_data, sqe, uring_sqe_size(req->ctx));
ioucmd->sqe = req->async_data;
return 0;
} }
int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
@ -245,8 +251,15 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags)
} }
ret = file->f_op->uring_cmd(ioucmd, issue_flags); ret = file->f_op->uring_cmd(ioucmd, issue_flags);
if (ret == -EAGAIN || ret == -EIOCBQUEUED) if (ret == -EAGAIN) {
return ret; struct uring_cache *cache = req->async_data;
if (ioucmd->sqe != (void *) cache)
memcpy(cache, ioucmd->sqe, uring_sqe_size(req->ctx));
return -EAGAIN;
} else if (ret == -EIOCBQUEUED) {
return -EIOCBQUEUED;
}
if (ret < 0) if (ret < 0)
req_set_fail(req); req_set_fail(req);