[PATCH v3 00/10] Read/Write with meta/integrity
Anuj Gupta
anuj20.g at samsung.com
Fri Aug 23 03:38:00 PDT 2024
This adds a new io_uring interface to exchange meta along with read/write.
Interface:
Meta information is represented using a newly introduced 'struct io_uring_meta'.
Application sets up a SQE128 ring, and prepares io_uring_meta within second
SQE. Application populates 'struct io_uring_meta' fields as below:
* meta_type: describes type of meta that is passed. Currently one type
"Integrity" is supported.
* meta_flags: these are meta-type specific flags. Three flags are exposed for
integrity type, namely INTEGRITY_CHK_GUARD/APPTAG/REFTAG.
* meta_len: length of the meta buffer
* meta_addr: address of the meta buffer
* app_tag: optional application-specific 16b value; this goes along with
INTEGRITY_CHK_APPTAG flag.
Block path (direct IO) , NVMe and SCSI driver are modified to support
this.
The first three patches are required to make the user metadata split
work correctly.
Patch 4,5 are prep patches.
Patch 6 adds the io_uring support.
Patch 7 gives us unified interface for user and kernel generated
integrity.
Patch 8 adds the support for block direct IO, patch 9 for NVMe, and
patch 10 for SCSI.
Some of the design choices came from this discussion [2].
Example program on how to use the interface is appended below [3]
Tree:
https://github.com/SamsungDS/linux/tree/feat/pi_us_v3
Testing:
has been done by modifying fio to use this interface.
https://github.sec.samsung.net/DS8-MemoryOpenSource/fio/tree/feat/test-meta-v4
Changes since v2:
https://lore.kernel.org/linux-block/20240626100700.3629-1-anuj20.g@samsung.com/
- io_uring error handling styling (Gabriel)
- add documented helper to get metadata bytes from data iter (hch)
- during clone specify "what flags to clone" rather than
"what not to clone" (hch)
- Move uio_meta defination to bio-integrity.h (hch)
- Rename apptag field to app_tag (hch)
- Change datatype of flags field in uio_meta to bitwise (hch)
- Don't introduce BIP_USER_CHK_FOO flags (hch, martin)
- Driver should rely on block layer flags instead of seeing if it is
user-passthrough (hch)
- update the scsi code for handling user-meta (hch, martin)
Changes since v1:
https://lore.kernel.org/linux-block/20240425183943.6319-1-joshi.k@samsung.com/
- Do not use new opcode for meta, and also add the provision to introduce new
meta types beyond integrity (Pavel)
- Stuff IOCB_HAS_META check in need_complete_io (Jens)
- Split meta handling in NVMe into a separate handler (Keith)
- Add meta handling for __blkdev_direct_IO too (Keith)
- Don't inherit BIP_COPY_USER flag for cloned bio's (Christoph)
- Better commit descriptions (Christoph)
Changes since RFC:
- modify io_uring plumbing based on recent async handling state changes
- fixes/enhancements to correctly handle the split for meta buffer
- add flags to specify guard/reftag/apptag checks
- add support to send apptag
[1] https://lore.kernel.org/linux-block/719d2e-b0e6-663c-ec38-acf939e4a04b@redhat.com/
[2] https://lore.kernel.org/linux-block/20240705083205.2111277-1-hch@lst.de/
[3]
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <linux/io_uring.h>
#include <linux/types.h>
#include "liburing.h"
/* write data/meta. read both. compare. send apptag too.
* prerequisite:
* unprotected xfer: format namespace with 4KB + 8b, pi_type = 0
* protected xfer: format namespace with 4KB + 8b, pi_type = 1
*/
#define DATA_LEN 4096
#define META_LEN 8
struct t10_pi_tuple {
__be16 guard;
__be16 apptag;
__be32 reftag;
};
int main(int argc, char *argv[])
{
struct io_uring ring;
struct io_uring_sqe *sqe = NULL;
struct io_uring_cqe *cqe = NULL;
void *wdb,*rdb;
char wmb[META_LEN], rmb[META_LEN];
char *data_str = "data buffer";
char *meta_str = "meta";
int fd, ret, blksize;
struct stat fstat;
unsigned long long offset = DATA_LEN;
struct t10_pi_tuple *pi;
struct io_uring_meta *md;
if (argc != 2) {
fprintf(stderr, "Usage: %s <block-device>", argv[0]);
return 1;
};
if (stat(argv[1], &fstat) == 0) {
blksize = (int)fstat.st_blksize;
} else {
perror("stat");
return 1;
}
if (posix_memalign(&wdb, blksize, DATA_LEN)) {
perror("posix_memalign failed");
return 1;
}
if (posix_memalign(&rdb, blksize, DATA_LEN)) {
perror("posix_memalign failed");
return 1;
}
strcpy(wdb, data_str);
strcpy(wmb, meta_str);
fd = open(argv[1], O_RDWR | O_DIRECT);
if (fd < 0) {
printf("Error in opening device\n");
return 0;
}
ret = io_uring_queue_init(8, &ring, IORING_SETUP_SQE128);
if (ret) {
fprintf(stderr, "ring setup failed: %d\n", ret);
return 1;
}
/* write data + meta-buffer to device */
sqe = io_uring_get_sqe(&ring);
if (!sqe) {
fprintf(stderr, "get sqe failed\n");
return 1;
}
io_uring_prep_write(sqe, fd, wdb, DATA_LEN, offset);
md = (struct io_uring_meta *) sqe->big_sqe_cmd;
md->meta_type = META_TYPE_INTEGRITY;
md->meta_addr = (__u64)wmb;
md->meta_len = META_LEN;
/* flags to ask for guard/reftag/apptag*/
md->meta_flags = INTEGRITY_CHK_APPTAG;
md->app_tag = 0x1234;
pi = (struct t10_pi_tuple *)wmb;
pi->apptag = 0x3412;
ret = io_uring_submit(&ring);
if (ret <= 0) {
fprintf(stderr, "sqe submit failed: %d\n", ret);
return 1;
}
ret = io_uring_wait_cqe(&ring, &cqe);
if (!cqe) {
fprintf(stderr, "cqe is NULL :%d\n", ret);
return 1;
}
if (cqe->res < 0) {
fprintf(stderr, "write cqe failure: %d", cqe->res);
return 1;
}
io_uring_cqe_seen(&ring, cqe);
/* read data + meta-buffer back from device */
sqe = io_uring_get_sqe(&ring);
if (!sqe) {
fprintf(stderr, "get sqe failed\n");
return 1;
}
io_uring_prep_read(sqe, fd, rdb, DATA_LEN, offset);
md = (struct io_uring_meta *) sqe->big_sqe_cmd;
md->meta_type = META_TYPE_INTEGRITY;
md->meta_addr = (__u64)rmb;
md->meta_len = META_LEN;
md->meta_flags = INTEGRITY_CHK_APPTAG;
md->app_tag = 0x1234;
ret = io_uring_submit(&ring);
if (ret <= 0) {
fprintf(stderr, "sqe submit failed: %d\n", ret);
return 1;
}
ret = io_uring_wait_cqe(&ring, &cqe);
if (!cqe) {
fprintf(stderr, "cqe is NULL :%d\n", ret);
return 1;
}
if (cqe->res < 0) {
fprintf(stderr, "read cqe failure: %d", cqe->res);
return 1;
}
io_uring_cqe_seen(&ring, cqe);
if (strncmp(wmb, rmb, META_LEN))
printf("Failure: meta mismatch!, wmb=%s, rmb=%s\n", wmb, rmb);
if (strncmp(wdb, rdb, DATA_LEN))
printf("Failure: data mismatch!\n");
io_uring_queue_exit(&ring);
free(rdb);
free(wdb);
return 0;
}
Anuj Gupta (7):
block: define set of integrity flags to be inherited by cloned bip
block: introduce a helper to determine metadata bytes from data iter
block: handle split correctly for user meta bounce buffer
block: modify bio_integrity_map_user to accept iov_iter as argument
io_uring/rw: add support to send meta along with read/write
block,nvme: introduce BIP_CHECK_GUARD/REFTAG/APPTAG bip_flags
scsi: add support for user-meta interface
Kanchan Joshi (3):
block: define meta io descriptor
block: add support to pass user meta buffer
nvme: add handling for app_tag
block/bio-integrity.c | 71 ++++++++++++++++++++++++++++++-----
block/fops.c | 25 ++++++++++++
block/t10-pi.c | 6 +++
drivers/nvme/host/core.c | 24 +++++++-----
drivers/nvme/host/ioctl.c | 11 +++++-
drivers/scsi/sd.c | 25 +++++++++++-
drivers/scsi/sd_dif.c | 2 +-
include/linux/bio-integrity.h | 33 ++++++++++++++--
include/linux/blk-integrity.h | 17 +++++++++
include/linux/fs.h | 1 +
include/uapi/linux/io_uring.h | 32 ++++++++++++++++
io_uring/io_uring.c | 6 +++
io_uring/rw.c | 70 ++++++++++++++++++++++++++++++++--
io_uring/rw.h | 10 ++++-
14 files changed, 302 insertions(+), 31 deletions(-)
--
2.25.1
More information about the Linux-nvme
mailing list