M2/NVM vs M2/Sata vs SATA/6GB benchmarks
Marc MERLIN
marc at merlins.org
Thu Feb 25 10:20:09 PST 2016
On Sun, Feb 14, 2016 at 04:32:58AM -0500, Matthew Wilcox wrote:
>
> Hi Marc,
>
> I shan't have time to look at this for another week, but I've added
> the NVMe mailing list so somebody else can comment in the meantime.
Hi Matthew :)
> On Sat, Feb 13, 2016 at 07:13:33PM -0800, Marc MERLIN wrote:
> > Hi Matthew,
> >
> > I didn't find a list for NVMe, I'll Cc linux-ide in case this is useful
> > to others.
> >
> > I just got a Thinkpad P70 skylake laptop and thought I'd try 3 SSDs:
> > NVMe/M2 vs Sata/M2 vs Sata/6GB.
> > I'm not a benchmarking filesystem expert, so my apologies if I made some
> > stupid choices, but hopefully the benchmarks are still useful.
> >
> > I'll give the conclusions first. The buffer cache issue with NVMe is weird,
> > maybe there is a non default setting I need to tweak?
> >
> > a) NVMe is faster for big contiguous I/O, even through dmcrypt
> > b) M2 NVMe and M2 sata do not seem to go through buffer cache for writes, but SATA/6GB does
> > c) When using buffer cache, SATA/6GB beats NVMe/M2 and SATA/M2 by a lot on block reads. Why?
> > d) for random I/O, NVMe is up to 50% faster, and faster with ext4 than btrfs
> > e) Encrypted NVMe is a bit faster than non encrypted Sata/M2
> > f) At $328 for 512GB M2/NVMe vs $168 for 500GB M2/Sata, unless you're doing big contiguous I/O, the 2x price difference is hard to justify, cached I/O and random I/O are not significantly faster on NVMe.
> >
> >
> >
> > 3 SSDs:
> > 1) M2/SATA Samsung 850 EVO 500 GB M.2 3.5-Inch SSD (MZ-N5E500BW) 33c/GB
> >
> > 2) M2/NVMe: Samsung 950 PRO -Series 512GB PCIe NVMe - M.2 Internal SSD 2-Inch MZ-V5P512BW 64c/GB
> > 70:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01)
> >
> > 3) SATA6G: Samsung 850 EVO 2 TB 2.5-Inch SATA III Internal SSD (MZ-75E2T0B/AM) 30c/GB
> >
> > For price comparison
> > Seagate 2TB Laptop HDD SATA 6Gb/s 32MB Cache 2.5-Inch Internal Bare Drive (ST2000LM003) 4.6c/GB
> >
> > Laptop: Skylake Lenovo Thinkpad P70
> > Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz x 8 (4 cores HT)
> >
> > Filesystem: btrfs on top of
> > cryptsetup luksFormat --align-payload=8192 -s 256 -c aes-xts-plain64 /dev/device
> > ddd test: sync; dd if=/mnt/ram/file of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=100M
> > dd0 test: sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> >
> > Summary:
> > 0) NVMe only really looks faster on big block sequential read/write (indeed 4X read, 2X write).
> > On Random I/O, it's no faster. Other overheads kill its performance advantage.
> > 1) For Cached I/O NVMe seems slowed down by dmcrypt in hdparm/ioz ranread/write whereas M2/Sata is not
> > 2) 2TB Sata 6GB SSD is faster for read/write encrypted than NVMe is non encrypted (except hdparm)
> > 3) iozone through block cache is faster on M2/SATA than M2/NVMe, weird
> > 4) iozone through block cache is much faster for everything on Sata6G, also weird
> > 5) iozone direct I/O shows NVMe is fastest (by 50% only) and external SATA just a bit faster than M2/Sata
> > 6) when using dmcrypt, NVMe is only marginally faster in iozone than Sata/M2 or Sata/6G
> > 7) kernel build speed is same on SATA/M2 and NVMe/M2, encrypted or not, but SATA6G is 10% faster (probably the same effect that block cache works better on SATA/6G than SATA/M2 or NVMe/M2)
> > 8) ext4 does better on iozone -I (directIO) than btrfs, 2x as fast on random read for NVMe
> >
> >
> >
> > SATA/M2 dmcrypt SATA/M2 NVMe dmcrypt NVMe SATA6GB/dmcrypt
> > hdparm -t 534MB/s 517MB/s 1307MB/s 2140MB/s 534MB/s
> > ddd 10GB read 550MB/s 560MB/s 2000MB/s 2400MB/s 550MB/s
> > ddd 10GB write 500MB/s 503MB/s 1300MB/s 1200MB/s 506MB/s
> >
> > dd with /dev/zero hits optimizations that actually penalize NVMe:
> > dd0 10GB read 7GB/s 6.7GB/s 2.3GB/s 4.4GB/s 6.6GB/s
> > dd0 10GB write 495MB/s 503MB/s 1.3GB/s 1.3GB/s 2.0GB/s
> >
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2 (cache makes Sata6G faster than M2, weird):
> > ioz read 4KB 1760MB/s 1790MB/s 675MB/s 766MB/s 7863MB/s
> > ioz write 4KB 268MB/s 298MB/s 295MB/s 291MB/s 777MB/s
> > ioz ranread4KB 5535MB/s 7392MB/s 3185MB/s 7261MB/s 7892MB/s
> > ioz ranwrite4KB 288MB/s 329MB/s 205MB/s 381MB/s 796MB/s
> >
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (direct IO):
> > SATA/M2 dmcrypt SATA/M2 NVMe dmcrypt NVMe SATA6GB
> > dmcrypt notcrypted
> > btrfs btrfs ext4 btrfs btrfs ext4 btrfs ext4
> > ioz read 4KB 255MB/s 303MB/s 408MB/s 303MB/s 396MB/s 562MB/s 310MB/s 379MB/s
> > ioz write 4KB 195MB/s 238MB/s 344MB/s 257MB/s 358MB/s 361MB/s 233MB/s 365MB/s
> > ioz ranread4KB 305MB/s 351MB/s 482MB/s 476MB/s 699MB/s1434MB/s 309MB/s 475MB/s
> > ioz ranwrite4KB 265MB/s 223MB/s 392MB/s 260MB/s 217MB/s 315MB/s 254MB/s 366MB/s
> >
> >
> > Details of each test, per type of drive;
> > ================================================================================
> > saruman M2 SATA 512GB encrypted btrfs
> >
> > saruman:/tmp# hdparm -t /dev/mapper/cryptroot
> > Timing buffered disk reads: 1604 MB in 3.00 seconds = 534.03 MB/sec
> >
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> > random random
> > kB reclen write rewrite read reread read write
> > 4096 4 257076 281089 1761711 6094733 6066754 290373
> > 4096 512 277108 295167 1588844 8426437 11128258 318731
> > 4096 4096 268940 308248 1760087 8062616 8377131 320802
> >
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> > 4096 4 38626 45227 102236 111141 30936 64317
> > 4096 512 138467 128878 134674 137832 140067 126248
> > 4096 4096 195439 273100 255379 307300 305605 265093
> >
> >
> > saruman:/mnt/mnt# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 21.1761 s, 495 MB/s
> > 10485760000 bytes (10 GB) copied, 1.49373 s, 7.0 GB/s
> >
> > kernel 4.4.1 make -j8:
> > real 17m54.712s
> > user 126m26.620s
> > sys 6m21.948s
> >
> > btrfs send/receive encrypted partition to non encrypted, 87GB: 10mn30
> >
> > saruman M2 SATA non encrypted btrfs
> > -----------------------------------
> > saruman:/mnt/mnt4# hdparm -t /dev/sdc4
> > /dev/sdc4:
> > Timing buffered disk reads: 1554 MB in 3.00 seconds = 517.93 MB/sec
> >
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> > random random
> > kB reclen write rewrite read reread read write
> > 4096 4 285020 297565 1739944 7434579 6605595 305785
> > 4096 512 310937 336617 1298278 3703511 5483860 299416
> > 4096 4096 298433 317101 1790911 7392988 7434579 337476
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> > 4096 4 47219 93497 118900 126065 39160 86462
> > 4096 512 214381 213388 194307 209215 206951 209289
> > 4096 4096 238792 221839 303768 352951 351342 223618
> > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> > 4096 4 119524 146035 150173 159105 53090 140697
> > 4096 512 358954 354810 366466 423088 419760 375542
> > 4096 4096 344087 398828 408259 478337 482569 392487
> >
> > saruman:/mnt/mnt3# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 20.8537 s, 503 MB/s
> > 10485760000 bytes (10 GB) copied, 1.55651 s, 6.7 GB/s
> >
> > kernel 4.4.1 make -j8:
> > real 17m55.612s
> > user 126m31.952s
> > sys 6m27.452s
> >
> >
> > ********************************************************************************
> > saruman M2 NVMe 512GB encrypted btrfs
> >
> > saruman:/tmp# hdparm -t /dev/mapper/cryptroot2
> > /dev/mapper/cryptroot2:
> > Timing buffered disk reads: 3924 MB in 3.00 seconds = 1307.56 MB/sec
> >
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> > random random
> > kB reclen write rewrite read reread read write
> > 4096 4 284205 341651 742181 6195843 6178018 341101
> > 4096 512 314351 319573 864883 8865630 8902382 384993
> > 4096 4096 295741 201042 675234 3281854 3185110 205974
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> > 4096 4 80042 83160 122616 127129 41421 53963
> > 4096 512 168713 153333 167710 209923 198123 168026
> > 4096 4096 257968 258566 303023 490828 476440 260014
> >
> >
> > saruman:/mnt/mnt2# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 8.01494 s, 1.3 GB/s
> > 10485760000 bytes (10 GB) copied, 4.63397 s, 2.3 GB/s
> >
> > kernel 4.4.1 make -j8:
> > real 17m57.513s
> > user 126m58.360s
> > sys 6m25.164s
> >
> > btrfs send/receive encrypted partition to non encrypted, 87GB: 7mn
> > (33% faster than Sata M2)
> >
> > saruman M2 NVMe non encrypted btrfs
> > -----------------------------------
> > saruman:/mnt/mnt4# hdparm -t /dev/nvme0n1p4
> > /dev/nvme0n1p4:
> > Timing buffered disk reads: 6422 MB in 3.00 seconds = 2140.38 MB/sec
> >
> >
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> > random random
> > kB reclen write rewrite read reread read write
> > 4096 4 274808 271851 684271 2634219 2280673 177630
> > 4096 512 290191 336755 668196 3362795 4039655 184779
> > 4096 4096 291427 340345 766588 7355007 7261741 381768
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> > 4096 4 94814 117116 147645 159695 44075 108039
> > 4096 512 271130 267800 239969 325933 301841 271062
> > 4096 4096 358602 275933 396591 719099 699939 217665
> > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> > 4096 4 107281 200735 205900 263035 54832 124219
> > 4096 512 537478 374641 525060 1173664 1139338 558807
> > 4096 4096 361545 526137 562632 1483477 1434069 315883
> >
> > saruman:/mnt/mnt4# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 7.90653 s, 1.3 GB/s
> > 10485760000 bytes (10 GB) copied, 2.35863 s, 4.4 GB/s
> >
> > kernel 4.4.1 make -j8:
> > real 17m54.221s
> > user 126m46.264s
> > sys 6m10.592s
> >
> >
> > ********************************************************************************
> > saruman Samsung Evo 850 2TB SSD encrypted btrfs
> >
> > hdparm -t
> > Timing buffered disk reads: 1606 MB in 3.00 seconds = 534.87 MB/sec
> >
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> > random random
> > kB reclen write rewrite read reread read write
> > 4096 4 513924 738957 8586475 9144037 8017464 697665
> > 4096 512 772100 820362 9287391 10136778 10586522 840551
> > 4096 4096 719099 777517 7863339 8047509 7892238 796108
> >
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2
> > 4096 4 76626 47961 92886 80947 36063 63788
> > 4096 512 135087 138998 143127 150981 144393 132219
> > 4096 4096 233245 234058 310797 311552 309855 254080
> >
> > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (unencrypted)
> > 4096 4 118013 141212 130575 111534 42343 138654
> > 4096 512 336347 312192 332847 414404 424542 342707
> > 4096 4096 365227 364638 379298 484160 475122 366536
> >
> >
> > saruman:/tmp# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 5.23919 s, 2.0 GB/s
> > 10485760000 bytes (10 GB) copied, 1.58287 s, 6.6 GB/s
> >
> >
> > kernel 4.4.1 make -j8:
> > real 16m4.305s
> > user 104m0.816s
> > sys 7m2.832s
> >
> >
> > --
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> > Microsoft is to operating systems ....
> > .... what McDonalds is to gourmet cooking
> > Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
>
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
More information about the Linux-nvme
mailing list