[LSF/MM/BPF TOPIC] Large block for I/O

[Keith Busch]

On Fri, Dec 22, 2023 at 11:23:26AM +0300, Viacheslav Dubeyko wrote:
> > On Dec 21, 2023, at 11:33 PM, Bart Van Assche <bvanassche at acm.org> wrote:
> > I'm interested in this topic. But I'm wondering whether the disadvantages of
> > large blocks will be covered? Some NAND storage vendors are less than
> > enthusiast about increasing the logical block size beyond 4 KiB because it
> > increases the size of many writes to the device and hence increases write
> > amplification.
> > 
> 
> I  am also interested in this discussion. Every SSD manufacturer carefully hides
> the details of architecture and FTL´s behavior. I believe that switching on bigger
> logical size (like 8KB, 16KB, etc) could be even better for SSD's internal mapping
> scheme and erase blocks management. I assume that it could require significant
> reworking the firmware and, potentially, ASIC logic. This could be the main pain
> for SSD manufactures. Frankly speaking, I don´t see the direct relation between
> increasing logical block size and increasing write amplification. If you have 16KB
> logical block size on SSD side and file system will continue to use 4KB logical
> block size, then, yes, I can see the problem. But if file system manages the space
> in 16KB logical blocks and carefully issue the I/O requests of proper size, then
> everything should be good. Again, FTL is simply trying to write logical blocks into
> erase block. And we have, for example, 8MB erase block, then mapping and writing
> 16KB logical blocks looks like more beneficial operation compared with 4KB logical
> block.

If the host really wants to write in small granularities, then larger
block sizes just shifts the write amplification from the device to the
host, which seems worse than letting the device deal with it.

I've done some early profiling on my fleet and there are definitely
applications that overwhelming prefer larger writes. Those should be
great candidates to use these kinds of logical block formats. It's
already flash-friendly, but aligning filesystems and memory management
to the same granularity is a nice plus.

Other applications, though, still need 4k writes. Turning those to RMW
on the host to modify 4k in the middle of a 16k block is obviously a bad
fit.

Anyway, your mileage may vary. This example BPF program provides an okay
starting point for examining disk usage to see if large logical block
sizes are a good fit for your application:

  https://github.com/iovisor/bpftrace/blob/master/tools/bitesize.bt
 
> So, I see more troubles on file systems side to support bigger logical
> size. For example, we discussed the 8KB folio size support recently.
> Matthew already shared the patch for supporting 8KB folio size, but
> everything should be carefully tested. Also, I experienced the issue
> with read ahead logic. For example, if I format my file system volume
> with 32KB logical block, then read ahead logic returns to me 16KB
> folios that was slightly surprising to me. So, I assume we can find a
> lot of potential issues on file systems side for bigger logical size
> from the point of view of efficiency of metadata and user data
> operations.  Also, high-loaded systems could have fragmented memory
> that could make the memory allocation more tricky operation. I mean
> here that it could be not easy to allocate one big folio.
> Log-structured file systems can easily aligned write I/O requests for
> bigger logical size. But in-place update file systems can increase
> write amplification for bigger logical size because of necessity to
> flush bigger portion of data for small modification. However, FTL can
> use delta-encoding and smart logic of compaction several logical
> blocks into one NAND flash page. And, by the way, NAND flash page
> usually is bigger than 4KB.