UBIFS: Advice request on how to reduce orph_buf size

[Artem Bityutskiy]

On Fri, 2012-11-16 at 12:14 +0100, Josselin Costanzi wrote:
> In order to reduce UBIFS memory consumption when used in RW I'm trying to
> reduce orph_buf size from one LEB to max_write_size. I changed avail_orphs and
> tot_avail_orphs functions to report correct orphan counts but I'm now stuck
> with the consolidate function. I don't see how I could atomically write all my
> orphans in a new LEB with a max_write_size sized orph_buf. One potential
> solution would be to try allocating a LEB sized buffer only when consolidating
> the orphans but this means commit_orphans may now fail due to memory exhaustion.
> 
> Have any of you an idea on how to correctly do this? What would be an acceptable
> solution?

Hi,

well, it does not look like you can easily do this. The difficult
solution would be to implement scatter-gather lists support in UBI and
UBIFS, so that it would not require continuous buffers at all. Then may
of the LEB-sized buffers would not be needed at all.

I mean, why do we allocate a lot of LEB-sized buggers on mount time?
Simply because later we may have no memory, and we do not want UBIFS to
fail at random point with an error like "cannot write your files - no
memeory".

But if we implement SC support, we will be doing small allocations
instead of 128KiB+ allocations. Small allocations, like allocating one
page, do not fail. I mean, thy can fail sometimes, but extremely rarely.
And if they do, your system is dead anyway. Besides, the kernel always
have some pool of memory for urgent needs. Additionally, we can always
create our own memory pool with some amount of pages which are always
guaranteed to be there.

So, this is basically my answer. This is a big work. But it would make
UBIFS use a lot less RAM. It would also solve another problem people
scream for years about - UBI/UBIFS use vmalloc() which is not DMA-able
in many ARM systems. With SC lists we'd allocate smaller amount of
memory (mostly one page, and sometimes 2 pages), and we could use normal
kmalloc().

CCing Ezequiel Garcia <elezegarcia at gmail.com> - may be he is
enthusiastic enough to make a proposal about this in http://elinux.org.

-- 
Best Regards,
Artem Bityutskiy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.infradead.org/pipermail/linux-mtd/attachments/20121130/28e2e21a/attachment.sig>