UBIFS: Advice request on how to reduce orph_buf size

[Ezequiel Garcia]

On Fri, Nov 30, 2012 at 6:31 AM, Artem Bityutskiy <dedekind1 at gmail.com> wrote:
> On Fri, 2012-11-16 at 12:14 +0100, Josselin Costanzi wrote:
>> In order to reduce UBIFS memory consumption when used in RW I'm trying to
>> reduce orph_buf size from one LEB to max_write_size. I changed avail_orphs and
>> tot_avail_orphs functions to report correct orphan counts but I'm now stuck
>> with the consolidate function. I don't see how I could atomically write all my
>> orphans in a new LEB with a max_write_size sized orph_buf. One potential
>> solution would be to try allocating a LEB sized buffer only when consolidating
>> the orphans but this means commit_orphans may now fail due to memory exhaustion.
>>
>> Have any of you an idea on how to correctly do this? What would be an acceptable
>> solution?
>
> Hi,
>
> well, it does not look like you can easily do this. The difficult
> solution would be to implement scatter-gather lists support in UBI and
> UBIFS, so that it would not require continuous buffers at all. Then may
> of the LEB-sized buffers would not be needed at all.
>
> I mean, why do we allocate a lot of LEB-sized buggers on mount time?
> Simply because later we may have no memory, and we do not want UBIFS to
> fail at random point with an error like "cannot write your files - no
> memeory".
>
> But if we implement SC support, we will be doing small allocations
> instead of 128KiB+ allocations. Small allocations, like allocating one
> page, do not fail. I mean, thy can fail sometimes, but extremely rarely.
> And if they do, your system is dead anyway. Besides, the kernel always
> have some pool of memory for urgent needs. Additionally, we can always
> create our own memory pool with some amount of pages which are always
> guaranteed to be there.
>
> So, this is basically my answer. This is a big work. But it would make
> UBIFS use a lot less RAM. It would also solve another problem people
> scream for years about - UBI/UBIFS use vmalloc() which is not DMA-able
> in many ARM systems. With SC lists we'd allocate smaller amount of
> memory (mostly one page, and sometimes 2 pages), and we could use normal
> kmalloc().
>
> CCing Ezequiel Garcia <elezegarcia at gmail.com> - may be he is
> enthusiastic enough to make a proposal about this in http://elinux.org.
>

Yes, indeed it sounds very interesting. I'll think about it.

Artem: is there any other place in the kernel you can point me at
that implement something similar to this SC list?

Thanks for Ccing me,

    Ezequiel