[PATCH 7/7] mailbox: goog-mba: Introduce the goog-mba mailbox driver

[Jassi Brar]

On Mon, Sep 21, 2026 at 5:19 PM Doug Anderson <dianders at chromium.org> wrote:
>
> Hi,
>
> On Sun, Sep 20, 2026 at 3:46 PM Jassi Brar <jassisinghbrar at gmail.com> wrote:
> >
> > > The numbers here for interrupt latency are made up for my example and
> > > I haven't personally measured them, but I think it's not completely
> > > absurd to say that interrupt latency (on both the Linux and remote
> > > sides) dominates the communication path.
> > >
> > Yes, the numbers do look biased. It takes 50us for remote to get the
> > irq and act upon it before ACKing but it takes 100us for that ACK to
> > get back.
> > And the benefit will be hard to achieve - it involves three unrelated
> > clk_prepare() requests done within 50us often enough. When the stars
> > align you save 300us on a clk_prepare()
> >
> > It feels you are trying to optimize a non-issue. clk_prepare() is
> > expected to be slow and anyways shouldn't be frequent enough from all
> > devices to give noticeable benefit.
>
> Fair enough. I've jumped into a pre-existing design. Let me see if I
> can find old information or gather evidence myself. Then with real
> data we can figure out what makes sense. OK, gathered some data...
>
> FWIW, to explain things clearly, I have been simplifying by saying
> that just clock prepare/unprepare goes over this channel. In reality,
> there is much more traffic. On Pixel 10, the mailbox using queue mode
> like this connects to the "CPM" (central power manager). Looking at
> the device tree, we see the following things using this mailbox:
> * One of the main clock controllers in the system.
> * Most of the power domains in the system
> * Devfreq controllers
> * Thermal controllers
> * A GPIO controller
> * A reset controller
> * An interrupt controller
> * A RTC
> * A pile of other stuff
>
> So basically a whole crap-ton of resources are managed over this
> mailbox. The team designing this Phone apparently decided that the
> mailbox is one of the primary communication pipelines in the system.
> Yes, everything that communicates over the mailbox needs to be able to
> sleep, but that doesn't mean we shouldn't keep it fast if possible.
>
> On an off-the-shelf Pixel 10, one can spy on the CPM mailbox like this:
>
> echo 1 > /sys/kernel/tracing/events/goog_mba_ctrl/enable
> echo 1 > /sys/kernel/tracing/tracing_on
> echo "" > /sys/kernel/tracing/trace
> cat /sys/kernel/tracing/trace_pipe | grep 'process_q_t\|send_data_q'
>
> When I do that, messages spew by pretty much constantly showing just
> how busy this mailbox is. I can see instances where the queue is
> actually utilized like this:
>
> cat /sys/kernel/tracing/trace_pipe | \
>   grep 'process_q_t\|send_data_q' | \
>   grep -C10 'eqs_completed=[^1]\|anding_msg=[^0]'
>
> If I do that, it's quite easy to see the queue being used. If I use
> the device actively, I get several hits per second. A few examples
> traces (using sed to shorten them slightly):
>
> 6946.884142: send_q: {0xa0016969,0xf,0x2,0x0} tx_q_wr_ptr=4
> 6946.884496: send_q: {0xa0026969,0x7,0x1,0x0} tx_q_wr_ptr=5
> 6946.884851: q_txdone: tx_q_rd_ptr=4 reqs_completed=2 outstanding_msgs=0
> 6946.884880: q_txdone: tx_q_rd_ptr=5 reqs_completed=1 outstanding_msgs=0
>
> Here you can see that it took 709 us to get the response to the first
> message. ...but, luckily we didn't have to wait for that 709 us before
> sending the second message. That means we still got both responses at
> once. Note that things aren't always so slow. The next messages
> through this mailbox only took 42 us.
>
> 6946.885000: send_q: {0xa0036969,0x7,0x1,0x0} tx_q_wr_ptr=6
> 6946.885042: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0
>
> Here's a pretty big usage of the queue. I assume the CPM was busy at the time:
>
> 7751.863328: send_q: {0xa0050708,0x1020001,0x0,0x0} tx_q_wr_ptr=7
> 7751.863619: send_q: {0xa0060708,0x1030001,0x0,0x0} tx_q_wr_ptr=0
> 7751.863850: send_q: {0xa0070708,0x1000001,0x0,0x0} tx_q_wr_ptr=1
> 7751.864123: send_q: {0xa0080708,0x1010001,0x0,0x0} tx_q_wr_ptr=2
> 7751.864329: q_txdone: tx_q_rd_ptr=7 reqs_completed=4 outstanding_msgs=0
> 7751.864338: q_txdone: tx_q_rd_ptr=0 reqs_completed=3 outstanding_msgs=0
> 7751.864347: q_txdone: tx_q_rd_ptr=1 reqs_completed=2 outstanding_msgs=0
> 7751.864378: q_txdone: tx_q_rd_ptr=2 reqs_completed=1 outstanding_msgs=0
>
> A full millisecond before the first response, but luckily we got them
> all at once.
>
> ...and this one is pretty interesting here:
>
> 7782.321344: send_q: {0xa0040708,0x30001,0x0,0x0} tx_q_wr_ptr=4
> 7782.321481: send_q: {0xa0150708,0x10001,0x0,0x0} tx_q_wr_ptr=5
> 7782.321769: send_q: {0xa0060708,0x1,0x0,0x0} tx_q_wr_ptr=6
> 7782.321799: q_txdone: tx_q_rd_ptr=4 reqs_completed=2 outstanding_msgs=1
> 7782.321813: q_txdone: tx_q_rd_ptr=5 reqs_completed=1 outstanding_msgs=1
> 7782.321842: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0
>
> This one is interesting because we can see that the remote side first
> ACKed two of the three outstanding messages. Then a short time later
> it managed to ACK the last message.
>
> ...and just to show the queue being used quite often, here are two
> instances right in a row (it's not hard to see this happening):
>
> 8521.693559: send_q: {0xa0010708,0x1030001,0x0,0x0} tx_q_wr_ptr=5
> 8521.693649: send_q: {0xa0120708,0x10001,0x0,0x0} tx_q_wr_ptr=6
> 8521.693690: q_txdone: tx_q_rd_ptr=5 reqs_completed=2 outstanding_msgs=0
> 8521.693701: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0
> 8521.693920: send_q: {0xa0030708,0x1,0x0,0x0} tx_q_wr_ptr=7
> 8521.693956: q_txdone: tx_q_rd_ptr=7 reqs_completed=1 outstanding_msgs=0
> 8521.694171: send_q: {0xa0040705,0x2,0x14,0x1} tx_q_wr_ptr=0
> 8521.694194: send_q: {0xa0150708,0x1000001,0x0,0x0} tx_q_wr_ptr=1
> 8521.694214: q_txdone: tx_q_rd_ptr=0 reqs_completed=2 outstanding_msgs=0
> 8521.694226: q_txdone: tx_q_rd_ptr=1 reqs_completed=1 outstanding_msgs=0
>
> ...and in case you want to see an even bigger use of the queue, here
> are 6 queued up at once:
>
> 9223.248560: send_q: {0xa0060403,0x3050600,0x0,0x0} tx_q_wr_ptr=7
> 9223.248598: q_txdone: tx_q_rd_ptr=7 reqs_completed=1 outstanding_msgs=0
> 9223.248824: send_q: {0xa007000b,0x1,0x418,0x7} tx_q_wr_ptr=0
> 9223.248960: send_q: {0xa008000b,0x1,0x18,0x7} tx_q_wr_ptr=1
> 9223.249289: send_q: {0xa0090008,0xa1800,0x0,0x0} tx_q_wr_ptr=2
> 9223.249489: send_q: {0xa00a0008,0x1803,0x0,0x0} tx_q_wr_ptr=3
> 9223.249612: send_q: {0xa00b0008,0xa1900,0x0,0x0} tx_q_wr_ptr=4
> 9223.250046: send_q: {0xa01c0403,0x3050600,0xffffffff,0x0} tx_q_wr_ptr=5
> 9223.250232: q_txdone: tx_q_rd_ptr=0 reqs_completed=6 outstanding_msgs=0
> 9223.250236: q_txdone: tx_q_rd_ptr=1 reqs_completed=5 outstanding_msgs=0
> 9223.250237: q_txdone: tx_q_rd_ptr=2 reqs_completed=4 outstanding_msgs=0
> 9223.250239: q_txdone: tx_q_rd_ptr=3 reqs_completed=3 outstanding_msgs=0
> 9223.250240: q_txdone: tx_q_rd_ptr=4 reqs_completed=2 outstanding_msgs=0
> 9223.250241: q_txdone: tx_q_rd_ptr=5 reqs_completed=1 outstanding_msgs=0
> 9223.250406: send_q: {0xa00d0008,0x1903,0x0,0x0} tx_q_wr_ptr=6
> 9223.250437: q_txdone: tx_q_rd_ptr=6 reqs_completed=1 outstanding_msgs=0
>
>
> This looks pretty clearly worth it. Have I convinced you? Is there
> other data you'd like to see?
>
OK, let's work with the assumption that it is really needed.  Though I
would still explore the option of lazy/debounced disabling of clocks
and other resources controlled by the remote host.

>
> > If you do have some real numbers and think it is worth it on your
> > platform, then maybe expose each doorbell/shm-slot as a generic
> > channel. clk_mailbox will request a generic channel, do the request
> > and free it. The same effect but without inventing a new api. I can
> > share a draft if you want, but I suggest let's not make things
> > complicated without proven benefit.
>
> FWIW, the downstream code in Pixel does what I think you're
> suggesting. I can confidently say that, while it doesn't require
> changes to the mailbox core, it is much more convoluted and
> complicated. It also bleeds into the device-tree representation, which
> doesn't feel great.
>
> Just to be concrete, I'll document how the downstream driver works. If
> this isn't what you were thinking, please correct me.
>
> Back to our simplified "clk_mailbox" driver. We'll say that our
> "clk_mailbox" driver talks over a single mailbox to the remote
> processor. Let's say each message is 4 words big. The message space is
> 32-words big. 32 / 4 = 8 which means this space is divided into 8
> queue slots. Downstream represents each of these queue slots as a
> generic channel. That means that, in the device tree, our
> "clk_mailbox" driver looks looks like this:
>
> mboxes = <&cpm_tx_mba 0>,
>          <&cpm_tx_mba 1>,
>          <&cpm_tx_mba 2>,
>          <&cpm_tx_mba 3>,
>          <&cpm_tx_mba 4>,
>          <&cpm_tx_mba 5>,
>          <&cpm_tx_mba 6>,
>          <&cpm_tx_mba 7>;
>
> Then the "clk_mailbox" driver is in charge of rotating through each of
> the channels. First it writes to channel 0, then it writes to channel
> 1, etc. This works with no changes to the core, but...
>
I was thinking something like   mboxes = <&cpm_tx_mba>
clk_mailbox simply asks for some channel, gets allocated the next free
slot. The mailbox controller driver keeps track of the channel-slot
map locally and is responsible for managing contiguous slots and
completing the tx upon receiving ack.

> I'm also a little confused about the resistance. I don't feel like the
> mailbox core change is that complicated. The diffstat shows 58
> insertions and 17 deletions. 14 of those added lines are comments.
> While we certainly don't want to add useless APIs, to me this truly
> seems like the correct way to add the functionality. It also doesn't
> seem absurd to me that some future mailbox controller out there will
> also support queuing like this.
>
It is not the diff stat but about inserting a flag in the api to
introduce special case behavior. It is like adding one person to the
party introduces N-1 handshakes - the has_queue flag doesn't play well
with other configurations and may allow future platforms to abuse
has_queue to implement hacks.

Regards
Jassi