Question about the ipi_raise filter usage and output

[Valentin Schneider]

On 06/02/24 16:42, richard clark wrote:
> On Tue, Feb 6, 2024 at 12:05 AM Valentin Schneider <vschneid at redhat.com> wrote:
>>
>> The CPUS{} thingie only works with an event field that is either declared as a
>> cpumask (__cpumask) or a scalar. That's not the case for ipi_raise, the
>> target_cpus event field is saved as a "raw" bitmask.
>>
>> There /should/ have been a warning about the event filter though, but I
>> think it's not happening because I'm allowing more than just FILTER_CPUMASK
>> in parse_pred() to make it work for scalars. I'll go poke around some more.
>>
>> Generally for this sort of IPI investigation I'd recommend using the newer
>> trace_ipi_send_cpu() and trace_ipi_send_cpumask() (for which CPUS{}
>> filtering works).
>> If it's only the function call interrupts you're interesting in, have a
>> look at trace_csd_queue_cpu().
>
> This should be supported by newer version kernels like v6.5, but I am
> using v6.1 and this trace event has not been supported yet... so ipi
> is more suitable for me. ipi_entry and ipi_exit is ok, but seems the
> filter doesn't support a specific cpu, maybe we need to add this?
>>

You should have access to the generic fields which include the CPU from
which the event happens. Any of "CPU", "cpu" or "common_cpu" would match
this.

So if you're on a recent enough kernel (v6.6 or above AFAICT), you should
be able to do something like so:

  trace-cmd record -e 'ipi_raise' -f 'CPU & CPUS{7-42}' ./foo.sh

If you just want to match a single CPU, or are on an older kernel, this
should work as well:

  trace-cmd record -e 'ipi_raise' -f 'CPU == 42' ./foo.sh

For example on a QEMU x86 environment:

  # trace-cmd record -e 'call_function*' -f 'CPU & CPUS{3}' hackbench
  Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks)
  Each sender will pass 100 messages of 100 bytes
  Time: 0.396
  CPU0 data recorded at offset=0x738000
      0 bytes in size
  CPU1 data recorded at offset=0x738000
      0 bytes in size
  CPU2 data recorded at offset=0x738000
      0 bytes in size
  CPU3 data recorded at offset=0x738000
      4096 bytes in size

  # trace-cmd report
  CPU 0 is empty
  CPU 1 is empty
  CPU 2 is empty
  cpus=4
            <idle>-0     [003]    29.704387: call_function_single_entry: vector=251
            <idle>-0     [003]    29.704388: call_function_single_exit: vector=251
            <idle>-0     [003]    29.705950: call_function_single_entry: vector=251
            <idle>-0     [003]    29.705951: call_function_single_exit: vector=251
            <idle>-0     [003]    29.706462: call_function_single_entry: vector=251
            <idle>-0     [003]    29.706463: call_function_single_exit: vector=251
         hackbench-962   [003]    29.706501: call_function_single_entry: vector=251
         hackbench-962   [003]    29.706502: call_function_single_exit: vector=251
         hackbench-955   [003]    29.706521: call_function_single_entry: vector=251
         hackbench-955   [003]    29.706522: call_function_single_exit: vector=251
            <idle>-0     [003]    30.101812: call_function_single_entry: vector=251
            <idle>-0     [003]    30.101814: call_function_single_exit: vector=251
            <idle>-0     [003]    30.101897: call_function_single_entry: vector=251
            <idle>-0     [003]    30.101898: call_function_single_exit: vector=251
            <idle>-0     [003]    30.101985: call_function_single_entry: vector=251
            <idle>-0     [003]    30.101986: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102072: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102072: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102161: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102161: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102250: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102251: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102372: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102372: call_function_single_exit: vector=251


  CPU 0 is empty
  CPU 1 is empty
  CPU 2 is empty
  cpus=4
          <idle>-0     [003]  1067.718304: call_function_single_entry: vector=251
          <idle>-0     [003]  1067.718309: call_function_single_exit: vector=251

and that behaves the same as

  trace-cmd record -e 'call_function*' -f 'CPU == 3' hackbench