[PATCH V2] coresight: tmc: Make etr buffer mode user configurable from sysfs

[Suzuki K Poulose]

Hi Steve

On 04/12/2023 21:41, Steve Clevenger wrote:
> 
> Hi Suzuki,
> 
> On 9/7/2023 2:31 PM, Steve Clevenger wrote:
>>
>> Hi Suzuki,
>>
>> On 9/1/2023 5:22 AM, Suzuki K Poulose wrote:
>>> Hi Steve
>>>
>>> On 30/08/2023 17:04, Suzuki K Poulose wrote:
>>>> Cc: Sudeep
>>>>
>>>> Hi Steve
>>>>
>>>> On 28/08/2023 17:35, Steve Clevenger wrote:
>>>>>
>>>>> Hi Suzuki,
>>>>>
>>>>> On 8/27/2023 2:35 PM, Suzuki K Poulose wrote:
>>>>>> Hi Steve
>>>>>>
>>>>>> On 26/08/2023 01:14, Steve Clevenger wrote:
>>>>>>>
>>>>>>> Unfortunately, I tested with the original patch not [PATCH V2]. I've
>>>>>>> remedied this. My results below:
>>>>>>>
>>>>>>> [root at sut01sys-b212 linux]# cat
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_modes_available
>>>>>>> auto flat catu
>>>>>>> [root at sut01sys-b212 linux]# cat
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>>> auto
>>>>>>> [root at sut01sys-b212 linux]# echo "catu" >
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>>> [root at sut01sys-b212 linux]# cat
>>>>>>> /sys/devices/system/cpu/cpu123/ARMHC501\:23/tmc_etr35/buf_mode_preferred
>>>>>>> catu
>>>>>>>
>>>>>>> As with the V1 patch, auto defaults to catu.
>>>>>>>
>>>>>>> I expected to see tmc-sg (former default) as an available mode, but do
>>>>>>> not. As I recall, the buffer mode defaulted to ETR scatter-gather
>>>>>>> prior
>>>>>>> to this patch. Must this capability now be explicitly advertised? I've
>>>>>>> seen this done as "arm,scatter-gather" in device trees, but not
>>>>>>> used by
>>>>>>> Ampere. Perhaps someone can enlighten me.
>>>>>>
>>>>>> Yes, you must add that property to the TMC-ETR node (for both DT and
>>>>>> ACPI). In the past, almost all of the TMC-ETRs (except Juno board)
>>>>>> locked up the system while using the SG mode (due to the interconnect
>>>>>> issues, something to do with the transaction). Thus, we decided to
>>>>>> add a property explicitly enabling this for a given platform.
>>>>>>
>>>>>> When you mentioned, it was using TMC-ETR SG mode, how did you verify
>>>>>> this ? Please be aware that the table allocation code etc are shared
>>>>>> by both TMC-SG and CATU.
>>>>>>
>>>>>
>>>>> You might recall how this started. I had no way to test the CATU due to
>>>>> the order the ETR modes defaulted (Flat, ETR-SG, CATU). For test
>>>>> purposes, I programmatically swapped the ETR-SG/CATU order and could
>>>>> then verify CATU operation by the driver calling into CATU code. This
>>>>
>>>> So, were you using the DT based boot for the above runs ?
>>>>
>>>>> suggests Flat mode was bypassed, and the driver defaulted to ETR-SG
>>>>> prior to this hack. This didn't offer the user any control, hence my
>>>>> feature request. Note that most of the early Ampere self-hosted trace
>>>>> collection used ETR-SG. Now I can't select it.
>>>>
>>>>
>>>>>
>>>>> How is this property described in the ACPI? The "ACPI for CoreSight™ 1.1
>>>>> Platform Design Document" (DEN0067) doesn't describe this.
>>>>
>>>> This is not specified in the ACPI platform design document. I can get
>>>> it fixed. Ideally we need a property describing that the scatter-gather
>>>> mode is safe to use.
>>>
>>> Looks like this is not straight forward copying of DT property. We are
>>> investigating this on our side and will get back to you.
> 
> I noticed this work is queued for 6.7 (coresight-next-6.7). Do you have
> an update to the ACPI platform design document Ampere can use to base an
> update so the scatter-gather mode can be used?

I have chased this with the specification team, and we should have
something published soon. Apologies for the delay.

Suzuki

> 
> Thanks,
> Steve
> 
>>>
>>
>> The intent behind my request was to have a way to SysFS configure the
>> (available) ETR mode. Unless there's a change to the ACPI, the CATU is
>> the only SG option for Ampere in the near term.
>>
>> Thanks,
>> Steve
>>
>>
>>> Suzuki
>>>
>>>
>>>>
>>>> DT uses "arm,scatter-gather" property [0] and this is what we now expect
>>>> in the ACPI based systems too.
>>>>
>>>> https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/arm/arm,coresight-tmc.yaml#:~:text=71-,72,-73
>>>>
>>>> Does it sound fine ?
>>>>
>>>> Suzuki
>>>>
>>>>>
>>>>> Thanks,
>>>>> Steve
>>>>>
>>>>>
>>>>>> Kind regards
>>>>>> Suzuki
>>>>>>
>>>>>>>
>>>>>>> Steve C.
>>>>>>>
>>>>>>> On 8/23/2023 4:10 PM, Steve Clevenger wrote:
>>>>>>>>
>>>>>>>> Here's some quick feedback. My system shows two modes available; auto
>>>>>>>> catu
>>>>>>>>
>>>>>>>> etr_buf_mode_current is writable. I expected to see tmc-sg (former
>>>>>>>> default) listed in etr_buf_modes_available but it doesn't show up.
>>>>>>>>
>>>>>>>> Note that both the auto and catu etr_buf_mode_current settings
>>>>>>>> default
>>>>>>>> to catu. My understanding is auto should revert to the default
>>>>>>>> behavior.
>>>>>>>> On my system the default was tmc-sg.
>>>>>>>>
>>>>>>>> More later.
>>>>>>>>
>>>>>>>> [root at sut01sys-b212 kernel]# cat
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_modes_available
>>>>>>>>
>>>>>>>> auto catu
>>>>>>>> [root at sut01sys-b212 kernel]# cat
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>>> catu
>>>>>>>> [root at sut01sys-b212 kernel]# echo "catu" >
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>>> [root at sut01sys-b212 kernel]# cat
>>>>>>>> /sys/devices/system/cpu/cpu20/ARMHC501\:60/tmc_etr96/etr_buf_mode_current
>>>>>>>> catu
>>>>>>>>
>>>>>>>> Steve C.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/21/2023 12:40 PM, Steve Clevenger wrote:
>>>>>>>>>
>>>>>>>>> Hi Suzuki,
>>>>>>>>>
>>>>>>>>> I may be able to test it this week. You've already pointed me at the
>>>>>>>>> patch thread(s). The main holdup is I need to merge the 6.6 pending
>>>>>>>>> platform work in order to use the Ampere ACPI. I couldn't get these
>>>>>>>>> patches to apply directly to 6.4 last I tried.
>>>>>>>>>
>>>>>>>>> Steve C.
>>>>>>>>>
>>>>>>>>> On 8/18/2023 2:39 AM, Suzuki K Poulose wrote:
>>>>>>>>>> Cc: Steve
>>>>>>>>>>
>>>>>>>>>> Steve,
>>>>>>>>>>
>>>>>>>>>> Are you able to test this with CATU ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 18/08/2023 09:21, Anshuman Khandual wrote:
>>>>>>>>>>> Currently TMC-ETR automatically selects the buffer mode from all
>>>>>>>>>>> available
>>>>>>>>>>> methods in the following sequentially fallback manner - also in
>>>>>>>>>>> that
>>>>>>>>>>> order.
>>>>>>>>>>>
>>>>>>>>>>> 1. FLAT mode with or without IOMMU
>>>>>>>>>>> 2. TMC-ETR-SG (scatter gather) mode when available
>>>>>>>>>>> 3. CATU mode when available
>>>>>>>>>>>
>>>>>>>>>>> But this order might not be ideal for all situations. For
>>>>>>>>>>> example if
>>>>>>>>>>> there
>>>>>>>>>>> is a CATU connected to ETR, it may be better to use TMC-ETR
>>>>>>>>>>> scatter
>>>>>>>>>>> gather
>>>>>>>>>>> method, rather than CATU. But hard coding such order changes will
>>>>>>>>>>> prevent
>>>>>>>>>>> us from testing or using a particular mode. This change provides
>>>>>>>>>>> following
>>>>>>>>>>> new sysfs tunables for the user to control TMC-ETR buffer mode
>>>>>>>>>>> explicitly,
>>>>>>>>>>> if required. This adds following new sysfs files for buffer mode
>>>>>>>>>>> selection
>>>>>>>>>>> purpose explicitly in the user space.
>>>>>>>>>>>
>>>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_modes_available
>>>>>>>>>>> /sys/bus/coresight/devices/tmc_etr<N>/buf_mode_preferred
>>>>>>>>>>>
>>>>>>>>>>> $ cat buf_modes_available
>>>>>>>>>>> auto flat tmc-sg catu    ------------------> Supported TMC-ETR
>>>>>>>>>>> buffer
>>>>>>>>>>> modes
>>>>>>>>>>>
>>>>>>>>>>> $ echo catu > buf_mode_preferred   -------> Explicit buffer mode
>>>>>>>>>>> request
>>>>>>>>>>>
>>>>>>>>>>> But explicit user request has to be within supported ETR buffer
>>>>>>>>>>> modes
>>>>>>>>>>> only.
>>>>>>>>>>> These sysfs interface files are exclussive to ETR, and hence these
>>>>>>>>>>> are
>>>>>>>>>>> not
>>>>>>>>>>> available for other TMC devices such as ETB or ETF etc.
>>>>>>>>>>>
>>>>>>>>>>> A new auto' mode (i.e ETR_MODE_AUTO) has been added to help
>>>>>>>>>>> fallback
>>>>>>>>>>> to the
>>>>>>>>>>> existing default behaviour, when user provided preferred buffer
>>>>>>>>>>> mode
>>>>>>>>>>> fails.
>>>>>>>>>>> ETR_MODE_FLAT and ETR_MODE_AUTO are always available as preferred
>>>>>>>>>>> modes.
>>>>>>>>>>>
>>>>>>>>>>> Cc: Suzuki K Poulose <suzuki.poulose at arm.com>
>>>>>>>>>>> Cc: Mike Leach <mike.leach at linaro.org>
>>>>>>>>>>> Cc: James Clark <james.clark at arm.com>
>>>>>>>>>>> Cc: Leo Yan <leo.yan at linaro.org>
>>>>>>>>>>> Cc: Alexander Shishkin <alexander.shishkin at linux.intel.com>
>>>>>>>>>>> Cc: coresight at lists.linaro.org
>>>>>>>>>>> Cc: linux-arm-kernel at lists.infradead.org
>>>>>>>>>>> Cc: linux-kernel at vger.kernel.org
>>>>>>>>>>> Signed-off-by: Anshuman Khandual <anshuman.khandual at arm.com>
>>>>>>>>>>> ---
>>>>>>>>>>> This applies on v6.5-rc6
>>>>>>>>>>>
>>>>>>>>>>> Changes in V2:
>>>>>>>>>>>
>>>>>>>>>>> - Renamed sysfs file etr_buf_modes_available as
>>>>>>>>>>> buf_modes_available
>>>>>>>>>>> - Renamed sysfs file buf_mode_current as buf_mode_preferred
>>>>>>>>>>> - Renamed etr_supports_flat_mode() as etr_can_use_flat_mode()
>>>>>>>>>>> - Renamed coresight_tmc_groups[] as coresight_etf_groups[]
>>>>>>>>>>> - Reused coresight_tmc_group[] for trigger_cntr and buffer_size
>>>>>>>>>>> - Fallback trying ETR_MODE_AUTO when user preferred mode fails
>>>>>>>>>>> - Moved ETR sysfs details into coresight-tmc-etr.c
>>>>>>>>>>> - Dropped etr_can_use_flat_mode() check while offering
>>>>>>>>>>> ETR_MODE_FLAT
>>>>>>>>>>> in sysfs
>>>>>>>>>>> - Moved struct etr_buf_hw inside coresight-tmc-etr.c
>>>>>>>>>>> - Moved get_etr_buf_hw() and etr_can_use_flat_mode() inside
>>>>>>>>>>> coresight-tmc-etr.c
>>>>>>>>>>> - Updated month in
>>>>>>>>>>> Documentation/ABI/testing/sysfs-bus-coresight-devices-tmc
>>>>>>>>>>>
>>>>>>>>>>> Changes in V1:
>>>>>>>>>>>
>>>>>>>>>>> https://lore.kernel.org/all/20230728084837.276551-1-anshuman.khandual@arm.com/
>>>>>>>>>>>
>>>>>>>>>>>      .../testing/sysfs-bus-coresight-devices-tmc   |  16 +++
>>>>>>>>>>>      .../hwtracing/coresight/coresight-tmc-core.c  |  15 ++-
>>>>>>>>>>>      .../hwtracing/coresight/coresight-tmc-etr.c   | 111
>>>>>>>>>>> ++++++++++++++++--
>>>>>>>>>>>      drivers/hwtracing/coresight/coresight-tmc.h   |   3 +
>>>>>>>>>>>      4 files changed, 131 insertions(+), 14 deletions(-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Looks good to me.
>>>>>>>>>>
>>>>>>>>>> Suzuki
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>
>>>