Linux coresight arm64 : Incorrect data in cstrace.bin
James Clark
james.clark at arm.com
Fri Jul 7 01:49:43 PDT 2023
On 06/07/2023 13:47, Pandey, Radhey Shyam wrote:
>> -----Original Message-----
>> From: Pandey, Radhey Shyam
>> Sent: Thursday, July 6, 2023 5:40 PM
>> To: James Clark <james.clark at arm.com>
>> Cc: coresight at lists.linaro.org; linux-arm-kernel at lists.infradead.org; Suzuki K
>> Poulose <suzuki.poulose at arm.com>; mathieu.poirier at linaro.org;
>> mike.leach at linaro.org; leo.yan at linaro.org;
>> alexander.shishkin at linux.intel.com; Sarangi, Anirudha
>> <anirudha.sarangi at amd.com>
>> Subject: RE: Linux coresight arm64 : Incorrect data in cstrace.bin
>>
>>> -----Original Message-----
>>> From: James Clark <james.clark at arm.com>
>>> Sent: Wednesday, July 5, 2023 5:44 PM
>>> To: Pandey, Radhey Shyam <radhey.shyam.pandey at amd.com>
>>> Cc: coresight at lists.linaro.org; linux-arm-kernel at lists.infradead.org;
>>> Suzuki K Poulose <suzuki.poulose at arm.com>; mathieu.poirier at linaro.org;
>>> mike.leach at linaro.org; leo.yan at linaro.org;
>>> alexander.shishkin at linux.intel.com; Sarangi, Anirudha
>>> <anirudha.sarangi at amd.com>
>>> Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
>>>
>>>
>>>
>>> On 05/07/2023 12:48, Pandey, Radhey Shyam wrote:
>>>>> -----Original Message-----
>>>>> From: James Clark <james.clark at arm.com>
>>>>> Sent: Wednesday, July 5, 2023 4:29 PM
>>>>> To: Pandey, Radhey Shyam <radhey.shyam.pandey at amd.com>
>>>>> Cc: coresight at lists.linaro.org;
>>>>> linux-arm-kernel at lists.infradead.org;
>>>>> Suzuki K Poulose <suzuki.poulose at arm.com>;
>>>>> mathieu.poirier at linaro.org; mike.leach at linaro.org;
>>>>> leo.yan at linaro.org; alexander.shishkin at linux.intel.com
>>>>> Subject: Re: Linux coresight arm64 : Incorrect data in cstrace.bin
>>>>>
>>>>>
>>>>>
>>>>> On 05/07/2023 10:56, Pandey, Radhey Shyam wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: Suzuki K Poulose <suzuki.poulose at arm.com>
>>>>>>> Sent: Wednesday, July 5, 2023 1:57 PM
>>>>>>> To: Pandey, Radhey Shyam <radhey.shyam.pandey at amd.com>;
>>>>>>> mathieu.poirier at linaro.org; mike.leach at linaro.org;
>>>>>>> leo.yan at linaro.org; alexander.shishkin at linux.intel.com
>>>>>>> Cc: coresight at lists.linaro.org;
>>>>>>> linux-arm-kernel at lists.infradead.org
>>>>>>> Subject: Re: Linux coresight arm64 : Incorrect data in
>>>>>>> cstrace.bin
>>>>>>>
>>>>>>> Hi Radhe Shyam
>>>>>>>
>>>>>>> On 05/07/2023 05:38, Pandey, Radhey Shyam wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am using linux 6.1 kernel coresight framework to capture ETM
>> trace.
>>>>>>>> Enabled coresight driver and added coresight component device
>>>>>>>> node in DTS. With that, we could probe all coresight components.
>>>>>>>>
>>>>>>>> But when we do capture using sysfs and then read the trace.bin
>>>>>>>> using ptm2human we see invalid trace data. Trace content
>>>>>>>> changes on each capture. Any pointers to help narrow down the
>> issue?
>>>>>>>>
>>>>>>>> Tried: sysfs capture and perf (with OpenCSD enabled)
>>>>>>>>
>>>>>>>> Development board: VCK190 :
>>>>>>>> https://docs.xilinx.com/r/en-US/ug1366-vck190-eval-bd
>>>>>>>> https://www.xilinx.com/support/documents/architecture-
>>>>>>> manuals/am011-ve
>>>>>>>> rsal-acap-trm.pdf
>>>>>>>>
>>>>>>>> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i amba
>>>>>>>> [ 0.301054] Serial: AMBA PL011 UART driver
>>>>>>>> [ 0.306226] amba f0c20000.funnel: Fixing up cyclic dependency
>> with
>>>>>>> f0d70000.etm
>>>>>>>> [ 0.313624] amba f0c20000.funnel: Fixing up cyclic dependency
>> with
>>>>>>> f0d30000.etm
>>>>>>>> [ 0.321110] amba f0c30000.etf: Fixing up cyclic dependency with
>>>>>>> f0c20000.funnel
>>>>>>>> [ 1.470247] Serial: AMBA driver
>>>>>>>> xilinx-vck190-20231:/home/petalinux# dmesg | grep -i coresight
>>>>>>>> [ 1.865006] cs_system_cfg: CoreSight Configuration manager
>>> initialised
>>>>>>>> [ 1.876858] coresight etm0: CPU0: etm v4.0 initialized
>>>>>>>> [ 1.885666] coresight etm1: CPU1: etm v4.0 initialized
>>>>>>>> [ 1.894435] coresight-cpu-debug f0d00000.debug1: Coresight
>> debug-
>>>>> CPU0
>>>>>>> initialized
>>>>>>>> [ 1.902072] coresight-cpu-debug f0d40000.debug1: Coresight
>> debug-
>>>>> CPU1
>>>>>>> initialized
>>>>>>>>
>>>>>>>> xilinx-vck190-20231:/home/petalinux# ls
>> /sys/bus/coresight/devices/
>>>>>>>> etm0 etm1 funnel0 tmc_etf0
>>>>>>>>
>>>>>>>> cd /sys/bus/coresight/devices/ echo 1 > tmc_etf0/enable_sink
>>>>>>>> echo 1 > etm0/enable_source echo 0 > etm0/enable_source echo 0 >
>>>>>>>> tmc_etf0/enable_sink cd /root/ dd if=/dev/tmc_etf0
>>>>>>>> of=cstrace_28Jun.bin
>>>>>>>>
>>>>>>>> ./ptm2human/ptm2human -e -i cstrace.bin -d
>>>>>>>
>>>>>>> Please note that ptm2human is for PTM trace decoding and
>>>>>>> ETMv4 uses a different format and thus is not compatible.
>>>>>>>
>>>>>>>>
>>>>>>>> I also tried OpenCSD integration with PERF.
>>>>>>>
>>>>>>> Have you made sure the perf is "linked" to the opencsd ?
>>>>>>
>>>>>> Earlier I statically linked but after adding CORESIGHT=1.
>>>>>>
>>>>>>
>>>>>> linux-xlnx$ make ARCH=arm64 NO_LIBELF=1 NO_JVMTI=1 VF=1
>>>>> CORESIGHT=1 -C
>>>>>> tools/perf/
>>>>>> make: Entering directory 'linux-xlnx/tools/perf'
>>>>>> BUILD: Doing 'make -j24' parallel build
>>>>>>
>>>>>> Makefile.config:520: *** Error: No libopencsd library found or the
>>>>>> version
>>>>> is not up-to-date.
>>>>>> Please install recent libopencsd to build with CORESIGHT=1. Stop.
>>>>>>
>>>>>>
>>>>>> echo $CSINCLUDES
>>>>>> /scratch/development/coresight/my-opencsd/decoder/include
>>>>>> radheys at xhdradheys41:/scratch/development/linux-xlnx$ ls
>>>>>> /scratch/development/coresight/my-opencsd/decoder/include
>>>>>> common i_dec interfaces mem_acc opencsd opencsd.h
>>>>>> pkt_printers
>>>>>>
>>>>>> echo $CSLIBS
>>>>>> /scratch/development/coresight/my-opencsd/decoder/lib/builddir
>>>>>> radheys at xhdradheys41:/scratch/development/linux-xlnx$ ls
>>>>> /scratch/development/coresight/my-opencsd/decoder/lib/builddir
>>>>>> libopencsd.a libopencsd_c_api.so libopencsd_c_api.so.1.4.0
>>>>> libopencsd.so.1
>>>>>> libopencsd_c_api.a libopencsd_c_api.so.1 libopencsd.so
>>>>> libopencsd.so.1.4.0
>>>>>>
>>>>>> Anything I am missing to fix this opencsd lib not found?
>>>>>
>>>>> There is a minimum version of 1.1.1 but it looks like you are using
>>>>> higher than that so that part should be fine.
>>>>>
>>>>> Personally I "make install" OpenCSD to the system path. I had a
>>>>> play around with $CSLIBS it and maybe there is also some stickyness
>>>>> to the feature detection so a make clean might help to make sure
>>>>> the errors you are seeing are real.
>>>>>
>>>>> Lastly you can print the output of why the feature test compilation
>>>>> failed which might help. I see some linking issues in there if I
>>>>> delete my system installed version of OpenCSD, so there might be a
>>>>> bug with just using CSINCLUDES and CSLIBS:
>>>>>
>>>>> $ cat tools/build/feature/test-libopencsd.make.output
>>>>>
>>>>> /usr/bin/ld: warning: libopencsd.so.1, needed by opencsd-
>>>>> local/decoder/lib/builddir//libopencsd_c_api.so, not found (try
>>>>> using -rpath or -rpath-link)
>>>>> /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so:
>>>>> undefined reference to `vtable for StmTrcPacket'
>>>>> /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so:
>>>>> undefined reference to `DecodeTree::getDecoderStats(unsigned char,
>>>>> _ocsd_decode_stats**)'
>>>>> /usr/bin/ld: opencsd-local/decoder/lib/builddir//libopencsd_c_api.so:
>>>>> undefined reference to
>>>>> `DecodeTree::addRawFramePrinter(RawFramePrinter**, unsigned int)'
>>>>>
>>>>> Can you try the make install and see if that works? If that works
>>>>> but the other way doesn't I can try looking into why its not and make a
>> fix.
>>>>
>>>> I tried make install but still it errors out. Looking at make.output
>>>> I am seeing multiple undefined references.
>>>>
>>>> NOTE I have to still set CSINCLUDES and CSLIBS as we are
>>>> cross-compiling Perf on a x86 host machine.
>>>
>>> This should still work if you make install them to the arch specific
>>> folder for example /usr/lib/aarch64-linux-gnu/. If do 'apt install
>> libopencsd-dev:arm64'
>>> I would get the arm ones in there.
>>>
>>> Then if I cross build like this I get openCSD linked:
>>>
>>> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C tools/perf
>>> CORESIGHT=1
>>>
>>> So for now you should be able to work around the issue in this way.
>>
>> I tried copying opencsd libraries to /usr/lib/aarch64-linux-gnu but still I see
>> undefined reference. Is it because I am using cross-compiler from server
>> (toolchain path set to $PATH variable)?
>>
>> /proj/<snip>/Vitis/2023.2/gnu/aarch64/lin/aarch64-linux/bin/aarch64-linux-
>> gnu-gcc
>> I assume CSLIBS and CSINCLUDES should be sufficient to cross compile perf
>> with coresight support?
>
> FYI, I tried with static linking and could proceed to perf compilation.
>
Static compilation works because the link line is slightly different (it
includes -lopencsd rather than just -lopencsd_c_api:
ifeq ($(findstring -static,${LDFLAGS}),-static)
OPENCSDLIBS += -lopencsd -lstdc++
I think that this is related to the issue and there have been some
recent changes around here. So yes I'll look into the fix for the non
static and $CSLIBS mode.
> make LDFLAGS=-static NO_LIBELF=1 NO_JVMTI=1 VF=1 CORESIGHT=1 -C tools/perf/
> <snip>
> ... libbfd-liberty-z: [ OFF ]
> ... libopencsd: [ on ]
> ... libunwind-x86: [ OFF ]
>
> But it ended with compilation failure. Seems like OpenCSD version mismatch.
> I will try to downgrade opencsd library and see if it fixes.
>
> util/cs-etm-decoder/cs-etm-decoder.c: In function 'cs_etm_decoder__gen_trace_elem_printer':
> util/cs-etm-decoder/cs-etm-decoder.c:567:9: error: enumeration value
> 'OCSD_GEN_TRC_ELEM_INSTRUMENTATION' not handled in switch [-Werror=switch-enum]
> 567 | switch (elem->elem_type) {
>
This should be fixed here [1]. Are you building the latest version of
Perf? Downgrading OpenCSD would work to fix this, but so would building
the latest version of Perf.
It's also possible that you are pointing to an old OpenCSD header but
new libraries. The OpenCSD version number is pulled out of
/usr/include/opencsd/ocsd_if_version.h so it's possible to get out of
sync with the .so file.
>>
>>>
>>>>
>>>> cat tools/build/feature/test-libopencsd.make.output <snip library
>>>> paths>
>>>>
>>>> aarch64-xilinx-linux-ld.real: warning: libopencsd.so.1, needed by
>>>> libopencsd_c_api.so, not found (try using -rpath or -rpath-link)
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `vtable for StmTrcPacket'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::getDecoderStats(unsigned char, _ocsd_decode_stats**)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::addRawFramePrinter(RawFramePrinter**, unsigned int)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::logMappedRanges()'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `OcsdLibDcdRegister::deregisterAllDecoders()'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `PtmTrcPacket::toString(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&) const'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::s_error_logger'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::addBinFileRegionMemAcc(_ocsd_file_mem_region const*,
>> int,
>>> _ocsd_mem_space_acc_t, std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> > const&)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `EtmV4ITrcPacket::~EtmV4ITrcPacket()'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `OcsdLibDcdRegister::getDecoderRegister()'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `typeinfo for TraceComponent'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `EtmV3TrcPacket::~EtmV3TrcPacket()'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `OcsdTraceElement::toString(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&) const'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::addGenElemPrinter(TrcGenericElementPrinter**)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::DestroyDecodeTree(DecodeTree*)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `ocsdError::getErrorString[abi:cxx11](ocsdError const&)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::setGenTraceElemOutI(ITrcGenElemIn*)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `ocsdError::ocsdError(ocsdError const*)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `ocsdError::ocsdError(_ocsd_err_severity_t, _ocsd_err_t, unsigned int,
>>> unsigned char, std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> > const&)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `EtmV4ITrcPacket::toString(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&) const'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `DecodeTree::createDecoder(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> > const&, int, CSConfig const*)'
>>>> aarch64-xilinx-linux-ld.real: libopencsd_c_api.so: undefined
>>>> reference to
>>> `StmTrcPacket::StmTrcPacket()'
>>>>
>>>
>>> Yes this looks like what I see as well. There is probably some issue
>>> with CSLIBS not being propagated to all parts of the build. I would
>>> recommend using the system library path for now and I will look into a fix
>> next week.
>>> Unless you are happy to look into it and send a fix that would be great too.
>>>
>>> James
>>>
>>>> Thanks,
>>>> Radhey
>>>>
>>>>>
>>>>> Thanks
>>>>> James
>>>>
More information about the linux-arm-kernel
mailing list