[PATCH 0/1] Fix for riscv vmcore issue

Tue Jul 15 23:58:05 PDT 2025

Hi Pnina,

On 7/14/25 14:00, Pnina Feder wrote:
>>>> Hi Pnina,
>>>>> Pnina!
>>>>>
>>>>> Pnina Feder <pnina.feder at mobileye.com> writes:
>>>>>
>>>>>> We are creating a vmcore using kexec on a Linux 6.15 RISC-V system
>>>>>> and analyzing it with the crash tool on the host. This workflow
>>>>>> used to work on Linux 6.14 but is now broken in 6.15.
>>>>> Thanks for reporting this!
>>>>>
>>>>>> The issue is caused by a change in the kernel:
>>>>>> In Linux 6.15, certain memblock sections are now marked as Reserved
>>>>>> in /proc/iomem. The kexec tool excludes all Reserved regions when
>>>>>> generating the vmcore, so these sections are missing from the dump.
>>>>>> How are you collecting the /proc/vmcore file? A full set of commands would be helpful.
>>>> We’ve defined in our system that when a process crashes, we call panic().
>>>> To handle crash recovery, we're using kexec with the following command:
>>>> kexec -p /Image --initrd=/rootfs.cpio --append "console=${con} earlycon=${earlycon} no4lvl"
>>>>
>>>> To simulate crash, we trigger it using:
>>>> sleep 100 & kill -6 $!
>>>>
>>>> This boots into the crash kernel (kdump), where we then copy the /proc/vmcore file back to the host for analysis.
>>>>
>>>>>> However, the kernel still uses addresses in these regions—for
>>>>>> example, for IRQ pointers. Since the crash tool needs access to
>>>>>> these memory areas to function correctly, their exclusion breaks the analysis.
>>>>> Wdym with "IRQ pointers"? Also, what version (sha1) of crash are you using?
>>>>>
>>>>> We are currently using crash-utility version 9.0.0 (master).
>>>>>   From the crash analysis logs, we observed errors like:
>>>>>
>>>> "......
>>>> IRQ stack pointer[0] is  ffffffd6fbdcc068
>>>>> crash: read error: kernel virtual address: ffffffd6fbdcc068  type: "IRQ stack pointer"
>>>> .....
>>>>
>>>>> <read_kdump: addr: ffffffff80edf1cc paddr: 8010df1cc cnt: 4>
>>>> <readmem: ffffffd6fbdd6880, KVADDR, "runqueues entry (per_cpu)",
>>>> 3456, (FOE), 55acf03963e0>
>>>>> read_kdump: addr: ffffffd6fbdd6880 paddr: 8fbdd6880 cnt: 1920<
>>>> crash: read error: kernel virtual address: ffffffd6fbdd6880  type: "runqueues entry (per_cpu)"
>>>
>>> I can't reproduce this issue on qemu, booting with sv39. I'm using the latest kexec-tools (which recently merged riscv .support), crash 9.0.0 and kernel 6.16.0-rc4. Note that I'm using crash in qemu.
>>>
>>> Are you able to reproduce this on qemu too?
>> Yes, I am using qemu too on main and crash kernel, with latest kexec-tools, crash 9.0.0 and kernel 6.15
>>
>>
>>> Maybe that's related to the config, can you share your config?
>> this is my dev_config
>>
>> CONFIG_SYSVIPC=y
>> CONFIG_POSIX_MQUEUE=y
>> CONFIG_AUDIT=y
>> CONFIG_NO_HZ_IDLE=y
>> CONFIG_HIGH_RES_TIMERS=y
>> CONFIG_BPF_SYSCALL=y
>> CONFIG_PREEMPT_RT=y
>> CONFIG_TASKSTATS=y
>> CONFIG_TASK_DELAY_ACCT=y
>> CONFIG_PSI=y
>> CONFIG_IKCONFIG=y
>> CONFIG_IKCONFIG_PROC=y
>> CONFIG_CGROUPS=y
>> CONFIG_MEMCG=y
>> CONFIG_CGROUP_SCHED=y
>> CONFIG_CFS_BANDWIDTH=y
>> CONFIG_RT_GROUP_SCHED=y
>> CONFIG_CGROUP_PIDS=y
>> CONFIG_CGROUP_FREEZER=y
>> CONFIG_CGROUP_HUGETLB=y
>> CONFIG_CPUSETS=y
>> CONFIG_CGROUP_DEVICE=y
>> CONFIG_CGROUP_CPUACCT=y
>> CONFIG_CGROUP_PERF=y
>> CONFIG_CGROUP_BPF=y
>> CONFIG_NAMESPACES=y
>> CONFIG_USER_NS=y
>> CONFIG_CHECKPOINT_RESTORE=y
>> CONFIG_BLK_DEV_INITRD=y
>> CONFIG_EXPERT=y
>> CONFIG_PROFILING=y
>> CONFIG_KEXEC=y
>> CONFIG_ARCH_VIRT=y
>> CONFIG_NONPORTABLE=y
>> CONFIG_SMP=y
>> CONFIG_NR_CPUS=32
>> CONFIG_HZ_1000=y
>> CONFIG_CPU_IDLE=y
>> CONFIG_MODULES=y
>> CONFIG_MODULE_UNLOAD=y
>> CONFIG_IOSCHED_BFQ=y
>> CONFIG_PAGE_REPORTING=y
>> CONFIG_PERCPU_STATS=y
>> CONFIG_NET=y
>> CONFIG_PACKET=y
>> CONFIG_UNIX=y
>> CONFIG_XFRM_USER=m
>> CONFIG_INET=y
>> CONFIG_IP_MULTICAST=y
>> CONFIG_IP_ADVANCED_ROUTER=y
>> CONFIG_INET_ESP=m
>> CONFIG_NETWORK_SECMARK=y
>> CONFIG_NETFILTER=y
>> CONFIG_IP_NF_IPTABLES=y
>> CONFIG_IP_NF_FILTER=y
>> CONFIG_BRIDGE=m
>> CONFIG_BRIDGE_VLAN_FILTERING=y
>> CONFIG_VLAN_8021Q=m
>> CONFIG_NET_SCHED=y
>> CONFIG_NET_CLS_CGROUP=m
>> CONFIG_NETLINK_DIAG=y
>> CONFIG_NET_L3_MASTER_DEV=y
>> CONFIG_CGROUP_NET_PRIO=y
>> CONFIG_FAILOVER=y
>> CONFIG_DEVTMPFS=y
>> CONFIG_DEVTMPFS_MOUNT=y
>> CONFIG_MTD=y
>> CONFIG_MTD_BLOCK=y
>> CONFIG_MTD_CFI=y
>> CONFIG_MTD_CFI_INTELEXT=y
>> CONFIG_MTD_PHYSMAP=y
>> CONFIG_MTD_PHYSMAP_OF=y
>> CONFIG_BLK_DEV_LOOP=y
>> CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
>> CONFIG_VIRTIO_BLK=y
>> CONFIG_SCSI=y
>> CONFIG_BLK_DEV_SD=y
>> CONFIG_SCSI_VIRTIO=y
>> CONFIG_MD=y
>> CONFIG_BLK_DEV_DM=y
>> CONFIG_NETDEVICES=y
>> CONFIG_MACB=y
>> CONFIG_PCS_XPCS=m
>> CONFIG_SERIO_LIBPS2=y
>> CONFIG_VT_HW_CONSOLE_BINDING=y
>> CONFIG_LEGACY_PTY_COUNT=16
>> CONFIG_SERIAL_8250=y
>> CONFIG_SERIAL_8250_CONSOLE=y
>> CONFIG_SERIAL_OF_PLATFORM=y
>> CONFIG_SERIAL_EARLYCON_RISCV_SBI=y
>> CONFIG_VIRTIO_CONSOLE=y
>> CONFIG_HW_RANDOM=y
>> CONFIG_HW_RANDOM_VIRTIO=y
>> CONFIG_I2C=y
>> CONFIG_I2C_DESIGNWARE_CORE=y
>> CONFIG_SPI=y
>> CONFIG_PINCTRL=y
>> CONFIG_PINCTRL_SINGLE=y
>> CONFIG_GPIOLIB=y
>> CONFIG_GPIO_SYSFS=y
>> CONFIG_GPIO_DWAPB=y
>> CONFIG_GPIO_SIFIVE=y
>> CONFIG_POWER_SUPPLY=y
>> CONFIG_WATCHDOG=y
>> CONFIG_WATCHDOG_CORE=y
>> CONFIG_REGULATOR=y
>> CONFIG_REGULATOR_FIXED_VOLTAGE=y
>> CONFIG_BACKLIGHT_CLASS_DEVICE=m
>> CONFIG_SCSI_UFSHCD=y
>> CONFIG_SCSI_UFSHCD_PLATFORM=y
>> CONFIG_SCSI_UFS_DWC_TC_PLATFORM=y
>> CONFIG_RTC_CLASS=y
>> CONFIG_RTC_DRV_M41T80=y
>> CONFIG_DMADEVICES=y
>> CONFIG_SYNC_FILE=y
>> CONFIG_COMMON_CLK_EYEQ=y
>> CONFIG_RPMSG_CHAR=y
>> CONFIG_RPMSG_CTRL=y
>> CONFIG_RPMSG_VIRTIO=y
>> CONFIG_RESET_CONTROLLER=y
>> CONFIG_RESET_SIMPLE=y
>> CONFIG_GENERIC_PHY=y
>> CONFIG_EXT4_FS=y
>> CONFIG_EXT4_FS_POSIX_ACL=y
>> CONFIG_EXT4_FS_SECURITY=y
>> CONFIG_MSDOS_FS=y
>> CONFIG_VFAT_FS=y
>> CONFIG_TMPFS=y
>> CONFIG_TMPFS_POSIX_ACL=y
>> CONFIG_HUGETLBFS=y
>> CONFIG_KEYS=y
>> CONFIG_SECURITY=y
>> CONFIG_SECURITYFS=y
>> CONFIG_SECURITY_NETWORK=y
>> CONFIG_SECURITY_PATH=y
>> CONFIG_CRYPTO_RSA=y
>> CONFIG_CRYPTO_ECB=y
>> CONFIG_CRYPTO_BLAKE2B=m
>> CONFIG_CRYPTO_XXHASH=m
>> CONFIG_CRYPTO_USER_API_HASH=y
>> CONFIG_CRC_CCITT=m
>> CONFIG_CRC_ITU_T=y
>> CONFIG_CRC7=y
>> CONFIG_LIBCRC32C=m
>> CONFIG_PRINTK_TIME=y
>> CONFIG_DYNAMIC_DEBUG=y
>> CONFIG_DEBUG_INFO_DWARF5=y
>> CONFIG_DEBUG_FS=y
>> CONFIG_DEBUG_PAGEALLOC=y
>> CONFIG_PTDUMP_DEBUGFS=y
>> CONFIG_SCHED_STACK_END_CHECK=y
>> CONFIG_DEBUG_VM=y
>> CONFIG_DEBUG_VM_PGFLAGS=y
>> CONFIG_DEBUG_MEMORY_INIT=y
>> CONFIG_DEBUG_PER_CPU_MAPS=y
>> CONFIG_SOFTLOCKUP_DETECTOR=y
>> CONFIG_WQ_WATCHDOG=y
>> CONFIG_DEBUG_RT_MUTEXES=y
>> CONFIG_DEBUG_SPINLOCK=y
>> CONFIG_DEBUG_ATOMIC_SLEEP=y
>> CONFIG_DEBUG_LIST=y
>> CONFIG_DEBUG_PLIST=y
>> CONFIG_DEBUG_SG=y
>> CONFIG_RCU_EQS_DEBUG=y
>> CONFIG_MEMTEST=y
>>
>>>> These failures occur consistently for addresses in the 0xffffffd000000000 region.
>>>
>>> FYI, this region is the direct mapping (see Documentation/arch/riscv/vm-layout.rst).
>>>
>>> Thanks,
>>>
>>> Alex
>>>
> Hi Alex!
>
> Do I have something to try or help to process this issue?
> maybe, can you give your Config and I will try it on my system?
> Any more information I can share?

So I'm able to reproduce your issue with your config, it only happens 
with kexec_load(), not kexec_file_load().

Your patch does not fix the problem for me, makedumpfile still fails. I 
spent quite some time looking for the code that parses the memory 
regions and exposes them as PT_LOAD segments in vmcore, but I did not 
find it, do you know where that happens for kexec_load()?

Thanks,

Alex

>
> Thanks a lot,
> Pnina
>
>>>> Upon inspection, we confirmed that the physical addresses corresponding to those virtual addresses are not present in the vmcore, as they fall under Reserved memory sections.
>>>> We tested a patch to kexec-tools that prevents exclusion of the Reserved-memblock section from the vmcore. With this patch, the issue no longer occurs, and crash analysis succeeds.
>>>> Note: I suspect the same issue exists on ARM64, as both the signal.c and kexec-tools implementations are similar.
>>>>
>>>>> Thanks!
>>>>> Björn
> _______________________________________________
> linux-riscv mailing list
> linux-riscv at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv