From takahiro.akashi at linaro.org  Sun Dec  3 18:57:50 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:50 +0900
Subject: [PATCH v7 00/11] arm64: kexec: add kexec_file_load() support
Message-ID: <20171204025801.12161-1-takahiro.akashi@linaro.org>

This is the seventh round of implementing kexec_file_load() support
on arm64.[1]
Most of the code is based on kexec-tools (along with some kernel code
from x86, which also came from kexec-tools).


This patch series enables us to
  * load the kernel, Image, via kexec_file_load() system call, and
  * optionally verify its signature at load time for trusted boot.

To load the kernel via kexec_file_load() system call, a small change
is also required on kexec-tools. See [2]. This enables '-s' option.
(Please use v7.2.1+ crash utility for v4.14+ kernel)

As we discussed a long time ago, users may not be allowed to specify
device-tree file of the 2nd kernel explicitly with kexec-tools, hence
re-using the blob of the first kernel.

Regarding a kernel image verification, a signature must be presented
along with the binary itself. A signature is basically a hash value
calculated against the whole binary data and encrypted by a key which
will be authenticated by the system's trusted certificate.
Any attempt to read and load a to-be-kexec-ed kernel image through
a system call will be checked and blocked if the binary's hash value
doesn't match its associated signature.

There are two methods available now:
1. implementing arch-specific verification hook of kexec_file_load()
2. utilizing IMA(Integrity Measurement Architecture)[3] appraisal framework

Before my v7, I believed that my patch only supports (1) but am now
confident that (2) comes free if IMA is enabled and properly configured.


(1) Arch-specific verification hook
If CONFIG_KEXEC_VERIFY_SIG is enabled, kexec_file_load() invokes an arch-
defined (and hence file-format-specific) hook function to check for the
validity of kernel binary.

On x86, a signature is embedded into a PE file (Microsoft's format) header
of binary. Since arm64's "Image" can also be seen as a PE file as far as
CONFIG_EFI is enabled, we adopt this format for kernel signing.  

As in the case of UEFI applications, we can create a signed kernel image:
    $ sbsign --key ${KEY} --cert ${CERT} Image

You may want to use certs/signing_key.pem, which is intended to be used
for module sigining (CONFIG_MODULE_SIG), as ${KEY} and ${CERT} for test
purpose.


(2) IMA appraisal-based
IMA was first introduced in linux in order to meet TCG (Trusted Computing
Group) requirement that all the sensitive files be *measured* before
reading/executing them to detect any untrusted changes/modification.
Then appraisal feature, which allows us to ensure the integrity of
files and even prevent them from reading/executing, was added later.

Meanwhile, kexec_file_load() has been merged since v3.17 and evolved to
enable IMA-appraisal type verification by the commit b804defe4297 ("kexec:
replace call to copy_file_from_fd() with kernel version").

In this scheme, a signature will be stored in a extended file attribute,
"security.ima" while a decryption key is hold in a dedicated keyring,
".ima" or "_ima".  All the necessary process of verification is confined
in a secure API, kernel_read_file_from_fd(), called by kexec_file_load().

    Please note that powerpc is one of the two architectures now
    supporting KEXEC_FILE, and that it wishes to exntend IMA,
    where a signature may be appended to "vmlinux" file[4], like module
    signing, instead of using an extended file attribute.

While IMA meant to be used with TPM (Trusted Platform Module) on secure
platform, IMA is still usable without TPM. Here is an example procedure
about how we can give it a try to run the feature using a self-signed
root ca for demo/test purposes:

 1) Generate needed keys and certificates, following "Generate trusted
    keys" section in README of ima-evm-utils[5].

 2) Build the kernel with the following kernel configurations, specifying
    "ima-local-ca.pem" for CONFIG_SYSTEM_TRUSTED_KEYS:
	CONFIG_EXT4_FS_SECURITY
	CONFIG_INTEGRITY_SIGNATURE
	CONFIG_INTEGRITY_ASYMMETRIC_KEYS
	CONFIG_INTEGRITY_TRUSTED_KEYRING
	CONFIG_IMA
	CONFIG_IMA_WRITE_POLICY
	CONFIG_IMA_READ_POLICY
	CONFIG_IMA_APPRAISE
	CONFIG_IMA_APPRAISE_BOOTPARAM
	CONFIG_SYSTEM_TRUSTED_KEYS
    Please note that CONFIG_KEXEC_VERIFY_SIG is not, actually should
    not be, enabled.

 3) Sign(label) a kernel image binary to be kexec-ed on target filesystem:
    $ evmctl ima_sign --key /path/to/private_key.pem /your/Image

 4) Add a command line parameter and boot the kernel:
    ima_appraise=enforce

 On live system,
 5) Set a security policy:
    $ mount -t securityfs none /sys/kernel/security
    $ echo "appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig" \
      > /sys/kernel/security/ima/policy

 6) Add a key for ima:
    $ keyctl padd asymmetric my_ima_key %:.ima < /path/to/x509_ima.der
    (or evmctl import /path/to/x509_ima.der <ima_keyring_id>)

 7) Then try kexec as normal.


Concerns(or future works):
* Even if the kernel is configured with CONFIG_RANDOMIZE_BASE, the 2nd
  kernel won't be placed at a randomized address. We will have to
  add some boot code similar to efi-stub to implement the randomization.
for approach (1),
* While big-endian kernel can support kernel signing, I'm not sure that
  Image can be recognized as in PE format because x86 standard only
  defines little-endian-based format.
* vmlinux support

  [1] http://git.linaro.org/people/takahiro.akashi/linux-aarch64.git
	branch:arm64/kexec_file
  [2] http://git.linaro.org/people/takahiro.akashi/kexec-tools.git
	branch:arm64/kexec_file
  [3] https://sourceforge.net/p/linux-ima/wiki/Home/
  [4] http://lkml.iu.edu//hypermail/linux/kernel/1707.0/03669.html
  [5] https://sourceforge.net/p/linux-ima/ima-evm-utils/ci/master/tree/


Changes in v7 (Dec 4, 2017)
* rebased to v4.15-rc2
* re-organize the patch set to separate KEXEC_FILE_VERIFY_SIG-related
  code from the others
* revamp factored-out code in kernel/kexec_file.c due to the changes
  in original x86 code
* redefine walk_sys_ram_res_rev() prototype due to change of callback
  type in the counterpart, walk_sys_ram_res()
* make KEXEC_FILE_IMAGE_FMT defaut on if KEXEC_FILE selected

Changes in v6 (Oct 24, 2017)
* fix a for-loop bug in _kexec_kernel_image_probe() per Julien

Changes in v5 (Oct 10, 2017)
* fix kbuild errors around patch #3
per Julien's comments,
* fix a bug in walk_system_ram_res_rev() with some cleanup
* modify fdt_setprop_range() to use vmalloc()
* modify fill_property() to use memset()

Changes in v4 (Oct 2, 2017)
* reinstate x86's arch_kexec_kernel_image_load()
* rename weak arch_kexec_kernel_xxx() to _kexec_kernel_xxx() for
  better re-use
* constify kexec_file_loaders[]

Changes in v3 (Sep 15, 2017)
* fix kbuild test error
* factor out arch_kexec_kernel_*() & arch_kimage_file_post_load_cleanup()
* remove CONFIG_CRASH_CORE guard from kexec_file.c
* add vmapped kernel region to vmcore for gdb backtracing
  (see prepare_elf64_headers())
* merge asm/kexec_file.h into asm/kexec.h
* and some cleanups

Changes in v2 (Sep 8, 2017)
* move core-header-related functions from crash_core.c to kexec_file.c
* drop hash-check code from purgatory
* modify purgatory asm to remove arch_kexec_apply_relocations_add()
* drop older kernel support
* drop vmlinux support (at least, for this series)


Patch #1 to #9 are essential for KEXEC_FILE support (plus IMA-based
verification):
  Patch #1 to #4 are all preparatory patches on generic side.
  Patch #5 to #8 are common for enabling kexec_file_load.
  Patch #9 is for 'Image'-specific loading.

Patch #10 to #11 are for KEXEC_VERIFY_SIG (arch-specific verification)
support

AKASHI Takahiro (11):
  resource: add walk_system_ram_res_rev()
  kexec_file: factor out arch_kexec_kernel_*() from x86, powerpc
  kexec_file: factor out crashdump elf header function from x86
  asm-generic: add kexec_file_load system call to unistd.h
  arm64: kexec_file: create purgatory
  arm64: kexec_file: load initrd, device-tree and purgatory segments
  arm64: kexec_file: set up for crash dump adding elf core header
  arm64: kexec_file: enable KEXEC_FILE config
  arm64: kexec_file: add Image format support
  include: pe.h: remove message[] from mz header definition
  arm64: kexec_file: enable KEXEC_VERIFY_SIG for Image

 arch/arm64/Kconfig                          |  30 +++
 arch/arm64/Makefile                         |   1 +
 arch/arm64/include/asm/kexec.h              |  93 +++++++
 arch/arm64/kernel/Makefile                  |   4 +-
 arch/arm64/kernel/kexec_image.c             | 105 ++++++++
 arch/arm64/kernel/machine_kexec_file.c      | 368 ++++++++++++++++++++++++++++
 arch/arm64/purgatory/Makefile               |  24 ++
 arch/arm64/purgatory/entry.S                |  55 +++++
 arch/powerpc/include/asm/kexec.h            |   2 +-
 arch/powerpc/kernel/kexec_elf_64.c          |   2 +-
 arch/powerpc/kernel/machine_kexec_file_64.c |  39 +--
 arch/x86/include/asm/kexec-bzimage64.h      |   2 +-
 arch/x86/kernel/crash.c                     | 324 ------------------------
 arch/x86/kernel/kexec-bzimage64.c           |   2 +-
 arch/x86/kernel/machine_kexec_64.c          |  45 +---
 include/linux/ioport.h                      |   3 +
 include/linux/kexec.h                       |  32 ++-
 include/linux/pe.h                          |   2 +-
 include/uapi/asm-generic/unistd.h           |   4 +-
 kernel/kexec_file.c                         | 365 ++++++++++++++++++++++++++-
 kernel/kexec_internal.h                     |  20 ++
 kernel/resource.c                           |  57 +++++
 22 files changed, 1158 insertions(+), 421 deletions(-)
 create mode 100644 arch/arm64/kernel/kexec_image.c
 create mode 100644 arch/arm64/kernel/machine_kexec_file.c
 create mode 100644 arch/arm64/purgatory/Makefile
 create mode 100644 arch/arm64/purgatory/entry.S

-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:52 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:52 +0900
Subject: [PATCH v7 02/11] kexec_file: factor out arch_kexec_kernel_*() from
 x86, powerpc
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-3-takahiro.akashi@linaro.org>

arch_kexec_kernel_*() and arch_kimage_file_post_load_cleanup can now be
duplicated among some architectures, so let's factor them out.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Dave Young <dyoung at redhat.com>
Cc: Vivek Goyal <vgoyal at redhat.com>
Cc: Baoquan He <bhe at redhat.com>
Cc: Michael Ellerman <mpe at ellerman.id.au>
Cc: Thiago Jung Bauermann <bauerman at linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/kexec.h            |  2 +-
 arch/powerpc/kernel/kexec_elf_64.c          |  2 +-
 arch/powerpc/kernel/machine_kexec_file_64.c | 39 ++------------------
 arch/x86/include/asm/kexec-bzimage64.h      |  2 +-
 arch/x86/kernel/kexec-bzimage64.c           |  2 +-
 arch/x86/kernel/machine_kexec_64.c          | 45 +----------------------
 include/linux/kexec.h                       | 15 ++++----
 kernel/kexec_file.c                         | 57 +++++++++++++++++++++++++++--
 8 files changed, 70 insertions(+), 94 deletions(-)

diff --git a/arch/powerpc/include/asm/kexec.h b/arch/powerpc/include/asm/kexec.h
index 4419d435639a..3fc293b090ee 100644
--- a/arch/powerpc/include/asm/kexec.h
+++ b/arch/powerpc/include/asm/kexec.h
@@ -93,7 +93,7 @@ static inline bool kdump_in_progress(void)
 }
 
 #ifdef CONFIG_KEXEC_FILE
-extern struct kexec_file_ops kexec_elf64_ops;
+extern const struct kexec_file_ops kexec_elf64_ops;
 
 #ifdef CONFIG_IMA_KEXEC
 #define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/powerpc/kernel/kexec_elf_64.c b/arch/powerpc/kernel/kexec_elf_64.c
index 9a42309b091a..6c78c11c7faf 100644
--- a/arch/powerpc/kernel/kexec_elf_64.c
+++ b/arch/powerpc/kernel/kexec_elf_64.c
@@ -657,7 +657,7 @@ static void *elf64_load(struct kimage *image, char *kernel_buf,
 	return ret ? ERR_PTR(ret) : fdt;
 }
 
-struct kexec_file_ops kexec_elf64_ops = {
+const struct kexec_file_ops kexec_elf64_ops = {
 	.probe = elf64_probe,
 	.load = elf64_load,
 };
diff --git a/arch/powerpc/kernel/machine_kexec_file_64.c b/arch/powerpc/kernel/machine_kexec_file_64.c
index e4395f937d63..a27ec647350c 100644
--- a/arch/powerpc/kernel/machine_kexec_file_64.c
+++ b/arch/powerpc/kernel/machine_kexec_file_64.c
@@ -31,52 +31,19 @@
 
 #define SLAVE_CODE_SIZE		256
 
-static struct kexec_file_ops *kexec_file_loaders[] = {
+const struct kexec_file_ops * const kexec_file_loaders[] = {
 	&kexec_elf64_ops,
+	NULL
 };
 
 int arch_kexec_kernel_image_probe(struct kimage *image, void *buf,
 				  unsigned long buf_len)
 {
-	int i, ret = -ENOEXEC;
-	struct kexec_file_ops *fops;
-
 	/* We don't support crash kernels yet. */
 	if (image->type == KEXEC_TYPE_CRASH)
 		return -ENOTSUPP;
 
-	for (i = 0; i < ARRAY_SIZE(kexec_file_loaders); i++) {
-		fops = kexec_file_loaders[i];
-		if (!fops || !fops->probe)
-			continue;
-
-		ret = fops->probe(buf, buf_len);
-		if (!ret) {
-			image->fops = fops;
-			return ret;
-		}
-	}
-
-	return ret;
-}
-
-void *arch_kexec_kernel_image_load(struct kimage *image)
-{
-	if (!image->fops || !image->fops->load)
-		return ERR_PTR(-ENOEXEC);
-
-	return image->fops->load(image, image->kernel_buf,
-				 image->kernel_buf_len, image->initrd_buf,
-				 image->initrd_buf_len, image->cmdline_buf,
-				 image->cmdline_buf_len);
-}
-
-int arch_kimage_file_post_load_cleanup(struct kimage *image)
-{
-	if (!image->fops || !image->fops->cleanup)
-		return 0;
-
-	return image->fops->cleanup(image->image_loader_data);
+	return _kexec_kernel_image_probe(image, buf, buf_len);
 }
 
 /**
diff --git a/arch/x86/include/asm/kexec-bzimage64.h b/arch/x86/include/asm/kexec-bzimage64.h
index 9f07cff43705..df89ee7d3e9e 100644
--- a/arch/x86/include/asm/kexec-bzimage64.h
+++ b/arch/x86/include/asm/kexec-bzimage64.h
@@ -2,6 +2,6 @@
 #ifndef _ASM_KEXEC_BZIMAGE64_H
 #define _ASM_KEXEC_BZIMAGE64_H
 
-extern struct kexec_file_ops kexec_bzImage64_ops;
+extern const struct kexec_file_ops kexec_bzImage64_ops;
 
 #endif  /* _ASM_KEXE_BZIMAGE64_H */
diff --git a/arch/x86/kernel/kexec-bzimage64.c b/arch/x86/kernel/kexec-bzimage64.c
index fb095ba0c02f..705654776c0c 100644
--- a/arch/x86/kernel/kexec-bzimage64.c
+++ b/arch/x86/kernel/kexec-bzimage64.c
@@ -538,7 +538,7 @@ static int bzImage64_verify_sig(const char *kernel, unsigned long kernel_len)
 }
 #endif
 
-struct kexec_file_ops kexec_bzImage64_ops = {
+const struct kexec_file_ops kexec_bzImage64_ops = {
 	.probe = bzImage64_probe,
 	.load = bzImage64_load,
 	.cleanup = bzImage64_cleanup,
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 1f790cf9d38f..2cdd29d64181 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -30,8 +30,9 @@
 #include <asm/set_memory.h>
 
 #ifdef CONFIG_KEXEC_FILE
-static struct kexec_file_ops *kexec_file_loaders[] = {
+const struct kexec_file_ops * const kexec_file_loaders[] = {
 		&kexec_bzImage64_ops,
+		NULL
 };
 #endif
 
@@ -363,27 +364,6 @@ void arch_crash_save_vmcoreinfo(void)
 /* arch-dependent functionality related to kexec file-based syscall */
 
 #ifdef CONFIG_KEXEC_FILE
-int arch_kexec_kernel_image_probe(struct kimage *image, void *buf,
-				  unsigned long buf_len)
-{
-	int i, ret = -ENOEXEC;
-	struct kexec_file_ops *fops;
-
-	for (i = 0; i < ARRAY_SIZE(kexec_file_loaders); i++) {
-		fops = kexec_file_loaders[i];
-		if (!fops || !fops->probe)
-			continue;
-
-		ret = fops->probe(buf, buf_len);
-		if (!ret) {
-			image->fops = fops;
-			return ret;
-		}
-	}
-
-	return ret;
-}
-
 void *arch_kexec_kernel_image_load(struct kimage *image)
 {
 	vfree(image->arch.elf_headers);
@@ -398,27 +378,6 @@ void *arch_kexec_kernel_image_load(struct kimage *image)
 				 image->cmdline_buf_len);
 }
 
-int arch_kimage_file_post_load_cleanup(struct kimage *image)
-{
-	if (!image->fops || !image->fops->cleanup)
-		return 0;
-
-	return image->fops->cleanup(image->image_loader_data);
-}
-
-#ifdef CONFIG_KEXEC_VERIFY_SIG
-int arch_kexec_kernel_verify_sig(struct kimage *image, void *kernel,
-				 unsigned long kernel_len)
-{
-	if (!image->fops || !image->fops->verify_sig) {
-		pr_debug("kernel loader does not support signature verification.");
-		return -EKEYREJECTED;
-	}
-
-	return image->fops->verify_sig(kernel, kernel_len);
-}
-#endif
-
 /*
  * Apply purgatory relocations.
  *
diff --git a/include/linux/kexec.h b/include/linux/kexec.h
index f16f6ceb3875..325980537125 100644
--- a/include/linux/kexec.h
+++ b/include/linux/kexec.h
@@ -209,7 +209,7 @@ struct kimage {
 	unsigned long cmdline_buf_len;
 
 	/* File operations provided by image loader */
-	struct kexec_file_ops *fops;
+	const struct kexec_file_ops *fops;
 
 	/* Image loader handling the kernel can store a pointer here */
 	void *image_loader_data;
@@ -277,12 +277,13 @@ int crash_shrink_memory(unsigned long new_size);
 size_t crash_get_memory_size(void);
 void crash_free_reserved_phys_range(unsigned long begin, unsigned long end);
 
-int __weak arch_kexec_kernel_image_probe(struct kimage *image, void *buf,
-					 unsigned long buf_len);
-void * __weak arch_kexec_kernel_image_load(struct kimage *image);
-int __weak arch_kimage_file_post_load_cleanup(struct kimage *image);
-int __weak arch_kexec_kernel_verify_sig(struct kimage *image, void *buf,
-					unsigned long buf_len);
+int _kexec_kernel_image_probe(struct kimage *image, void *buf,
+			      unsigned long buf_len);
+void *_kexec_kernel_image_load(struct kimage *image);
+int _kimage_file_post_load_cleanup(struct kimage *image);
+int _kexec_kernel_verify_sig(struct kimage *image, void *buf,
+			     unsigned long buf_len);
+
 int __weak arch_kexec_apply_relocations_add(const Elf_Ehdr *ehdr,
 					Elf_Shdr *sechdrs, unsigned int relsec);
 int __weak arch_kexec_apply_relocations(const Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index e5bcd94c1efb..c95e3ffd4be8 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -26,30 +26,79 @@
 #include <linux/vmalloc.h>
 #include "kexec_internal.h"
 
+const __weak struct kexec_file_ops * const kexec_file_loaders[] = {NULL};
+
 static int kexec_calculate_store_digests(struct kimage *image);
 
+int _kexec_kernel_image_probe(struct kimage *image, void *buf,
+			     unsigned long buf_len)
+{
+	const struct kexec_file_ops * const *fops;
+	int ret = -ENOEXEC;
+
+	for (fops = &kexec_file_loaders[0]; *fops && (*fops)->probe; ++fops) {
+		ret = (*fops)->probe(buf, buf_len);
+		if (!ret) {
+			image->fops = *fops;
+			return ret;
+		}
+	}
+
+	return ret;
+}
+
 /* Architectures can provide this probe function */
 int __weak arch_kexec_kernel_image_probe(struct kimage *image, void *buf,
 					 unsigned long buf_len)
 {
-	return -ENOEXEC;
+	return _kexec_kernel_image_probe(image, buf, buf_len);
+}
+
+void *_kexec_kernel_image_load(struct kimage *image)
+{
+	if (!image->fops || !image->fops->load)
+		return ERR_PTR(-ENOEXEC);
+
+	return image->fops->load(image, image->kernel_buf,
+				 image->kernel_buf_len, image->initrd_buf,
+				 image->initrd_buf_len, image->cmdline_buf,
+				 image->cmdline_buf_len);
 }
 
 void * __weak arch_kexec_kernel_image_load(struct kimage *image)
 {
-	return ERR_PTR(-ENOEXEC);
+	return _kexec_kernel_image_load(image);
+}
+
+int _kimage_file_post_load_cleanup(struct kimage *image)
+{
+	if (!image->fops || !image->fops->cleanup)
+		return 0;
+
+	return image->fops->cleanup(image->image_loader_data);
 }
 
 int __weak arch_kimage_file_post_load_cleanup(struct kimage *image)
 {
-	return -EINVAL;
+	return _kimage_file_post_load_cleanup(image);
 }
 
 #ifdef CONFIG_KEXEC_VERIFY_SIG
+int _kexec_kernel_verify_sig(struct kimage *image, void *buf,
+			    unsigned long buf_len)
+{
+	if (!image->fops || !image->fops->verify_sig) {
+		pr_debug("kernel loader does not support signature verification.\n");
+		return -EKEYREJECTED;
+	}
+
+	return image->fops->verify_sig(buf, buf_len);
+}
+
 int __weak arch_kexec_kernel_verify_sig(struct kimage *image, void *buf,
 					unsigned long buf_len)
 {
-	return -EKEYREJECTED;
+	return _kexec_kernel_verify_sig(image, buf, buf_len);
 }
 #endif
 
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:51 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:51 +0900
Subject: [PATCH v7 01/11] resource: add walk_system_ram_res_rev()
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-2-takahiro.akashi@linaro.org>

This function, being a variant of walk_system_ram_res() introduced in
commit 8c86e70acead ("resource: provide new functions to walk through
resources"), walks through a list of all the resources of System RAM
in reversed order, i.e., from higher to lower.

It will be used in kexec_file implementation on arm64.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Vivek Goyal <vgoyal at redhat.com>
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Linus Torvalds <torvalds at linux-foundation.org>
---
 include/linux/ioport.h |  3 +++
 kernel/resource.c      | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)

diff --git a/include/linux/ioport.h b/include/linux/ioport.h
index 93b4183cf53d..322ac1f58b36 100644
--- a/include/linux/ioport.h
+++ b/include/linux/ioport.h
@@ -277,6 +277,9 @@ extern int
 walk_system_ram_res(u64 start, u64 end, void *arg,
 		    int (*func)(struct resource *, void *));
 extern int
+walk_system_ram_res_rev(u64 start, u64 end, void *arg,
+			int (*func)(struct resource *, void *));
+extern int
 walk_iomem_res_desc(unsigned long desc, unsigned long flags, u64 start, u64 end,
 		    void *arg, int (*func)(struct resource *, void *));
 
diff --git a/kernel/resource.c b/kernel/resource.c
index 54ba6de3757c..aefc1d5d6b75 100644
--- a/kernel/resource.c
+++ b/kernel/resource.c
@@ -23,6 +23,8 @@
 #include <linux/pfn.h>
 #include <linux/mm.h>
 #include <linux/resource_ext.h>
+#include <linux/string.h>
+#include <linux/vmalloc.h>
 #include <asm/io.h>
 
 
@@ -486,6 +488,61 @@ int walk_mem_res(u64 start, u64 end, void *arg,
 				     arg, func);
 }
 
+int walk_system_ram_res_rev(u64 start, u64 end, void *arg,
+				int (*func)(struct resource *, void *))
+{
+	struct resource res, *rams;
+	int rams_size = 16, i;
+	int ret = -1;
+
+	/* create a list */
+	rams = vmalloc(sizeof(struct resource) * rams_size);
+	if (!rams)
+		return ret;
+
+	res.start = start;
+	res.end = end;
+	res.flags = IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY;
+	i = 0;
+	while ((res.start < res.end) &&
+		(!find_next_iomem_res(&res, IORES_DESC_NONE, true))) {
+		if (i >= rams_size) {
+			/* re-alloc */
+			struct resource *rams_new;
+			int rams_new_size;
+
+			rams_new_size = rams_size + 16;
+			rams_new = vmalloc(sizeof(struct resource)
+							* rams_new_size);
+			if (!rams_new)
+				goto out;
+
+			memcpy(rams_new, rams,
+					sizeof(struct resource) * rams_size);
+			vfree(rams);
+			rams = rams_new;
+			rams_size = rams_new_size;
+		}
+
+		rams[i].start = res.start;
+		rams[i++].end = res.end;
+
+		res.start = res.end + 1;
+		res.end = end;
+	}
+
+	/* go reverse */
+	for (i--; i >= 0; i--) {
+		ret = (*func)(&rams[i], arg);
+		if (ret)
+			break;
+	}
+
+out:
+	vfree(rams);
+	return ret;
+}
+
 #if !defined(CONFIG_ARCH_HAS_WALK_MEMORY)
 
 /*
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:54 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:54 +0900
Subject: [PATCH v7 04/11] asm-generic: add kexec_file_load system call to
 unistd.h
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-5-takahiro.akashi@linaro.org>

The initial user of this system call number is arm64.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Acked-by: Arnd Bergmann <arnd at arndb.de>
---
 include/uapi/asm-generic/unistd.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 8b87de067bc7..33761525ed2f 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -732,9 +732,11 @@ __SYSCALL(__NR_pkey_alloc,    sys_pkey_alloc)
 __SYSCALL(__NR_pkey_free,     sys_pkey_free)
 #define __NR_statx 291
 __SYSCALL(__NR_statx,     sys_statx)
+#define __NR_kexec_file_load 292
+__SYSCALL(__NR_kexec_file_load,     sys_kexec_file_load)
 
 #undef __NR_syscalls
-#define __NR_syscalls 292
+#define __NR_syscalls 293
 
 /*
  * All syscalls below here should go away really,
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:53 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:53 +0900
Subject: [PATCH v7 03/11] kexec_file: factor out crashdump elf header function
 from x86
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-4-takahiro.akashi@linaro.org>

prepare_elf_headers() can also be useful for other architectures,
including arm64. So let it factored out.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Dave Young <dyoung at redhat.com>
Cc: Vivek Goyal <vgoyal at redhat.com>
Cc: Baoquan He <bhe at redhat.com>
---
 arch/x86/kernel/crash.c | 324 ------------------------------------------------
 include/linux/kexec.h   |  17 +++
 kernel/kexec_file.c     | 308 +++++++++++++++++++++++++++++++++++++++++++++
 kernel/kexec_internal.h |  20 +++
 4 files changed, 345 insertions(+), 324 deletions(-)

diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 10e74d4778a1..bb8f3dcddaaa 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -21,7 +21,6 @@
 #include <linux/elf.h>
 #include <linux/elfcore.h>
 #include <linux/export.h>
-#include <linux/slab.h>
 #include <linux/vmalloc.h>
 
 #include <asm/processor.h>
@@ -41,34 +40,6 @@
 /* Alignment required for elf header segment */
 #define ELF_CORE_HEADER_ALIGN   4096
 
-/* This primarily represents number of split ranges due to exclusion */
-#define CRASH_MAX_RANGES	16
-
-struct crash_mem_range {
-	u64 start, end;
-};
-
-struct crash_mem {
-	unsigned int nr_ranges;
-	struct crash_mem_range ranges[CRASH_MAX_RANGES];
-};
-
-/* Misc data about ram ranges needed to prepare elf headers */
-struct crash_elf_data {
-	struct kimage *image;
-	/*
-	 * Total number of ram ranges we have after various adjustments for
-	 * crash reserved region, etc.
-	 */
-	unsigned int max_nr_ranges;
-
-	/* Pointer to elf header */
-	void *ehdr;
-	/* Pointer to next phdr */
-	void *bufp;
-	struct crash_mem mem;
-};
-
 /* Used while preparing memory map entries for second kernel */
 struct crash_memmap_data {
 	struct boot_params *params;
@@ -209,301 +180,6 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 }
 
 #ifdef CONFIG_KEXEC_FILE
-static int get_nr_ram_ranges_callback(struct resource *res, void *arg)
-{
-	unsigned int *nr_ranges = arg;
-
-	(*nr_ranges)++;
-	return 0;
-}
-
-
-/* Gather all the required information to prepare elf headers for ram regions */
-static void fill_up_crash_elf_data(struct crash_elf_data *ced,
-				   struct kimage *image)
-{
-	unsigned int nr_ranges = 0;
-
-	ced->image = image;
-
-	walk_system_ram_res(0, -1, &nr_ranges,
-				get_nr_ram_ranges_callback);
-
-	ced->max_nr_ranges = nr_ranges;
-
-	/* Exclusion of crash region could split memory ranges */
-	ced->max_nr_ranges++;
-
-	/* If crashk_low_res is not 0, another range split possible */
-	if (crashk_low_res.end)
-		ced->max_nr_ranges++;
-}
-
-static int exclude_mem_range(struct crash_mem *mem,
-		unsigned long long mstart, unsigned long long mend)
-{
-	int i, j;
-	unsigned long long start, end;
-	struct crash_mem_range temp_range = {0, 0};
-
-	for (i = 0; i < mem->nr_ranges; i++) {
-		start = mem->ranges[i].start;
-		end = mem->ranges[i].end;
-
-		if (mstart > end || mend < start)
-			continue;
-
-		/* Truncate any area outside of range */
-		if (mstart < start)
-			mstart = start;
-		if (mend > end)
-			mend = end;
-
-		/* Found completely overlapping range */
-		if (mstart == start && mend == end) {
-			mem->ranges[i].start = 0;
-			mem->ranges[i].end = 0;
-			if (i < mem->nr_ranges - 1) {
-				/* Shift rest of the ranges to left */
-				for (j = i; j < mem->nr_ranges - 1; j++) {
-					mem->ranges[j].start =
-						mem->ranges[j+1].start;
-					mem->ranges[j].end =
-							mem->ranges[j+1].end;
-				}
-			}
-			mem->nr_ranges--;
-			return 0;
-		}
-
-		if (mstart > start && mend < end) {
-			/* Split original range */
-			mem->ranges[i].end = mstart - 1;
-			temp_range.start = mend + 1;
-			temp_range.end = end;
-		} else if (mstart != start)
-			mem->ranges[i].end = mstart - 1;
-		else
-			mem->ranges[i].start = mend + 1;
-		break;
-	}
-
-	/* If a split happend, add the split to array */
-	if (!temp_range.end)
-		return 0;
-
-	/* Split happened */
-	if (i == CRASH_MAX_RANGES - 1) {
-		pr_err("Too many crash ranges after split\n");
-		return -ENOMEM;
-	}
-
-	/* Location where new range should go */
-	j = i + 1;
-	if (j < mem->nr_ranges) {
-		/* Move over all ranges one slot towards the end */
-		for (i = mem->nr_ranges - 1; i >= j; i--)
-			mem->ranges[i + 1] = mem->ranges[i];
-	}
-
-	mem->ranges[j].start = temp_range.start;
-	mem->ranges[j].end = temp_range.end;
-	mem->nr_ranges++;
-	return 0;
-}
-
-/*
- * Look for any unwanted ranges between mstart, mend and remove them. This
- * might lead to split and split ranges are put in ced->mem.ranges[] array
- */
-static int elf_header_exclude_ranges(struct crash_elf_data *ced,
-		unsigned long long mstart, unsigned long long mend)
-{
-	struct crash_mem *cmem = &ced->mem;
-	int ret = 0;
-
-	memset(cmem->ranges, 0, sizeof(cmem->ranges));
-
-	cmem->ranges[0].start = mstart;
-	cmem->ranges[0].end = mend;
-	cmem->nr_ranges = 1;
-
-	/* Exclude crashkernel region */
-	ret = exclude_mem_range(cmem, crashk_res.start, crashk_res.end);
-	if (ret)
-		return ret;
-
-	if (crashk_low_res.end) {
-		ret = exclude_mem_range(cmem, crashk_low_res.start, crashk_low_res.end);
-		if (ret)
-			return ret;
-	}
-
-	return ret;
-}
-
-static int prepare_elf64_ram_headers_callback(struct resource *res, void *arg)
-{
-	struct crash_elf_data *ced = arg;
-	Elf64_Ehdr *ehdr;
-	Elf64_Phdr *phdr;
-	unsigned long mstart, mend;
-	struct kimage *image = ced->image;
-	struct crash_mem *cmem;
-	int ret, i;
-
-	ehdr = ced->ehdr;
-
-	/* Exclude unwanted mem ranges */
-	ret = elf_header_exclude_ranges(ced, res->start, res->end);
-	if (ret)
-		return ret;
-
-	/* Go through all the ranges in ced->mem.ranges[] and prepare phdr */
-	cmem = &ced->mem;
-
-	for (i = 0; i < cmem->nr_ranges; i++) {
-		mstart = cmem->ranges[i].start;
-		mend = cmem->ranges[i].end;
-
-		phdr = ced->bufp;
-		ced->bufp += sizeof(Elf64_Phdr);
-
-		phdr->p_type = PT_LOAD;
-		phdr->p_flags = PF_R|PF_W|PF_X;
-		phdr->p_offset  = mstart;
-
-		/*
-		 * If a range matches backup region, adjust offset to backup
-		 * segment.
-		 */
-		if (mstart == image->arch.backup_src_start &&
-		    (mend - mstart + 1) == image->arch.backup_src_sz)
-			phdr->p_offset = image->arch.backup_load_addr;
-
-		phdr->p_paddr = mstart;
-		phdr->p_vaddr = (unsigned long long) __va(mstart);
-		phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;
-		phdr->p_align = 0;
-		ehdr->e_phnum++;
-		pr_debug("Crash PT_LOAD elf header. phdr=%p vaddr=0x%llx, paddr=0x%llx, sz=0x%llx e_phnum=%d p_offset=0x%llx\n",
-			phdr, phdr->p_vaddr, phdr->p_paddr, phdr->p_filesz,
-			ehdr->e_phnum, phdr->p_offset);
-	}
-
-	return ret;
-}
-
-static int prepare_elf64_headers(struct crash_elf_data *ced,
-		void **addr, unsigned long *sz)
-{
-	Elf64_Ehdr *ehdr;
-	Elf64_Phdr *phdr;
-	unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
-	unsigned char *buf, *bufp;
-	unsigned int cpu;
-	unsigned long long notes_addr;
-	int ret;
-
-	/* extra phdr for vmcoreinfo elf note */
-	nr_phdr = nr_cpus + 1;
-	nr_phdr += ced->max_nr_ranges;
-
-	/*
-	 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
-	 * area on x86_64 (ffffffff80000000 - ffffffffa0000000).
-	 * I think this is required by tools like gdb. So same physical
-	 * memory will be mapped in two elf headers. One will contain kernel
-	 * text virtual addresses and other will have __va(physical) addresses.
-	 */
-
-	nr_phdr++;
-	elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
-	elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
-
-	buf = vzalloc(elf_sz);
-	if (!buf)
-		return -ENOMEM;
-
-	bufp = buf;
-	ehdr = (Elf64_Ehdr *)bufp;
-	bufp += sizeof(Elf64_Ehdr);
-	memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
-	ehdr->e_ident[EI_CLASS] = ELFCLASS64;
-	ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
-	ehdr->e_ident[EI_VERSION] = EV_CURRENT;
-	ehdr->e_ident[EI_OSABI] = ELF_OSABI;
-	memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
-	ehdr->e_type = ET_CORE;
-	ehdr->e_machine = ELF_ARCH;
-	ehdr->e_version = EV_CURRENT;
-	ehdr->e_phoff = sizeof(Elf64_Ehdr);
-	ehdr->e_ehsize = sizeof(Elf64_Ehdr);
-	ehdr->e_phentsize = sizeof(Elf64_Phdr);
-
-	/* Prepare one phdr of type PT_NOTE for each present cpu */
-	for_each_present_cpu(cpu) {
-		phdr = (Elf64_Phdr *)bufp;
-		bufp += sizeof(Elf64_Phdr);
-		phdr->p_type = PT_NOTE;
-		notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
-		phdr->p_offset = phdr->p_paddr = notes_addr;
-		phdr->p_filesz = phdr->p_memsz = sizeof(note_buf_t);
-		(ehdr->e_phnum)++;
-	}
-
-	/* Prepare one PT_NOTE header for vmcoreinfo */
-	phdr = (Elf64_Phdr *)bufp;
-	bufp += sizeof(Elf64_Phdr);
-	phdr->p_type = PT_NOTE;
-	phdr->p_offset = phdr->p_paddr = paddr_vmcoreinfo_note();
-	phdr->p_filesz = phdr->p_memsz = VMCOREINFO_NOTE_SIZE;
-	(ehdr->e_phnum)++;
-
-#ifdef CONFIG_X86_64
-	/* Prepare PT_LOAD type program header for kernel text region */
-	phdr = (Elf64_Phdr *)bufp;
-	bufp += sizeof(Elf64_Phdr);
-	phdr->p_type = PT_LOAD;
-	phdr->p_flags = PF_R|PF_W|PF_X;
-	phdr->p_vaddr = (Elf64_Addr)_text;
-	phdr->p_filesz = phdr->p_memsz = _end - _text;
-	phdr->p_offset = phdr->p_paddr = __pa_symbol(_text);
-	(ehdr->e_phnum)++;
-#endif
-
-	/* Prepare PT_LOAD headers for system ram chunks. */
-	ced->ehdr = ehdr;
-	ced->bufp = bufp;
-	ret = walk_system_ram_res(0, -1, ced,
-			prepare_elf64_ram_headers_callback);
-	if (ret < 0)
-		return ret;
-
-	*addr = buf;
-	*sz = elf_sz;
-	return 0;
-}
-
-/* Prepare elf headers. Return addr and size */
-static int prepare_elf_headers(struct kimage *image, void **addr,
-					unsigned long *sz)
-{
-	struct crash_elf_data *ced;
-	int ret;
-
-	ced = kzalloc(sizeof(*ced), GFP_KERNEL);
-	if (!ced)
-		return -ENOMEM;
-
-	fill_up_crash_elf_data(ced, image);
-
-	/* By default prepare 64bit headers */
-	ret =  prepare_elf64_headers(ced, addr, sz);
-	kfree(ced);
-	return ret;
-}
-
 static int add_e820_entry(struct boot_params *params, struct e820_entry *entry)
 {
 	unsigned int nr_e820_entries;
diff --git a/include/linux/kexec.h b/include/linux/kexec.h
index 325980537125..a8ac11633969 100644
--- a/include/linux/kexec.h
+++ b/include/linux/kexec.h
@@ -163,6 +163,23 @@ int __weak arch_kexec_walk_mem(struct kexec_buf *kbuf,
 			       int (*func)(struct resource *, void *));
 extern int kexec_add_buffer(struct kexec_buf *kbuf);
 int kexec_locate_mem_hole(struct kexec_buf *kbuf);
+extern int prepare_elf_headers(struct kimage *image, void **addr,
+				unsigned long *sz);
+
+/* This primarily represents number of split ranges due to exclusion */
+#define CRASH_MAX_RANGES        16
+
+struct crash_mem_range {
+	u64 start, end;
+};
+
+struct crash_mem {
+	unsigned int nr_ranges;
+	struct crash_mem_range ranges[CRASH_MAX_RANGES];
+};
+
+extern int exclude_mem_range(struct crash_mem *mem,
+		unsigned long long mstart, unsigned long long mend);
 #endif /* CONFIG_KEXEC_FILE */
 
 struct kimage {
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index c95e3ffd4be8..b2c425e6afc6 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -22,6 +22,11 @@
 #include <linux/ima.h>
 #include <crypto/hash.h>
 #include <crypto/sha.h>
+#include <linux/elf.h>
+#include <linux/elfcore.h>
+#include <linux/kernel.h>
+#include <linux/kexec.h>
+#include <linux/slab.h>
 #include <linux/syscalls.h>
 #include <linux/vmalloc.h>
 #include "kexec_internal.h"
@@ -1071,3 +1076,306 @@ int kexec_purgatory_get_set_symbol(struct kimage *image, const char *name,
 
 	return 0;
 }
+
+static int get_nr_ram_ranges_callback(struct resource *res, void *arg)
+{
+	unsigned int *nr_ranges = arg;
+
+	(*nr_ranges)++;
+	return 0;
+}
+
+
+/* Gather all the required information to prepare elf headers for ram regions */
+static void fill_up_crash_elf_data(struct crash_elf_data *ced,
+				   struct kimage *image)
+{
+	unsigned int nr_ranges = 0;
+
+	ced->image = image;
+
+	walk_system_ram_res(0, -1, &nr_ranges,
+				get_nr_ram_ranges_callback);
+
+	ced->max_nr_ranges = nr_ranges;
+
+	/* Exclusion of crash region could split memory ranges */
+	ced->max_nr_ranges++;
+
+#ifdef CONFIG_X86_64
+	/* If crashk_low_res is not 0, another range split possible */
+	if (crashk_low_res.end)
+		ced->max_nr_ranges++;
+#endif
+}
+
+int exclude_mem_range(struct crash_mem *mem,
+		unsigned long long mstart, unsigned long long mend)
+{
+	int i, j;
+	unsigned long long start, end;
+	struct crash_mem_range temp_range = {0, 0};
+
+	for (i = 0; i < mem->nr_ranges; i++) {
+		start = mem->ranges[i].start;
+		end = mem->ranges[i].end;
+
+		if (mstart > end || mend < start)
+			continue;
+
+		/* Truncate any area outside of range */
+		if (mstart < start)
+			mstart = start;
+		if (mend > end)
+			mend = end;
+
+		/* Found completely overlapping range */
+		if (mstart == start && mend == end) {
+			mem->ranges[i].start = 0;
+			mem->ranges[i].end = 0;
+			if (i < mem->nr_ranges - 1) {
+				/* Shift rest of the ranges to left */
+				for (j = i; j < mem->nr_ranges - 1; j++) {
+					mem->ranges[j].start =
+						mem->ranges[j+1].start;
+					mem->ranges[j].end =
+							mem->ranges[j+1].end;
+				}
+			}
+			mem->nr_ranges--;
+			return 0;
+		}
+
+		if (mstart > start && mend < end) {
+			/* Split original range */
+			mem->ranges[i].end = mstart - 1;
+			temp_range.start = mend + 1;
+			temp_range.end = end;
+		} else if (mstart != start)
+			mem->ranges[i].end = mstart - 1;
+		else
+			mem->ranges[i].start = mend + 1;
+		break;
+	}
+
+	/* If a split happened, add the split to array */
+	if (!temp_range.end)
+		return 0;
+
+	/* Split happened */
+	if (i == CRASH_MAX_RANGES - 1) {
+		pr_err("Too many crash ranges after split\n");
+		return -ENOMEM;
+	}
+
+	/* Location where new range should go */
+	j = i + 1;
+	if (j < mem->nr_ranges) {
+		/* Move over all ranges one slot towards the end */
+		for (i = mem->nr_ranges - 1; i >= j; i--)
+			mem->ranges[i + 1] = mem->ranges[i];
+	}
+
+	mem->ranges[j].start = temp_range.start;
+	mem->ranges[j].end = temp_range.end;
+	mem->nr_ranges++;
+	return 0;
+}
+
+/*
+ * Look for any unwanted ranges between mstart, mend and remove them. This
+ * might lead to split and split ranges are put in ced->mem.ranges[] array
+ */
+static int elf_header_exclude_ranges(struct crash_elf_data *ced,
+		unsigned long long mstart, unsigned long long mend)
+{
+	struct crash_mem *cmem = &ced->mem;
+	int ret = 0;
+
+	memset(cmem->ranges, 0, sizeof(cmem->ranges));
+
+	cmem->ranges[0].start = mstart;
+	cmem->ranges[0].end = mend;
+	cmem->nr_ranges = 1;
+
+	/* Exclude crashkernel region */
+	ret = exclude_mem_range(cmem, crashk_res.start, crashk_res.end);
+	if (ret)
+		return ret;
+
+#ifdef CONFIG_X86_64
+	if (crashk_low_res.end) {
+		ret = exclude_mem_range(cmem, crashk_low_res.start,
+						crashk_low_res.end);
+		if (ret)
+			return ret;
+	}
+#endif
+
+	return ret;
+}
+
+static int prepare_elf64_ram_headers_callback(struct resource *res, void *arg)
+{
+	struct crash_elf_data *ced = arg;
+	Elf64_Ehdr *ehdr;
+	Elf64_Phdr *phdr;
+	unsigned long mstart, mend;
+#ifdef CONFIG_X86_64
+	struct kimage *image = ced->image;
+#endif
+	struct crash_mem *cmem;
+	int ret, i;
+
+	ehdr = ced->ehdr;
+
+	/* Exclude unwanted mem ranges */
+	ret = elf_header_exclude_ranges(ced, res->start, res->end);
+	if (ret)
+		return ret;
+
+	/* Go through all the ranges in ced->mem.ranges[] and prepare phdr */
+	cmem = &ced->mem;
+
+	for (i = 0; i < cmem->nr_ranges; i++) {
+		mstart = cmem->ranges[i].start;
+		mend = cmem->ranges[i].end;
+
+		phdr = ced->bufp;
+		ced->bufp += sizeof(Elf64_Phdr);
+
+		phdr->p_type = PT_LOAD;
+		phdr->p_flags = PF_R|PF_W|PF_X;
+		phdr->p_offset  = mstart;
+
+#ifdef CONFIG_X86_64
+		/*
+		 * If a range matches backup region, adjust offset to backup
+		 * segment.
+		 */
+		if (mstart == image->arch.backup_src_start &&
+		    (mend - mstart + 1) == image->arch.backup_src_sz)
+			phdr->p_offset = image->arch.backup_load_addr;
+#endif
+
+		phdr->p_paddr = mstart;
+		phdr->p_vaddr = (unsigned long long) __va(mstart);
+		phdr->p_filesz = phdr->p_memsz = mend - mstart + 1;
+		phdr->p_align = 0;
+		ehdr->e_phnum++;
+		pr_debug("Crash PT_LOAD elf header. phdr=%p vaddr=0x%llx, paddr=0x%llx, sz=0x%llx e_phnum=%d p_offset=0x%llx\n",
+			phdr, phdr->p_vaddr, phdr->p_paddr, phdr->p_filesz,
+			ehdr->e_phnum, phdr->p_offset);
+	}
+
+	return ret;
+}
+
+static int prepare_elf64_headers(struct crash_elf_data *ced,
+		void **addr, unsigned long *sz)
+{
+	Elf64_Ehdr *ehdr;
+	Elf64_Phdr *phdr;
+	unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
+	unsigned char *buf, *bufp;
+	unsigned int cpu;
+	unsigned long long notes_addr;
+	int ret;
+
+	/* extra phdr for vmcoreinfo elf note */
+	nr_phdr = nr_cpus + 1;
+	nr_phdr += ced->max_nr_ranges;
+
+	/*
+	 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
+	 * area on x86_64 (ffffffff80000000 - ffffffffa0000000).
+	 * I think this is required by tools like gdb. So same physical
+	 * memory will be mapped in two elf headers. One will contain kernel
+	 * text virtual addresses and other will have __va(physical) addresses.
+	 */
+
+	nr_phdr++;
+	elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
+	elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
+
+	buf = vzalloc(elf_sz);
+	if (!buf)
+		return -ENOMEM;
+
+	bufp = buf;
+	ehdr = (Elf64_Ehdr *)bufp;
+	bufp += sizeof(Elf64_Ehdr);
+	memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
+	ehdr->e_ident[EI_CLASS] = ELFCLASS64;
+	ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
+	ehdr->e_ident[EI_VERSION] = EV_CURRENT;
+	ehdr->e_ident[EI_OSABI] = ELF_OSABI;
+	memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
+	ehdr->e_type = ET_CORE;
+	ehdr->e_machine = ELF_ARCH;
+	ehdr->e_version = EV_CURRENT;
+	ehdr->e_phoff = sizeof(Elf64_Ehdr);
+	ehdr->e_ehsize = sizeof(Elf64_Ehdr);
+	ehdr->e_phentsize = sizeof(Elf64_Phdr);
+
+	/* Prepare one phdr of type PT_NOTE for each present cpu */
+	for_each_present_cpu(cpu) {
+		phdr = (Elf64_Phdr *)bufp;
+		bufp += sizeof(Elf64_Phdr);
+		phdr->p_type = PT_NOTE;
+		notes_addr = per_cpu_ptr_to_phys(per_cpu_ptr(crash_notes, cpu));
+		phdr->p_offset = phdr->p_paddr = notes_addr;
+		phdr->p_filesz = phdr->p_memsz = sizeof(note_buf_t);
+		(ehdr->e_phnum)++;
+	}
+
+	/* Prepare one PT_NOTE header for vmcoreinfo */
+	phdr = (Elf64_Phdr *)bufp;
+	bufp += sizeof(Elf64_Phdr);
+	phdr->p_type = PT_NOTE;
+	phdr->p_offset = phdr->p_paddr = paddr_vmcoreinfo_note();
+	phdr->p_filesz = phdr->p_memsz = VMCOREINFO_NOTE_SIZE;
+	(ehdr->e_phnum)++;
+
+#ifdef CONFIG_X86_64
+	/* Prepare PT_LOAD type program header for kernel text region */
+	phdr = (Elf64_Phdr *)bufp;
+	bufp += sizeof(Elf64_Phdr);
+	phdr->p_type = PT_LOAD;
+	phdr->p_flags = PF_R|PF_W|PF_X;
+	phdr->p_vaddr = (Elf64_Addr)_text;
+	phdr->p_filesz = phdr->p_memsz = _end - _text;
+	phdr->p_offset = phdr->p_paddr = __pa_symbol(_text);
+	(ehdr->e_phnum)++;
+#endif
+
+	/* Prepare PT_LOAD headers for system ram chunks. */
+	ced->ehdr = ehdr;
+	ced->bufp = bufp;
+	ret = walk_system_ram_res(0, -1, ced,
+			prepare_elf64_ram_headers_callback);
+	if (ret < 0)
+		return ret;
+
+	*addr = buf;
+	*sz = elf_sz;
+	return 0;
+}
+
+/* Prepare elf headers. Return addr and size */
+int prepare_elf_headers(struct kimage *image, void **addr, unsigned long *sz)
+{
+	struct crash_elf_data *ced;
+	int ret;
+
+	ced = kzalloc(sizeof(*ced), GFP_KERNEL);
+	if (!ced)
+		return -ENOMEM;
+
+	fill_up_crash_elf_data(ced, image);
+
+	/* By default prepare 64bit headers */
+	ret =  prepare_elf64_headers(ced, addr, sz);
+	kfree(ced);
+	return ret;
+}
diff --git a/kernel/kexec_internal.h b/kernel/kexec_internal.h
index 48aaf2ac0d0d..2415e1f5e558 100644
--- a/kernel/kexec_internal.h
+++ b/kernel/kexec_internal.h
@@ -17,6 +17,26 @@ extern struct mutex kexec_mutex;
 
 #ifdef CONFIG_KEXEC_FILE
 #include <linux/purgatory.h>
+
+/* Alignment required for elf header segment */
+#define ELF_CORE_HEADER_ALIGN   4096
+
+/* Misc data about ram ranges needed to prepare elf headers */
+struct crash_elf_data {
+	struct kimage *image;
+	/*
+	 * Total number of ram ranges we have after various adjustments for
+	 * crash reserved region, etc.
+	 */
+	unsigned int max_nr_ranges;
+
+	/* Pointer to elf header */
+	void *ehdr;
+	/* Pointer to next phdr */
+	void *bufp;
+	struct crash_mem mem;
+};
+
 void kimage_file_post_load_cleanup(struct kimage *image);
 extern char kexec_purgatory[];
 extern size_t kexec_purgatory_size;
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:55 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:55 +0900
Subject: [PATCH v7 05/11] arm64: kexec_file: create purgatory
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-6-takahiro.akashi@linaro.org>

This is a basic purgatory, or a kind of glue code between the two kernels,
for arm64.

Since purgatory is assumed to be relocatable (not executable) object by
kexec generic code, arch_kexec_apply_relocations_add() is required in
general. Arm64's purgatory, however, is a simple asm and all the references
can be resolved as local, no re-linking is needed here.

Please note that even if we don't support digest check at purgatory we
need purgatory_sha_regions and purgatory_sha256_digest as they are
referenced by generic kexec code.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Catalin Marinas <catalin.marinas at arm.com>
Cc: Will Deacon <will.deacon at arm.com>
---
 arch/arm64/Makefile           |  1 +
 arch/arm64/purgatory/Makefile | 24 +++++++++++++++++++
 arch/arm64/purgatory/entry.S  | 55 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 80 insertions(+)
 create mode 100644 arch/arm64/purgatory/Makefile
 create mode 100644 arch/arm64/purgatory/entry.S

diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index b481b4a7c011..0f0742e98c08 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -113,6 +113,7 @@ core-$(CONFIG_XEN) += arch/arm64/xen/
 core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
 libs-y		:= arch/arm64/lib/ $(libs-y)
 core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
+core-$(CONFIG_KEXEC_FILE) += arch/arm64/purgatory/
 
 # Default target when executing plain make
 boot		:= arch/arm64/boot
diff --git a/arch/arm64/purgatory/Makefile b/arch/arm64/purgatory/Makefile
new file mode 100644
index 000000000000..c2127a2cbd51
--- /dev/null
+++ b/arch/arm64/purgatory/Makefile
@@ -0,0 +1,24 @@
+OBJECT_FILES_NON_STANDARD := y
+
+purgatory-y := entry.o
+
+targets += $(purgatory-y)
+PURGATORY_OBJS = $(addprefix $(obj)/,$(purgatory-y))
+
+LDFLAGS_purgatory.ro := -e purgatory_start -r --no-undefined \
+					-nostdlib -z nodefaultlib
+targets += purgatory.ro
+
+$(obj)/purgatory.ro: $(PURGATORY_OBJS) FORCE
+		$(call if_changed,ld)
+
+targets += kexec_purgatory.c
+
+CMD_BIN2C = $(objtree)/scripts/basic/bin2c
+quiet_cmd_bin2c = BIN2C $@
+	cmd_bin2c = $(CMD_BIN2C) kexec_purgatory < $< > $@
+
+$(obj)/kexec_purgatory.c: $(obj)/purgatory.ro FORCE
+	$(call if_changed,bin2c)
+
+obj-${CONFIG_KEXEC_FILE}	+= kexec_purgatory.o
diff --git a/arch/arm64/purgatory/entry.S b/arch/arm64/purgatory/entry.S
new file mode 100644
index 000000000000..fe6e968076db
--- /dev/null
+++ b/arch/arm64/purgatory/entry.S
@@ -0,0 +1,55 @@
+/*
+ * kexec core purgatory
+ */
+#include <linux/linkage.h>
+#include <uapi/linux/kexec.h>
+
+#define SHA256_DIGEST_SIZE	32 /* defined in crypto/sha.h */
+
+.text
+
+ENTRY(purgatory_start)
+	/* Start new image. */
+	ldr	x17, __kernel_entry
+	ldr	x0, __dtb_addr
+	mov	x1, xzr
+	mov	x2, xzr
+	mov	x3, xzr
+	br	x17
+END(purgatory_start)
+
+/*
+ * data section:
+ * kernel_entry and dtb_addr are global but also labelled as local,
+ * "__xxx:", to avoid unwanted re-linking.
+ *
+ * purgatory_sha_regions and purgatory_sha256_digest are referenced
+ * by kexec generic code and so must exist, but not actually used
+ * here because hash check is not that useful in purgatory.
+ */
+.align 3
+
+.globl kernel_entry
+kernel_entry:
+__kernel_entry:
+	.quad	0
+END(kernel_entry)
+
+.globl dtb_addr
+dtb_addr:
+__dtb_addr:
+	.quad	0
+END(dtb_addr)
+
+.globl purgatory_sha_regions
+purgatory_sha_regions:
+	.rept	KEXEC_SEGMENT_MAX
+	.quad	0
+	.quad	0
+	.endr
+END(purgatory_sha_regions)
+
+.globl purgatory_sha256_digest
+purgatory_sha256_digest:
+        .skip   SHA256_DIGEST_SIZE
+END(purgatory_sha256_digest)
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:56 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:56 +0900
Subject: [PATCH v7 06/11] arm64: kexec_file: load initrd,
 device-tree and purgatory segments
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-7-takahiro.akashi@linaro.org>

load_other_segments() sets up and adds all the memory segments necessary
other than kernel, including initrd, device-tree blob and purgatory.
Most of the code was borrowed from kexec-tools' counterpart.

arch_kimage_kernel_post_load_cleanup() is meant to free arm64-specific data
allocated for loading kernel.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Catalin Marinas <catalin.marinas at arm.com>
Cc: Will Deacon <will.deacon at arm.com>
---
 arch/arm64/include/asm/kexec.h         |  22 ++++
 arch/arm64/kernel/Makefile             |   3 +-
 arch/arm64/kernel/machine_kexec_file.c | 214 +++++++++++++++++++++++++++++++++
 3 files changed, 238 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/kernel/machine_kexec_file.c

diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index e17f0529a882..2fadd3cbf3af 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -93,6 +93,28 @@ static inline void crash_prepare_suspend(void) {}
 static inline void crash_post_resume(void) {}
 #endif
 
+#ifdef CONFIG_KEXEC_FILE
+#define ARCH_HAS_KIMAGE_ARCH
+
+struct kimage_arch {
+	void *dtb_buf;
+};
+
+struct kimage;
+
+#define arch_kimage_file_post_load_cleanup arch_kimage_file_post_load_cleanup
+extern int arch_kimage_file_post_load_cleanup(struct kimage *image);
+
+extern int setup_dtb(struct kimage *image,
+		unsigned long initrd_load_addr, unsigned long initrd_len,
+		char *cmdline, unsigned long cmdline_len,
+		char **dtb_buf, size_t *dtb_buf_len);
+extern int load_other_segments(struct kimage *image,
+		unsigned long kernel_load_addr,
+		char *initrd, unsigned long initrd_len,
+		char *cmdline, unsigned long cmdline_len);
+#endif
+
 #endif /* __ASSEMBLY__ */
 
 #endif
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 067baace74a0..2cd982b779b9 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -47,8 +47,9 @@ arm64-obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL)	+= acpi_parking_protocol.o
 arm64-obj-$(CONFIG_PARAVIRT)		+= paravirt.o
 arm64-obj-$(CONFIG_RANDOMIZE_BASE)	+= kaslr.o
 arm64-obj-$(CONFIG_HIBERNATION)		+= hibernate.o hibernate-asm.o
-arm64-obj-$(CONFIG_KEXEC)		+= machine_kexec.o relocate_kernel.o	\
+arm64-obj-$(CONFIG_KEXEC_CORE)		+= machine_kexec.o relocate_kernel.o	\
 					   cpu-reset.o
+arm64-obj-$(CONFIG_KEXEC_FILE)		+= machine_kexec_file.o
 arm64-obj-$(CONFIG_ARM64_RELOC_TEST)	+= arm64-reloc-test.o
 arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o
 arm64-obj-$(CONFIG_CRASH_DUMP)		+= crash_dump.o
diff --git a/arch/arm64/kernel/machine_kexec_file.c b/arch/arm64/kernel/machine_kexec_file.c
new file mode 100644
index 000000000000..8300d68087d2
--- /dev/null
+++ b/arch/arm64/kernel/machine_kexec_file.c
@@ -0,0 +1,214 @@
+/*
+ * kexec_file for arm64
+ *
+ * Copyright (C) 2017 Linaro Limited
+ * Author: AKASHI Takahiro <takahiro.akashi at linaro.org>
+ *
+ * Most code is derived from arm64 port of kexec-tools
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#define pr_fmt(fmt) "kexec_file: " fmt
+
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/kexec.h>
+#include <linux/libfdt.h>
+#include <linux/memblock.h>
+#include <linux/of_fdt.h>
+
+static int __dt_root_addr_cells;
+static int __dt_root_size_cells;
+
+const struct kexec_file_ops * const kexec_file_loaders[] = {
+	NULL
+};
+
+int arch_kimage_file_post_load_cleanup(struct kimage *image)
+{
+	vfree(image->arch.dtb_buf);
+	image->arch.dtb_buf = NULL;
+
+	return _kimage_file_post_load_cleanup(image);
+}
+
+int arch_kexec_walk_mem(struct kexec_buf *kbuf,
+				int (*func)(struct resource *, void *))
+{
+	if (kbuf->image->type == KEXEC_TYPE_CRASH)
+		return walk_iomem_res_desc(crashk_res.desc,
+					IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY,
+					crashk_res.start, crashk_res.end,
+					kbuf, func);
+	else if (kbuf->top_down)
+		return walk_system_ram_res_rev(0, ULONG_MAX, kbuf, func);
+	else
+		return walk_system_ram_res(0, ULONG_MAX, kbuf, func);
+}
+
+int setup_dtb(struct kimage *image,
+		unsigned long initrd_load_addr, unsigned long initrd_len,
+		char *cmdline, unsigned long cmdline_len,
+		char **dtb_buf, size_t *dtb_buf_len)
+{
+	char *buf = NULL;
+	size_t buf_size;
+	int nodeoffset;
+	u64 value;
+	int range_len;
+	int ret;
+
+	/* duplicate dt blob */
+	buf_size = fdt_totalsize(initial_boot_params);
+	range_len = (__dt_root_addr_cells + __dt_root_size_cells) * sizeof(u32);
+
+	if (initrd_load_addr)
+		buf_size += fdt_prop_len("initrd-start", sizeof(u64))
+				+ fdt_prop_len("initrd-end", sizeof(u64));
+
+	if (cmdline)
+		buf_size += fdt_prop_len("bootargs", cmdline_len + 1);
+
+	buf = vmalloc(buf_size);
+	if (!buf) {
+		ret = -ENOMEM;
+		goto out_err;
+	}
+
+	ret = fdt_open_into(initial_boot_params, buf, buf_size);
+	if (ret)
+		goto out_err;
+
+	nodeoffset = fdt_path_offset(buf, "/chosen");
+	if (nodeoffset < 0)
+		goto out_err;
+
+	/* add bootargs */
+	if (cmdline) {
+		ret = fdt_setprop(buf, nodeoffset, "bootargs",
+						cmdline, cmdline_len + 1);
+		if (ret)
+			goto out_err;
+	}
+
+	/* add initrd-* */
+	if (initrd_load_addr) {
+		value = cpu_to_fdt64(initrd_load_addr);
+		ret = fdt_setprop(buf, nodeoffset, "initrd-start",
+				&value, sizeof(value));
+		if (ret)
+			goto out_err;
+
+		value = cpu_to_fdt64(initrd_load_addr + initrd_len);
+		ret = fdt_setprop(buf, nodeoffset, "initrd-end",
+				&value, sizeof(value));
+		if (ret)
+			goto out_err;
+	}
+
+	/* trim a buffer */
+	fdt_pack(buf);
+	*dtb_buf = buf;
+	*dtb_buf_len = fdt_totalsize(buf);
+
+	return 0;
+
+out_err:
+	vfree(buf);
+	return ret;
+}
+
+int load_other_segments(struct kimage *image, unsigned long kernel_load_addr,
+			char *initrd, unsigned long initrd_len,
+			char *cmdline, unsigned long cmdline_len)
+{
+	struct kexec_buf kbuf;
+	unsigned long initrd_load_addr = 0;
+	unsigned long purgatory_load_addr, dtb_load_addr;
+	char *dtb = NULL;
+	unsigned long dtb_len;
+	int ret = 0;
+
+	kbuf.image = image;
+	/* not allocate anything below the kernel */
+	kbuf.buf_min = kernel_load_addr;
+
+	/* Load initrd */
+	if (initrd) {
+		kbuf.buffer = initrd;
+		kbuf.bufsz = initrd_len;
+		kbuf.memsz = initrd_len;
+		kbuf.buf_align = PAGE_SIZE;
+		/* within 1GB-aligned window of up to 32GB in size */
+		kbuf.buf_max = round_down(kernel_load_addr, SZ_1G)
+						+ (unsigned long)SZ_1G * 31;
+		kbuf.top_down = 0;
+
+		ret = kexec_add_buffer(&kbuf);
+		if (ret)
+			goto out_err;
+		initrd_load_addr = kbuf.mem;
+
+		pr_debug("Loaded initrd at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+				initrd_load_addr, initrd_len, initrd_len);
+	}
+
+	/* Load dtb blob */
+	ret = setup_dtb(image, initrd_load_addr, initrd_len,
+				cmdline, cmdline_len, &dtb, &dtb_len);
+	if (ret) {
+		pr_err("Preparing for new dtb failed\n");
+		goto out_err;
+	}
+
+	kbuf.buffer = dtb;
+	kbuf.bufsz = dtb_len;
+	kbuf.memsz = dtb_len;
+	/* not across 2MB boundary */
+	kbuf.buf_align = SZ_2M;
+	kbuf.buf_max = ULONG_MAX;
+	kbuf.top_down = 1;
+
+	ret = kexec_add_buffer(&kbuf);
+	if (ret)
+		goto out_err;
+	dtb_load_addr = kbuf.mem;
+	image->arch.dtb_buf = dtb;
+
+	pr_debug("Loaded dtb at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+			dtb_load_addr, dtb_len, dtb_len);
+
+	/* Load purgatory  */
+	ret = kexec_load_purgatory(image, kernel_load_addr, ULONG_MAX, 1,
+				   &purgatory_load_addr);
+	if (ret) {
+		pr_err("Loading purgatory failed\n");
+		goto out_err;
+	}
+
+	ret = kexec_purgatory_get_set_symbol(image, "kernel_entry",
+				&kernel_load_addr, sizeof(kernel_load_addr), 0);
+	if (ret) {
+		pr_err("Setting symbol (kernel_entry) failed.\n");
+		goto out_err;
+	}
+
+	ret = kexec_purgatory_get_set_symbol(image, "dtb_addr",
+				&dtb_load_addr, sizeof(dtb_load_addr), 0);
+	if (ret) {
+		pr_err("Setting symbol (dtb_addr) failed.\n");
+		goto out_err;
+	}
+
+	pr_debug("Loaded purgatory at 0x%lx\n", purgatory_load_addr);
+
+	return 0;
+
+out_err:
+	vfree(dtb);
+	image->arch.dtb_buf = NULL;
+	return ret;
+}
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:58:00 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:58:00 +0900
Subject: [PATCH v7 10/11] include: pe.h: remove message[] from mz header
 definition
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-11-takahiro.akashi@linaro.org>

message[] field won't be part of the definition of mz header.

This change is crucial for enabling kexec_file_load on arm64 because
arm64's "Image" binary, as in PE format, doesn't have any data for it and
accordingly the following check in pefile_parse_binary() will fail:

	chkaddr(cursor, mz->peaddr, sizeof(*pe));

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Reviewed-by: Ard Biesheuvel <ard.biesheuvel at linaro.org>
Cc: David Howells <dhowells at redhat.com>
Cc: Vivek Goyal <vgoyal at redhat.com>
Cc: Herbert Xu <herbert at gondor.apana.org.au>
Cc: David S. Miller <davem at davemloft.net>
---
 include/linux/pe.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/pe.h b/include/linux/pe.h
index 143ce75be5f0..3482b18a48b5 100644
--- a/include/linux/pe.h
+++ b/include/linux/pe.h
@@ -166,7 +166,7 @@ struct mz_hdr {
 	uint16_t oem_info;	/* oem specific */
 	uint16_t reserved1[10];	/* reserved */
 	uint32_t peaddr;	/* address of pe header */
-	char     message[64];	/* message to print */
+	char     message[];	/* message to print */
 };
 
 struct mz_reloc {
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:57 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:57 +0900
Subject: [PATCH v7 07/11] arm64: kexec_file: set up for crash dump adding elf
 core header
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-8-takahiro.akashi@linaro.org>

load_crashdump_segments() creates and loads a memory segment of elf core
header for crash dump.

"linux,usable-memory-range" and "linux,elfcorehdr" will add to the 2nd
kernel's device-tree blob. The logic of this cod is also from kexec-tools.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Catalin Marinas <catalin.marinas at arm.com>
Cc: Will Deacon <will.deacon at arm.com>
---
 arch/arm64/include/asm/kexec.h         |   5 ++
 arch/arm64/kernel/machine_kexec_file.c | 151 +++++++++++++++++++++++++++++++++
 kernel/kexec_file.c                    |   2 +-
 3 files changed, 157 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 2fadd3cbf3af..edb702e64a8a 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -98,6 +98,10 @@ static inline void crash_post_resume(void) {}
 
 struct kimage_arch {
 	void *dtb_buf;
+	/* Core ELF header buffer */
+	void *elf_headers;
+	unsigned long elf_headers_sz;
+	unsigned long elf_load_addr;
 };
 
 struct kimage;
@@ -113,6 +117,7 @@ extern int load_other_segments(struct kimage *image,
 		unsigned long kernel_load_addr,
 		char *initrd, unsigned long initrd_len,
 		char *cmdline, unsigned long cmdline_len);
+extern int load_crashdump_segments(struct kimage *image);
 #endif
 
 #endif /* __ASSEMBLY__ */
diff --git a/arch/arm64/kernel/machine_kexec_file.c b/arch/arm64/kernel/machine_kexec_file.c
index 8300d68087d2..012b2af4e27b 100644
--- a/arch/arm64/kernel/machine_kexec_file.c
+++ b/arch/arm64/kernel/machine_kexec_file.c
@@ -19,6 +19,7 @@
 #include <linux/libfdt.h>
 #include <linux/memblock.h>
 #include <linux/of_fdt.h>
+#include <linux/vmalloc.h>
 
 static int __dt_root_addr_cells;
 static int __dt_root_size_cells;
@@ -32,6 +33,10 @@ int arch_kimage_file_post_load_cleanup(struct kimage *image)
 	vfree(image->arch.dtb_buf);
 	image->arch.dtb_buf = NULL;
 
+	vfree(image->arch.elf_headers);
+	image->arch.elf_headers = NULL;
+	image->arch.elf_headers_sz = 0;
+
 	return _kimage_file_post_load_cleanup(image);
 }
 
@@ -49,6 +54,78 @@ int arch_kexec_walk_mem(struct kexec_buf *kbuf,
 		return walk_system_ram_res(0, ULONG_MAX, kbuf, func);
 }
 
+static int __init arch_kexec_file_init(void)
+{
+	/* Those values are used later on loading the kernel */
+	__dt_root_addr_cells = dt_root_addr_cells;
+	__dt_root_size_cells = dt_root_size_cells;
+
+	return 0;
+}
+late_initcall(arch_kexec_file_init);
+
+#define FDT_ALIGN(x, a)	(((x) + (a) - 1) & ~((a) - 1))
+#define FDT_TAGALIGN(x)	(FDT_ALIGN((x), FDT_TAGSIZE))
+
+static int fdt_prop_len(const char *prop_name, int len)
+{
+	return (strlen(prop_name) + 1) +
+		sizeof(struct fdt_property) +
+		FDT_TAGALIGN(len);
+}
+
+static bool cells_size_fitted(unsigned long base, unsigned long size)
+{
+	/* if *_cells >= 2, cells can hold 64-bit values anyway */
+	if ((__dt_root_addr_cells == 1) && (base >= (1ULL << 32)))
+		return false;
+
+	if ((__dt_root_size_cells == 1) && (size >= (1ULL << 32)))
+		return false;
+
+	return true;
+}
+
+static void fill_property(void *buf, u64 val64, int cells)
+{
+	u32 val32;
+
+	if (cells == 1) {
+		val32 = cpu_to_fdt32((u32)val64);
+		memcpy(buf, &val32, sizeof(val32));
+	} else {
+		memset(buf, 0, cells * sizeof(u32) - sizeof(u64));
+		buf += cells * sizeof(u32) - sizeof(u64);
+
+		val64 = cpu_to_fdt64(val64);
+		memcpy(buf, &val64, sizeof(val64));
+	}
+}
+
+static int fdt_setprop_range(void *fdt, int nodeoffset, const char *name,
+				unsigned long addr, unsigned long size)
+{
+	void *buf, *prop;
+	size_t buf_size;
+	int result;
+
+	buf_size = (__dt_root_addr_cells + __dt_root_size_cells) * sizeof(u32);
+	prop = buf = vmalloc(buf_size);
+	if (!buf)
+		return -ENOMEM;
+
+	fill_property(prop, addr, __dt_root_addr_cells);
+	prop += __dt_root_addr_cells * sizeof(u32);
+
+	fill_property(prop, size, __dt_root_size_cells);
+
+	result = fdt_setprop(fdt, nodeoffset, name, buf, buf_size);
+
+	vfree(buf);
+
+	return result;
+}
+
 int setup_dtb(struct kimage *image,
 		unsigned long initrd_load_addr, unsigned long initrd_len,
 		char *cmdline, unsigned long cmdline_len,
@@ -61,10 +138,26 @@ int setup_dtb(struct kimage *image,
 	int range_len;
 	int ret;
 
+	/* check ranges against root's #address-cells and #size-cells */
+	if (image->type == KEXEC_TYPE_CRASH &&
+		(!cells_size_fitted(image->arch.elf_load_addr,
+				image->arch.elf_headers_sz) ||
+		 !cells_size_fitted(crashk_res.start,
+				crashk_res.end - crashk_res.start + 1))) {
+		pr_err("Crash memory region doesn't fit into DT's root cell sizes.\n");
+		ret = -EINVAL;
+		goto out_err;
+	}
+
 	/* duplicate dt blob */
 	buf_size = fdt_totalsize(initial_boot_params);
 	range_len = (__dt_root_addr_cells + __dt_root_size_cells) * sizeof(u32);
 
+	if (image->type == KEXEC_TYPE_CRASH)
+		buf_size += fdt_prop_len("linux,elfcorehdr", range_len)
+				+ fdt_prop_len("linux,usable-memory-range",
+								range_len);
+
 	if (initrd_load_addr)
 		buf_size += fdt_prop_len("initrd-start", sizeof(u64))
 				+ fdt_prop_len("initrd-end", sizeof(u64));
@@ -86,6 +179,23 @@ int setup_dtb(struct kimage *image,
 	if (nodeoffset < 0)
 		goto out_err;
 
+	if (image->type == KEXEC_TYPE_CRASH) {
+		/* add linux,elfcorehdr */
+		ret = fdt_setprop_range(buf, nodeoffset, "linux,elfcorehdr",
+				image->arch.elf_load_addr,
+				image->arch.elf_headers_sz);
+		if (ret)
+			goto out_err;
+
+		/* add linux,usable-memory-range */
+		ret = fdt_setprop_range(buf, nodeoffset,
+				"linux,usable-memory-range",
+				crashk_res.start,
+				crashk_res.end - crashk_res.start + 1);
+		if (ret)
+			goto out_err;
+	}
+
 	/* add bootargs */
 	if (cmdline) {
 		ret = fdt_setprop(buf, nodeoffset, "bootargs",
@@ -212,3 +322,44 @@ int load_other_segments(struct kimage *image, unsigned long kernel_load_addr,
 	image->arch.dtb_buf = NULL;
 	return ret;
 }
+
+int load_crashdump_segments(struct kimage *image)
+{
+	void *elf_addr;
+	unsigned long elf_sz;
+	struct kexec_buf kbuf;
+	int ret;
+
+	if (image->type != KEXEC_TYPE_CRASH)
+		return 0;
+
+	/* Prepare elf headers and add a segment */
+	ret = prepare_elf_headers(image, &elf_addr, &elf_sz);
+	if (ret) {
+		pr_err("Preparing elf core header failed\n");
+		return ret;
+	}
+
+	kbuf.image = image;
+	kbuf.buffer = elf_addr;
+	kbuf.bufsz = elf_sz;
+	kbuf.memsz = elf_sz;
+	kbuf.buf_align = PAGE_SIZE;
+	kbuf.buf_min = crashk_res.start;
+	kbuf.buf_max = crashk_res.end + 1;
+	kbuf.top_down = 1;
+
+	ret = kexec_add_buffer(&kbuf);
+	if (ret) {
+		vfree(elf_addr);
+		return ret;
+	}
+	image->arch.elf_headers = elf_addr;
+	image->arch.elf_headers_sz = elf_sz;
+	image->arch.elf_load_addr = kbuf.mem;
+
+	pr_debug("Loaded elf core header at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+			 image->arch.elf_load_addr, elf_sz, elf_sz);
+
+	return ret;
+}
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index b2c425e6afc6..0ec7181d16d9 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -1337,7 +1337,7 @@ static int prepare_elf64_headers(struct crash_elf_data *ced,
 	phdr->p_filesz = phdr->p_memsz = VMCOREINFO_NOTE_SIZE;
 	(ehdr->e_phnum)++;
 
-#ifdef CONFIG_X86_64
+#if defined(CONFIG_X86_64) || defined(CONFIG_ARM64)
 	/* Prepare PT_LOAD type program header for kernel text region */
 	phdr = (Elf64_Phdr *)bufp;
 	bufp += sizeof(Elf64_Phdr);
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:58:01 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:58:01 +0900
Subject: [PATCH v7 11/11] arm64: kexec_file: enable KEXEC_VERIFY_SIG for Image
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-12-takahiro.akashi@linaro.org>

With this patch, kernel verification can be done without IMA security
subsystem enabled. Turn on CONFIG_KEXEC_VERIFY_SIG instead.

On x86, a signature is embedded into a PE file (Microsoft's format) header
of binary. Since arm64's "Image" can also be seen as a PE file as far as
CONFIG_EFI is enabled, we adopt this format for kernel signing.

You can create a signed kernel image with:
    $ sbsign --key ${KEY} --cert ${CERT} Image

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Catalin Marinas <catalin.marinas at arm.com>
Cc: Will Deacon <will.deacon at arm.com>
---
 arch/arm64/Kconfig              | 10 ++++++++++
 arch/arm64/include/asm/kexec.h  | 16 ++++++++++++++++
 arch/arm64/kernel/kexec_image.c | 15 +++++++++++++++
 3 files changed, 41 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index c0b021736c10..289c7bede593 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -814,6 +814,16 @@ config KEXEC_FILE_IMAGE_FMT
 	---help---
 	  Select this option to enable 'Image' kernel loading.
 
+config KEXEC_VERIFY_SIG
+	bool "Verify kernel signature during kexec_file_load() syscall"
+	depends on KEXEC_FILE
+	select SYSTEM_DATA_VERIFICATION
+	select SIGNED_PE_FILE_VERIFICATION if KEXEC_FILE_IMAGE_FMT
+	---help---
+	  Select this option to verify a signature with loaded kernel
+	  image. If configured, any attempt of loading a image without
+	  valid signature will fail.
+
 config CRASH_DUMP
 	bool "Build kdump crash kernel"
 	help
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index b6469bc64a89..2a63bf5f32ea 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -131,6 +131,7 @@ struct arm64_image_header {
 };
 
 static const u8 arm64_image_magic[4] = {'A', 'R', 'M', 0x64U};
+static const u8 arm64_image_pe_sig[2] = {'M', 'Z'};
 
 /**
  * arm64_header_check_magic - Helper to check the arm64 image header.
@@ -152,6 +153,21 @@ static inline int arm64_header_check_magic(const struct arm64_image_header *h)
 		&& h->magic[3] == arm64_image_magic[3]);
 }
 
+/**
+ * arm64_header_check_pe_sig - Helper to check the arm64 image header.
+ *
+ * Returns non-zero if 'MZ' signature is found.
+ */
+
+static inline int arm64_header_check_pe_sig(const struct arm64_image_header *h)
+{
+	if (!h)
+		return 0;
+
+	return (h->pe_sig[0] == arm64_image_pe_sig[0]
+		&& h->pe_sig[1] == arm64_image_pe_sig[1]);
+}
+
 extern const struct kexec_file_ops kexec_image_ops;
 
 struct kimage;
diff --git a/arch/arm64/kernel/kexec_image.c b/arch/arm64/kernel/kexec_image.c
index 1c106237901d..b840b6ed6ed9 100644
--- a/arch/arm64/kernel/kexec_image.c
+++ b/arch/arm64/kernel/kexec_image.c
@@ -15,6 +15,7 @@
 #include <linux/errno.h>
 #include <linux/kernel.h>
 #include <linux/kexec.h>
+#include <linux/verification.h>
 #include <asm/byteorder.h>
 #include <asm/memory.h>
 
@@ -27,6 +28,9 @@ static int image_probe(const char *kernel_buf, unsigned long kernel_len)
 	if ((kernel_len < sizeof(*h)) || !arm64_header_check_magic(h))
 		return -EINVAL;
 
+	pr_debug("PE format: %s\n",
+			(arm64_header_check_pe_sig(h) ? "yes" : "no"));
+
 	return 0;
 }
 
@@ -84,7 +88,18 @@ static void *image_load(struct kimage *image, char *kernel,
 	return ERR_PTR(ret);
 }
 
+#ifdef CONFIG_KEXEC_VERIFY_SIG
+static int image_verify_sig(const char *kernel, unsigned long kernel_len)
+{
+	return verify_pefile_signature(kernel, kernel_len, NULL,
+				       VERIFYING_KEXEC_PE_SIGNATURE);
+}
+#endif
+
 const struct kexec_file_ops kexec_image_ops = {
 	.probe = image_probe,
 	.load = image_load,
+#ifdef CONFIG_KEXEC_VERIFY_SIG
+	.verify_sig = image_verify_sig,
+#endif
 };
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:59 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:59 +0900
Subject: [PATCH v7 09/11] arm64: kexec_file: add Image format support
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-10-takahiro.akashi@linaro.org>

The "Image" binary will be loaded at the offset of TEXT_OFFSET from
the start of system memory. TEXT_OFFSET is determined from the header
of the image.

This patch doesn't have CONFIG_KEXEC_VERIFY_SIG support. Nevertherless
kernel verification will be supported by enabling IMA security subsystem.
See more details about IMA here:
    https://sourceforge.net/p/linux-ima/wiki/Home/

You can sign(label) a kernel image binary to be kexec-ed on target
filesystem with:
    $ evmctl ima_sign --key /path/to/private_key.pem Image

On live system, you must have IMA enforced with, at least, the following
security policy:
    "appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig"

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Catalin Marinas <catalin.marinas at arm.com>
Cc: Will Deacon <will.deacon at arm.com>
---
 arch/arm64/Kconfig                     |  7 +++
 arch/arm64/include/asm/kexec.h         | 50 +++++++++++++++++++
 arch/arm64/kernel/Makefile             |  1 +
 arch/arm64/kernel/kexec_image.c        | 90 ++++++++++++++++++++++++++++++++++
 arch/arm64/kernel/machine_kexec_file.c |  3 ++
 5 files changed, 151 insertions(+)
 create mode 100644 arch/arm64/kernel/kexec_image.c

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 865d110809f9..c0b021736c10 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -807,6 +807,13 @@ config KEXEC_FILE
 	  In addition to this option, you need to enable a specific type
 	  of image support.
 
+config KEXEC_FILE_IMAGE_FMT
+	bool "Enable Image support"
+	default y
+	depends on KEXEC_FILE
+	---help---
+	  Select this option to enable 'Image' kernel loading.
+
 config CRASH_DUMP
 	bool "Build kdump crash kernel"
 	help
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index edb702e64a8a..b6469bc64a89 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -104,6 +104,56 @@ struct kimage_arch {
 	unsigned long elf_load_addr;
 };
 
+/**
+ * struct arm64_image_header - arm64 kernel image header
+ *
+ * @pe_sig: Optional PE format 'MZ' signature
+ * @branch_code: Instruction to branch to stext
+ * @text_offset: Image load offset, little endian
+ * @image_size: Effective image size, little endian
+ * @flags:
+ *	Bit 0: Kernel endianness. 0=little endian, 1=big endian
+ * @reserved: Reserved
+ * @magic: Magic number, "ARM\x64"
+ * @pe_header: Optional offset to a PE format header
+ **/
+
+struct arm64_image_header {
+	u8 pe_sig[2];
+	u8 pad[2];
+	u32 branch_code;
+	u64 text_offset;
+	u64 image_size;
+	u64 flags;
+	u64 reserved[3];
+	u8 magic[4];
+	u32 pe_header;
+};
+
+static const u8 arm64_image_magic[4] = {'A', 'R', 'M', 0x64U};
+
+/**
+ * arm64_header_check_magic - Helper to check the arm64 image header.
+ *
+ * Returns non-zero if header is OK.
+ */
+
+static inline int arm64_header_check_magic(const struct arm64_image_header *h)
+{
+	if (!h)
+		return 0;
+
+	if (!h->text_offset)
+		return 0;
+
+	return (h->magic[0] == arm64_image_magic[0]
+		&& h->magic[1] == arm64_image_magic[1]
+		&& h->magic[2] == arm64_image_magic[2]
+		&& h->magic[3] == arm64_image_magic[3]);
+}
+
+extern const struct kexec_file_ops kexec_image_ops;
+
 struct kimage;
 
 #define arch_kimage_file_post_load_cleanup arch_kimage_file_post_load_cleanup
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 2cd982b779b9..17906a62d795 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -50,6 +50,7 @@ arm64-obj-$(CONFIG_HIBERNATION)		+= hibernate.o hibernate-asm.o
 arm64-obj-$(CONFIG_KEXEC_CORE)		+= machine_kexec.o relocate_kernel.o	\
 					   cpu-reset.o
 arm64-obj-$(CONFIG_KEXEC_FILE)		+= machine_kexec_file.o
+arm64-obj-$(CONFIG_KEXEC_FILE_IMAGE_FMT)	+= kexec_image.o
 arm64-obj-$(CONFIG_ARM64_RELOC_TEST)	+= arm64-reloc-test.o
 arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o
 arm64-obj-$(CONFIG_CRASH_DUMP)		+= crash_dump.o
diff --git a/arch/arm64/kernel/kexec_image.c b/arch/arm64/kernel/kexec_image.c
new file mode 100644
index 000000000000..1c106237901d
--- /dev/null
+++ b/arch/arm64/kernel/kexec_image.c
@@ -0,0 +1,90 @@
+/*
+ * Kexec image loader
+
+ * Copyright (C) 2017 Linaro Limited
+ * Author: AKASHI Takahiro <takahiro.akashi at linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#define pr_fmt(fmt)	"kexec_file(Image): " fmt
+
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/kexec.h>
+#include <asm/byteorder.h>
+#include <asm/memory.h>
+
+static int image_probe(const char *kernel_buf, unsigned long kernel_len)
+{
+	const struct arm64_image_header *h;
+
+	h = (const struct arm64_image_header *)(kernel_buf);
+
+	if ((kernel_len < sizeof(*h)) || !arm64_header_check_magic(h))
+		return -EINVAL;
+
+	return 0;
+}
+
+static void *image_load(struct kimage *image, char *kernel,
+			    unsigned long kernel_len, char *initrd,
+			    unsigned long initrd_len, char *cmdline,
+			    unsigned long cmdline_len)
+{
+	struct kexec_buf kbuf;
+	struct arm64_image_header *h = (struct arm64_image_header *)kernel;
+	unsigned long text_offset, kernel_load_addr;
+	int ret;
+
+	/* Create elf core header segment */
+	ret = load_crashdump_segments(image);
+	if (ret)
+		goto out;
+
+	/* Load the kernel */
+	kbuf.image = image;
+	if (image->type == KEXEC_TYPE_CRASH) {
+		kbuf.buf_min = crashk_res.start;
+		kbuf.buf_max = crashk_res.end + 1;
+	} else {
+		kbuf.buf_min = 0;
+		kbuf.buf_max = ULONG_MAX;
+	}
+	kbuf.top_down = 0;
+
+	kbuf.buffer = kernel;
+	kbuf.bufsz = kernel_len;
+	kbuf.memsz = le64_to_cpu(h->image_size);
+	text_offset = le64_to_cpu(h->text_offset);
+	kbuf.buf_align = SZ_2M;
+
+	/* Adjust kernel segment with TEXT_OFFSET */
+	kbuf.memsz += text_offset;
+
+	ret = kexec_add_buffer(&kbuf);
+	if (ret)
+		goto out;
+
+	image->segment[image->nr_segments - 1].mem += text_offset;
+	image->segment[image->nr_segments - 1].memsz -= text_offset;
+	kernel_load_addr = kbuf.mem + text_offset;
+
+	pr_debug("Loaded kernel at 0x%lx bufsz=0x%lx memsz=0x%lx\n",
+		 kernel_load_addr, kbuf.bufsz, kbuf.memsz);
+
+	/* Load additional data */
+	ret = load_other_segments(image, kernel_load_addr,
+			    initrd, initrd_len, cmdline, cmdline_len);
+
+out:
+	return ERR_PTR(ret);
+}
+
+const struct kexec_file_ops kexec_image_ops = {
+	.probe = image_probe,
+	.load = image_load,
+};
diff --git a/arch/arm64/kernel/machine_kexec_file.c b/arch/arm64/kernel/machine_kexec_file.c
index 012b2af4e27b..2b6d9164df8a 100644
--- a/arch/arm64/kernel/machine_kexec_file.c
+++ b/arch/arm64/kernel/machine_kexec_file.c
@@ -25,6 +25,9 @@ static int __dt_root_addr_cells;
 static int __dt_root_size_cells;
 
 const struct kexec_file_ops * const kexec_file_loaders[] = {
+#ifdef CONFIG_KEXEC_FILE_IMAGE_FMT
+	&kexec_image_ops,
+#endif
 	NULL
 };
 
-- 
2.14.1



From takahiro.akashi at linaro.org  Sun Dec  3 18:57:58 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon,  4 Dec 2017 11:57:58 +0900
Subject: [PATCH v7 08/11] arm64: kexec_file: enable KEXEC_FILE config
In-Reply-To: <20171204025801.12161-1-takahiro.akashi@linaro.org>
References: <20171204025801.12161-1-takahiro.akashi@linaro.org>
Message-ID: <20171204025801.12161-9-takahiro.akashi@linaro.org>

Modify arm64/Kconfig and Makefile to enable kexec_file_load support.
File-format specific hook functions to load a kernel image will
follow this patch.

Signed-off-by: AKASHI Takahiro <takahiro.akashi at linaro.org>
Cc: Catalin Marinas <catalin.marinas at arm.com>
Cc: Will Deacon <will.deacon at arm.com>
---
 arch/arm64/Kconfig | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index a93339f5178f..865d110809f9 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -794,6 +794,19 @@ config KEXEC
 	  but it is independent of the system firmware.   And like a reboot
 	  you can start any kernel with it, not just Linux.
 
+config KEXEC_FILE
+	bool "kexec file based system call"
+	select KEXEC_CORE
+	select BUILD_BIN2C
+	---help---
+	  This is new version of kexec system call. This system call is
+	  file based and takes file descriptors as system call argument
+	  for kernel and initramfs as opposed to list of segments as
+	  accepted by previous system call.
+
+	  In addition to this option, you need to enable a specific type
+	  of image support.
+
 config CRASH_DUMP
 	bool "Build kdump crash kernel"
 	help
-- 
2.14.1



From bhsharma at redhat.com  Mon Dec  4 11:59:50 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Tue, 5 Dec 2017 01:29:50 +0530
Subject: kdump issues with 4.11 kernel
In-Reply-To: <BY2PR0701MB202353AA68A956CA64516B808E380@BY2PR0701MB2023.namprd07.prod.outlook.com>
References: <BY2PR0701MB202342812CFB9669B01930D98E3B0@BY2PR0701MB2023.namprd07.prod.outlook.com>
 <20171129073841.GA2129@dhcp-128-65.nay.redhat.com>
 <BY2PR0701MB2023DAF450CE2EB7270E05A38E3B0@BY2PR0701MB2023.namprd07.prod.outlook.com>
 <CACi5LpP9+5YVVh1DtnWXCMQbFMRJ1NVYh=_nQYcpShZS0es7hg@mail.gmail.com>
 <BY2PR0701MB2023FFA00EC7445B005BCB498E3B0@BY2PR0701MB2023.namprd07.prod.outlook.com>
 <CACi5LpNVRpHJW-AUuNnH0YQ-dTxD+fGjMAO_pL=2ieqzkG0mgw@mail.gmail.com>
 <BY2PR0701MB202353AA68A956CA64516B808E380@BY2PR0701MB2023.namprd07.prod.outlook.com>
Message-ID: <CACi5LpMVpGo7guKs7kscJk0yyu-tohx6nuwQ1F_o+s867DsepA@mail.gmail.com>

Hello Anil,

On Thu, Nov 30, 2017 at 5:59 PM, Gurumurthy, Anil
<Anil.Gurumurthy at cavium.com> wrote:
> Hi Bhupesh,
>   I tried to get some log messages for the crash kernel, but unable to get anything.
> echo c > /proc/sysrq-trigger
> simply hangs w/o any messages on the console.

Did you tree to set earlycon or earlyprintk in the bootargs. Something
like this:

earlycon=pl011,mmio32,0xff78ed1000

depending on the underlying uart device you have on the board. For
e.g. here I assumed a pl011 uart is used to display console messages.

Regards,
Bhupesh

>
> Thanks,
> Anil
> -----Original Message-----
> From: Gurumurthy, Anil
> Sent: 29 November 2017 16:02
> To: 'Bhupesh Sharma' <bhsharma at redhat.com>
> Cc: Dave Young <dyoung at redhat.com>; kexec at lists.infradead.org
> Subject: RE: kdump issues with 4.11 kernel
>
>
>
> -----Original Message-----
> From: Bhupesh Sharma [mailto:bhsharma at redhat.com]
> Sent: 29 November 2017 15:50
> To: Gurumurthy, Anil <Anil.Gurumurthy at cavium.com>
> Cc: Dave Young <dyoung at redhat.com>; kexec at lists.infradead.org
> Subject: Re: kdump issues with 4.11 kernel
>
> On Wed, Nov 29, 2017 at 3:36 PM, Gurumurthy, Anil <Anil.Gurumurthy at cavium.com> wrote:
>>
>>
>> -----Original Message-----
>> From: Bhupesh Sharma [mailto:bhsharma at redhat.com]
>> Sent: 29 November 2017 15:16
>> To: Gurumurthy, Anil <Anil.Gurumurthy at cavium.com>
>> Cc: Dave Young <dyoung at redhat.com>; kexec at lists.infradead.org
>> Subject: Re: kdump issues with 4.11 kernel
>>
>> Hi Anil,
>>
>> On Wed, Nov 29, 2017 at 2:44 PM, Gurumurthy, Anil <Anil.Gurumurthy at cavium.com> wrote:
>>> Thanks. That did help getting kexec to work.
>>> However I still do not get a crash dump - echo c  >
>>> /proc/sysrq-trigger does not get a crash dump.
>>>
>>> Any thoughts?
>>
>> Cam you share the console messages you see when the crash kernel
>> boots? Or do you see nothing after the crash is introduced via echo c
>>> /proc/sysrq-trigger
>> [Anil]  I do not see any messages after introducing the crash.
>
> There could be several reasons for this:
>
> - crashkernel might be missing some arch/machine specific options.
> - It may be that the purgatory sha verification has failed. If your arch supports a console in purgatory then it is easy to debug this.
> - It might be that the crash kernel itself crashed very early. Pass some earlycon/earlyprintk option for your system to the second kernel command line.
> - Also please share relevant dmesg log of both primary kernel boot and the commands you use to invoke the crashkernel.
>
> [Anil] Thanks for the quick response
>
>    This is what I have in the .config (for the primary kernel) CONFIG_EARLY_PRINTK=y CONFIG_EARLY_PRINTK_DBGP=y CONFIG_EARLY_PRINTK_EFI=y
>
> The log for the primary kernel boot:
>
> Nov 29 14:22:22 localhost journal: Runtime journal is using 8.0M (max 1.9G, leaving 2.9G of free 19.4G, current limit 1.9G).
> Nov 29 14:22:22 localhost kernel: Linux version 4.11.12+ (root at localhost.localdomain) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #4 SMP Thu Nov 23 12:11:02 IST 2017 Nov 29 14:22:22 localhost kernel: Command line: BOOT_IMAGE=/vmlinuz-4.11.12+ root=/dev/mapper/rhel-root ro rd.lvm.lv=rhel/swap crashkernel=128M rd.lvm.lv=rhel/root rhgb quiet Nov 29 14:22:22 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
> Nov 29 14:22:22 localhost kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
> Nov 29 14:22:22 localhost kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
> Nov 29 14:22:22 localhost kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256 Nov 29 14:22:22 localhost kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
> Nov 29 14:22:22 localhost kernel: e820: BIOS-provided physical RAM map:
> Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000006bfff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000000006c000-0x000000000006cfff] ACPI NVS Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000000006d000-0x000000000009efff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] ACPI NVS Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x0000000000100000-0x000000005d184fff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000005d185000-0x000000005d185fff] ACPI data Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000005d186000-0x000000005fb77fff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000005fb78000-0x000000005fb7cfff] reserved Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000005fb7d000-0x000000005ffdffff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000005ffe0000-0x000000005ffe1fff] ACPI data Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000005ffe2000-0x000000005fffafff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000005fffb000-0x0000000060001fff] ACPI data Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x0000000060002000-0x0000000060009fff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000006000a000-0x000000006000efff] ACPI data Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x000000006000f000-0x000000006000ffff] usable Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x0000000060010000-0x0000000060011fff] ACPI data Nov 29 14:22:22 localhost kernel: BIOS-e820: [mem 0x0000000060012000-0x00000000600ddfff] usable
>
>
> Will try to get the other details you needed too.
>
> -Anil
>
> Regards,
> Bhupesh
>
>
>> Generally, depending on your test machine arch, it is useful to use earlycon/earlyprintk to see if the crash kernel produced any useful message until the actual console device became operational.
>>
>> Can you try setting the earlycon/earlyprintk settings and share the crash kernel logs messages after the same?
>>
>> Thanks,
>> Bhupesh
>>
>>> -----Original Message-----
>>> From: Dave Young [mailto:dyoung at redhat.com]
>>> Sent: 29 November 2017 13:09
>>> To: Gurumurthy, Anil <Anil.Gurumurthy at cavium.com>
>>> Cc: kexec at lists.infradead.org
>>> Subject: Re: kdump issues with 4.11 kernel
>>>
>>> Hi,
>>> On 11/29/17 at 05:29am, Gurumurthy, Anil wrote:
>>>> Hello,
>>>>   I was facing trouble getting a crash dump on 4.11 kernel. Debugging a bit, I see that the kexec run from the cmd line fails. Any ideas on what I could be missing?
>>>>
>>>> [root at localhost ~]# kexec -p /boot/vmlinuz-`uname -r`
>>>> --initrd=/boot/initramfs-`uname -r`kdump.img ELF core (kcore) parse
>>>> failed Cannot load /boot/vmlinuz-4.11.12+
>>>>
>>>
>>> Can you try below kexec-tools commit:
>>> commit ed15ba1b9977e506637ff1697821d97127b2c919
>>> Author: Pratyush Anand <panand at redhat.com>
>>> Date:   Wed Mar 1 11:19:42 2017 +0530
>>>
>>>     build_mem_phdrs(): check if p_paddr is invalid
>>>
>>> Thanks
>>> Dave
>>>
>>> _______________________________________________
>>> kexec mailing list
>>> kexec at lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/kexec


From daniel.kiper at oracle.com  Wed Dec  6 11:01:24 2017
From: daniel.kiper at oracle.com (Daniel Kiper)
Date: Wed, 6 Dec 2017 20:01:24 +0100
Subject: [PATCH v1] kexec-tools: Perform run-time linking of libxenctrl.so
In-Reply-To: <1511973916-3915-1-git-send-email-eric.devolder@oracle.com>
References: <1511973916-3915-1-git-send-email-eric.devolder@oracle.com>
Message-ID: <20171206190124.GI16962@olila.local.net-space.pl>

On Wed, Nov 29, 2017 at 10:45:16AM -0600, Eric DeVolder wrote:
> When kexec is utilized in a Xen environment, it has an explicit
> run-time dependency on libxenctrl.so. This dependency occurs
> during the configure stage and when building kexec-tools.
>
> When kexec is utilized in a non-Xen environment (either bare
> metal or KVM), the configure and build of kexec-tools omits
> any reference to libxenctrl.so.
>
> Thus today it is not currently possible to configure and build
> a *single* kexec that will work in *both* Xen and non-Xen
> environments, unless the libxenctrl.so is *always* present.
>
> For example, a kexec configured for Xen in a Xen environment:
>
>  # ldd build/sbin/kexec
>         linux-vdso.so.1 =>  (0x00007ffdeba5c000)
>         libxenctrl.so.4.4 => /usr/lib64/libxenctrl.so.4.4 (0x00000038d8000000)
>         libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000)
>         libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000)
>         /lib64/ld-linux-x86-64.so.2 (0x000055e9f8c6c000)
>  # build/sbin/kexec -v
>  kexec-tools 2.0.16
>
> However, the *same* kexec executable fails in a non-Xen environment:
>
>  # copy xen kexec to .
>  # ldd ./kexec
>          linux-vdso.so.1 =>  (0x00007fffa9da7000)
>          libxenctrl.so.4.4 => not found
>          liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x0000003014e00000)
>          libz.so.1 => /lib64/libz.so.1 (0x000000300ea00000)
>          libc.so.6 => /lib64/libc.so.6 (0x000000300de00000)
>          libpthread.so.0 => /lib64/libpthread.so.0 (0x000000300e200000)
>          /lib64/ld-linux-x86-64.so.2 (0x0000558cc786c000)
>  # ./kexec -v
>  ./kexec: error while loading shared libraries:
>  libxenctrl.so.4.4: cannot open shared object file: No such file or directory
>
> At Oracle we "workaround" this by having two kexec-tools packages,
> one for Xen and another for non-Xen environments. At Oracle, the
> desire is to offer a single kexec-tools package that works in either
> environment. To achieve this, kexec-tools would either have to ship
> with libxenctrl.so (which we have deemed as unacceptable), or we can
> make kexec perform run-time linking against libxenctrl.so.
>
> This patch is one possible way to alleviate the explicit run-time
> dependency on libxenctrl.so. This implementation utilizes a set of
> macros to wrap calls into libxenctrl.so so that the library can
> instead be dlopen() and obtain the function via dlsym() and then
> make the call. The advantage of this implementation is that it
> requires few changes to the existing kexec-tools code. The dis-
> advantage is that it uses macros to remap libxenctrl functions
> and do work under the hood.
>
> Another possible implementation worth considering is the approach
> taken by libvmi. Reference the following file:
>
> https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/libxc_wrapper.h
>
> The libxc_wrapper_t structure definition that starts at line ~33
> has members that are function pointers into libxenctrl.so. This
> structure is populated once and then later referenced/dereferenced
> by the callers of libxenctrl.so members. The advantage of this
> implementation is it is more explicit in managing the use of
> libxenctrl.so and its versions, but the disadvantage is it would
> require touching more of the kexec-tools code.

After some thinking it seems to me that we can consider different approach.
Could not we pack all basic Xen libraries into small package called, e.g. libxen?
E.g. Debian provides libxen-4.8 which contains following files:

/usr/lib/x86_64-linux-gnu/libxencall-4.8.so.1
/usr/lib/x86_64-linux-gnu/libxencall-4.8.so.1.0
/usr/lib/x86_64-linux-gnu/libxenctrl-4.8.so
/usr/lib/x86_64-linux-gnu/libxenevtchn-4.8.so.1
/usr/lib/x86_64-linux-gnu/libxenevtchn-4.8.so.1.0
/usr/lib/x86_64-linux-gnu/libxenforeignmemory-4.8.so.1
/usr/lib/x86_64-linux-gnu/libxenforeignmemory-4.8.so.1.0
/usr/lib/x86_64-linux-gnu/libxengnttab-4.8.so.1
/usr/lib/x86_64-linux-gnu/libxengnttab-4.8.so.1.1
/usr/lib/x86_64-linux-gnu/libxenguest-4.8.so
/usr/lib/x86_64-linux-gnu/libxenlight-4.8.so
/usr/lib/x86_64-linux-gnu/libxentoollog-4.8.so.1
/usr/lib/x86_64-linux-gnu/libxentoollog-4.8.so.1.0
/usr/lib/x86_64-linux-gnu/libxlutil-4.8.so
/usr/share/bug/libxen-4.8/control
/usr/share/doc/libxen-4.8/changelog.Debian.gz
/usr/share/doc/libxen-4.8/changelog.gz
/usr/share/doc/libxen-4.8/copyright

Then we do not have to install all other Xen stuff to make kexec-tools happy
on non-Xen machines. And this should also work for other packages requiring
above listed Xen libraries.

Daniel


From ats-kumagai at wm.jp.nec.com  Fri Dec  8 00:15:30 2017
From: ats-kumagai at wm.jp.nec.com (Atsushi Kumagai)
Date: Fri, 8 Dec 2017 08:15:30 +0000
Subject: [PATCH] handle renamed init_level4_pgt -> init_top_pgt
In-Reply-To: <0910DD04CBD6DE4193FCF86B9C00BE9701F22E63@BPXM01GP.gisp.nec.co.jp>
References: <0910DD04CBD6DE4193FCF86B9C00BE9701F22E63@BPXM01GP.gisp.nec.co.jp>
Message-ID: <0910DD04CBD6DE4193FCF86B9C00BE9701F2A617@BPXM01GP.gisp.nec.co.jp>

>>Linux 4.13 renamed init_level4_pgt to init_top_pgt in preparation for
>>introducing 5-level page tables.  This patch follows the rename if
>>the lookup for init_level4_pgt fails.  It also checks to see if
>>5-level page tables are enabled and bails if it discovers they are.
>
>Thanks Jeff, but could you rebase it on the current devel branch ?
>vtop4 for x86_64 was modified in the commit below:
>
>commit 8c89727155f4994b4e75a659e28e5eff16ff6cbc
>Author: Takao Indoh <indou.takao at jp.fujitsu.com>
>Date:   Thu Oct 26 20:32:54 2017 +0900
>
>    [PATCH v3 2/4] Introduce vtop4_x86_64_pagetable
>
>Regards,
>Atsushi Kumagai

I rebased that and updated the devel branch:

https://sourceforge.net/p/makedumpfile/code/ci/97e156b493c46fabfc4b136ab834a6495166ac66/

I'll merge it into v1.6.3 if you have no comment.

Thanks,
Atsushi Kumagai

>>Signed-off-by: Jeff Mahoney <jeffm at suse.com>
>>---
>> arch/x86_64.c  | 24 +++++++++++++++++++++---
>> makedumpfile.c |  6 ++++++
>> makedumpfile.h |  2 ++
>> 3 files changed, 29 insertions(+), 3 deletions(-)
>>
>>diff --git a/arch/x86_64.c b/arch/x86_64.c
>>index 08dd6b2..9b09035 100644
>>--- a/arch/x86_64.c
>>+++ b/arch/x86_64.c
>>@@ -259,16 +259,26 @@ vtop4_x86_64(unsigned long vaddr)
>> {
>> 	unsigned long page_dir, pml4, pgd_paddr, pgd_pte, pmd_paddr, pmd_pte;
>> 	unsigned long pte_paddr, pte;
>>+	unsigned long init_level4_pgt;
>>
>>-	if (SYMBOL(init_level4_pgt) == NOT_FOUND_SYMBOL) {
>>+	if (SYMBOL(init_level4_pgt) != NOT_FOUND_SYMBOL)
>>+		init_level4_pgt = SYMBOL(init_level4_pgt);
>>+	else if (SYMBOL(init_top_pgt) != NOT_FOUND_SYMBOL)
>>+		init_level4_pgt = SYMBOL(init_top_pgt);
>>+	else {
>> 		ERRMSG("Can't get the symbol of init_level4_pgt.\n");
>> 		return NOT_PADDR;
>> 	}
>>
>>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
>>+		ERRMSG("Kernel is built with 5-level page tables\n");
>>+		return NOT_PADDR;
>>+	}
>>+
>> 	/*
>> 	 * Get PGD.
>> 	 */
>>-	page_dir = SYMBOL(init_level4_pgt) - __START_KERNEL_map + info->phys_base;
>>+	page_dir = init_level4_pgt - __START_KERNEL_map + info->phys_base;
>> 	if (is_xen_memory()) {
>> 		page_dir = ptom_xen(page_dir);
>> 		if (page_dir == NOT_PADDR)
>>@@ -549,8 +559,16 @@ find_vmemmap_x86_64()
>> 	struct vmap_pfns *vmapp, *vmaphead = NULL, *cur, *tail;
>>
>> 	init_level4_pgt = SYMBOL(init_level4_pgt);
>>+	if (init_level4_pgt == NOT_FOUND_SYMBOL)
>>+		init_level4_pgt = SYMBOL(init_top_pgt);
>>+
>> 	if (init_level4_pgt == NOT_FOUND_SYMBOL) {
>>-		ERRMSG("init_level4_pgt not found\n");
>>+		ERRMSG("init_level4_pgt/init_top_pgt not found\n");
>>+		return FAILED;
>>+	}
>>+
>>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
>>+		ERRMSG("kernel is configured for 5-level page tables\n");
>> 		return FAILED;
>> 	}
>> 	pagestructsize = size_table.page;
>>diff --git a/makedumpfile.c b/makedumpfile.c
>>index f85003a..6e5ec34 100644
>>--- a/makedumpfile.c
>>+++ b/makedumpfile.c
>>@@ -1486,6 +1486,8 @@ get_symbol_info(void)
>> 	SYMBOL_INIT(_stext, "_stext");
>> 	SYMBOL_INIT(swapper_pg_dir, "swapper_pg_dir");
>> 	SYMBOL_INIT(init_level4_pgt, "init_level4_pgt");
>>+	SYMBOL_INIT(level4_kernel_pgt, "level4_kernel_pgt");
>>+	SYMBOL_INIT(init_top_pgt, "init_top_pgt");
>> 	SYMBOL_INIT(vmlist, "vmlist");
>> 	SYMBOL_INIT(vmap_area_list, "vmap_area_list");
>> 	SYMBOL_INIT(node_online_map, "node_online_map");
>>@@ -2105,6 +2107,8 @@ write_vmcoreinfo_data(void)
>> 	WRITE_SYMBOL("_stext", _stext);
>> 	WRITE_SYMBOL("swapper_pg_dir", swapper_pg_dir);
>> 	WRITE_SYMBOL("init_level4_pgt", init_level4_pgt);
>>+	WRITE_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
>>+	WRITE_SYMBOL("init_top_pgt", init_top_pgt);
>> 	WRITE_SYMBOL("vmlist", vmlist);
>> 	WRITE_SYMBOL("vmap_area_list", vmap_area_list);
>> 	WRITE_SYMBOL("node_online_map", node_online_map);
>>@@ -2500,6 +2504,8 @@ read_vmcoreinfo(void)
>> 	READ_SYMBOL("_stext", _stext);
>> 	READ_SYMBOL("swapper_pg_dir", swapper_pg_dir);
>> 	READ_SYMBOL("init_level4_pgt", init_level4_pgt);
>>+	READ_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
>>+	READ_SYMBOL("init_top_pgt", init_top_pgt);
>> 	READ_SYMBOL("vmlist", vmlist);
>> 	READ_SYMBOL("vmap_area_list", vmap_area_list);
>> 	READ_SYMBOL("node_online_map", node_online_map);
>>diff --git a/makedumpfile.h b/makedumpfile.h
>>index 8a05794..9357e47 100644
>>--- a/makedumpfile.h
>>+++ b/makedumpfile.h
>>@@ -1517,6 +1517,8 @@ struct symbol_table {
>> 	unsigned long long	_stext;
>> 	unsigned long long	swapper_pg_dir;
>> 	unsigned long long	init_level4_pgt;
>>+	unsigned long long	level4_kernel_pgt;
>>+	unsigned long long	init_top_pgt;
>> 	unsigned long long	vmlist;
>> 	unsigned long long	vmap_area_list;
>> 	unsigned long long	phys_base;
>>
>>
>>--
>>Jeff Mahoney
>>SUSE Labs
>
>
>
>_______________________________________________
>kexec mailing list
>kexec at lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/kexec



From dyoung at redhat.com  Tue Dec 12 18:52:56 2017
From: dyoung at redhat.com (Dave Young)
Date: Wed, 13 Dec 2017 10:52:56 +0800
Subject: Regression: kexec/kdump boot hangs with x86/vector commits
Message-ID: <20171213025256.GA1913@dhcp-128-65.nay.redhat.com>

Hi,

Kexec reboot and kdump has broken on my laptop for long time with
4.15.0-rc1+ kernels. With the patch below an early panic been fixed:
https://patchwork.kernel.org/patch/10084289/

But still can not get a successful reboot, it looked like graphic
issue, but after bisecting the kernel, I got below:

[dyoung at dhcp-*-* linux]$ git bisect good
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
2db1f959d9dc16035f2eb44ed5fdb2789b754d6a
4900be83602b6be07366d3e69f756c1959f4169a
We cannot bisect more!

These two commits can no be reverted because of code conflicts, thus
I reverted the whole series from Thomas (below commits), with those
x86/vector changes reverted, kexec reboot works fine.

Could you help to take a look, any thoughts?  I can do the test
if you have some debug patch to try.

commit 90ad9e2d91067983f3328e21b306323877e5f48a
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Wed Sep 13 23:29:49 2017 +0200

    x86/io_apic: Reevaluate vector configuration on activate()

commit 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Wed Sep 13 23:29:50 2017 +0200

    x86/vector: Handle managed interrupts proper

commit 4900be83602b6be07366d3e69f756c1959f4169a
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Wed Sep 13 23:29:51 2017 +0200

    x86/vector/msi: Switch to global reservation mode

commit 464d12309e1b5829597793db551ae8ecaecf4036
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Wed Sep 13 23:29:52 2017 +0200

    x86/vector: Switch IOAPIC to global reservation mod

commit 2cffad7bad83157f89332872015f4305d2ac09ac
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Wed Sep 13 23:29:53 2017 +0200

    x86/irq: Simplify hotplug vector accounting

commit d6ffc6ac83b1f9f12652d89b9cb5bcbfbea7796c
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Wed Sep 13 23:29:54 2017 +0200

    x86/vector: Respect affinity mask in irq descriptor

commit 02edee152d6ea325c88898f3a702f5db2d78de7a
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Thu Oct 12 11:05:28 2017 +0200

    x86/apic/vector: Ignore set_affinity call for inactive interrupts

commit 0696d059f23c05f2dbc3b19ef50e5bdd175b782b
Author: Thomas Gleixner <tglx at linutronix.de>
Date:   Mon Oct 16 16:16:19 2017 +0200

    x86/vector: Use correct per cpu variable in free_moved_vector()
----------------

BTW, my git HEAD is below:
commit 43570f0383d6d5879ae585e6c3cf027ba321546f (origin/master, origin/HEAD, bisect)
Merge: 43f462f1c2e1 c14ca8386539
Author: Linus Torvalds <torvalds at linux-foundation.org>
Date:   Tue Nov 28 16:22:10 2017 -0800

    Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6



Thanks
Dave


From yu.c.chen at intel.com  Wed Dec 13 07:57:46 2017
From: yu.c.chen at intel.com (Yu Chen)
Date: Wed, 13 Dec 2017 23:57:46 +0800
Subject: Regression: kexec/kdump boot hangs with x86/vector commits
In-Reply-To: <20171213025256.GA1913@dhcp-128-65.nay.redhat.com>
References: <20171213025256.GA1913@dhcp-128-65.nay.redhat.com>
Message-ID: <20171213155746.GA29572@yu-chen.sh.intel.com>

On Wed, Dec 13, 2017 at 10:52:56AM +0800, Dave Young wrote:
> Hi,
> 
> Kexec reboot and kdump has broken on my laptop for long time with
> 4.15.0-rc1+ kernels. With the patch below an early panic been fixed:
> https://patchwork.kernel.org/patch/10084289/
> 
> But still can not get a successful reboot, it looked like graphic
> issue, but after bisecting the kernel, I got below:
> 
> [dyoung at dhcp-*-* linux]$ git bisect good
> There are only 'skip'ped commits left to test.
> The first bad commit could be any of:
> 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a
> 4900be83602b6be07366d3e69f756c1959f4169a
> We cannot bisect more!
> 
> These two commits can no be reverted because of code conflicts, thus
> I reverted the whole series from Thomas (below commits), with those
> x86/vector changes reverted, kexec reboot works fine.
> 
> Could you help to take a look, any thoughts?  I can do the test
> if you have some debug patch to try.
Is it possible that the "second" kernel runs on non-zero CPU? If yes,
what if some irqs are only delivered to cpu0? (use cpumask_of(0)
directly)

Thanks,
	Yu



From dyoung at redhat.com  Thu Dec 14 01:24:29 2017
From: dyoung at redhat.com (Dave Young)
Date: Thu, 14 Dec 2017 17:24:29 +0800
Subject: Regression: kexec/kdump boot hangs with x86/vector commits
In-Reply-To: <20171213155746.GA29572@yu-chen.sh.intel.com>
References: <20171213025256.GA1913@dhcp-128-65.nay.redhat.com>
 <20171213155746.GA29572@yu-chen.sh.intel.com>
Message-ID: <20171214092429.GA2004@dhcp-128-65.nay.redhat.com>

On 12/13/17 at 11:57pm, Yu Chen wrote:
> On Wed, Dec 13, 2017 at 10:52:56AM +0800, Dave Young wrote:
> > Hi,
> > 
> > Kexec reboot and kdump has broken on my laptop for long time with
> > 4.15.0-rc1+ kernels. With the patch below an early panic been fixed:
> > https://patchwork.kernel.org/patch/10084289/
> > 
> > But still can not get a successful reboot, it looked like graphic
> > issue, but after bisecting the kernel, I got below:
> > 
> > [dyoung at dhcp-*-* linux]$ git bisect good
> > There are only 'skip'ped commits left to test.
> > The first bad commit could be any of:
> > 2db1f959d9dc16035f2eb44ed5fdb2789b754d6a
> > 4900be83602b6be07366d3e69f756c1959f4169a
> > We cannot bisect more!
> > 
> > These two commits can no be reverted because of code conflicts, thus
> > I reverted the whole series from Thomas (below commits), with those
> > x86/vector changes reverted, kexec reboot works fine.
> > 
> > Could you help to take a look, any thoughts?  I can do the test
> > if you have some debug patch to try.
> Is it possible that the "second" kernel runs on non-zero CPU? If yes,
> what if some irqs are only delivered to cpu0? (use cpumask_of(0)
> directly)

Thanks for the reply.

For kdump, yes, for kexec, I'm not sure.  

Here is some kexec kernel boot log:
http://people.redhat.com/~ruyang/misc/kexec-regression.txt

Copy the lockup call trace here:
[   23.779285] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0             
[   23.779285] Modules linked in: arc4 rtsx_pci_sdmmc i915 iwlmvm kvm_intel mac8
0211 kvm irqbypass btusb btrtl btbcm intel_gtt btintel drm_kms_helper snd_hda_in
tel syscopyarea bluetooth iwlwifi snd_hda_codec snd_hwdep snd_hda_core sysfillre
ct snd_seq sysimgblt input_leds fb_sys_fops e1000e ecdh_generic cfg80211 snd_seq
_device drm snd_pcm serio_raw ptp pcspkr thinkpad_acpi i2c_i801 snd_timer rtsx_p
ci pps_core snd soundcore rfkill video                                          
[   23.779307] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc3+ #378       
[   23.779308] Hardware name: LENOVO 20ARS1BJ02/20ARS1BJ02, BIOS GJET92WW (2.42 
) 03/03/2017                                                                    
[   23.779312] RIP: 0010:poll_idle+0x2f/0x5f                                    
[   23.779313] RSP: 0018:ffffffff81c03e80 EFLAGS: 00000246                      
[   23.779314] RAX: ffffffff81c0f4c0 RBX: ffffffff81c6db80 RCX: 0000000000000000
[   23.779315] RDX: 0000000000000000 RSI: ffffffff81c6db80 RDI: ffff88021f2201e8
[   23.779316] RBP: ffff88021f2201e8 R08: 000000349a65b7dd R09: ffff88021f216db4
[   23.779317] R10: ffffffff81c03e68 R11: 0000000000000000 R12: 0000000000000000
[   23.779318] R13: ffffffff81c6db98 R14: 0000000000000000 R15: 0000000578a065b1
[   23.779319] FS:  0000000000000000(0000) GS:ffff88021f200000(0000) knlGS:00000
00000000000                                                                     
[   23.779320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                
[   23.779321] CR2: 00007ffed1d0ee60 CR3: 000000021ec0a006 CR4: 00000000001606b0
[   23.779322] Call Trace:                                                      
[   23.779328]  cpuidle_enter_state+0x6a/0x2c0                                  
[   23.779333]  do_idle+0x17b/0x1d0                                             
[   23.779335]  cpu_startup_entry+0x6f/0x80                                     
[   23.779338]  start_kernel+0x431/0x451                                        
[   23.779342]  secondary_startup_64+0xa5/0xb0                                  
[   23.779344] Code: 00 fb 66 0f 1f 44 00 00 65 48 8b 04 25 40 c4 00 00 f0 80 48
 02 20 48 8b 08 83 e1 08 74 0d eb 12 f3 90 65 48 8b 04 25 40 c4 00 00 <48> 8b 00
 a8 08 74 ee 65 48 8b 04 25 40 c4 00 00 f0 80 60 02 df

Thanks
Dave
> 
> Thanks,
> 	Yu
> 


From dyoung at redhat.com  Thu Dec 14 02:25:13 2017
From: dyoung at redhat.com (Dave Young)
Date: Thu, 14 Dec 2017 18:25:13 +0800
Subject: [PATCH] certs: always use secondary keyring first if possible
In-Reply-To: <20171118044711.GA7352@dhcp-128-65.nay.redhat.com>
References: <20171118044711.GA7352@dhcp-128-65.nay.redhat.com>
Message-ID: <20171214102513.GA2371@dhcp-128-65.nay.redhat.com>

On 11/18/17 at 12:47pm, Dave Young wrote:
> Commit d3bfe84129f6 introduced secondary_trusted_keys keyring, current
> users of verify_pkcs7_signature are below:
> net/wireless/reg.c : uses its own trusted_keys
> kernel/module_signing.c : pass NULL trusted_keys 
> crypto/asymmetric_keys/verify_pefile.c : pass NULL trusted_keys
> 
> For both module and pefile verification, there is no reason to use builtin
> keys only. Actually in Fedora kernel module signing code passes 1UL, but
> kexec code does not pass 1UL for pefile verification thus we have below bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1470995
> 
> Drop the hard code 1UL checking so that pefile verification can use
> secondary keyring as well.
> 
> Signed-off-by: Dave Young <dyoung at redhat.com>
> ---
>  certs/system_keyring.c |    2 --
>  1 file changed, 2 deletions(-)
> 
> --- linux-x86.orig/certs/system_keyring.c
> +++ linux-x86/certs/system_keyring.c
> @@ -229,8 +229,6 @@ int verify_pkcs7_signature(const void *d
>  		goto error;
>  
>  	if (!trusted_keys) {
> -		trusted_keys = builtin_trusted_keys;
> -	} else if (trusted_keys == (void *)1UL) {
>  #ifdef CONFIG_SECONDARY_TRUSTED_KEYRING
>  		trusted_keys = secondary_trusted_keys;
>  #else

Another ping.

If the (-1UL) is really needed, below file need update to use it
But I think it is ugly..
crypto/asymmetric_keys/verify_pefile.c

Thanks
Dave


From eric.devolder at oracle.com  Thu Dec 14 14:48:01 2017
From: eric.devolder at oracle.com (Eric DeVolder)
Date: Thu, 14 Dec 2017 16:48:01 -0600
Subject: [PATCH v2] kexec-tools: Perform run-time linking of libxenctrl.so
Message-ID: <1513291681-497-1-git-send-email-eric.devolder@oracle.com>

When kexec is utilized in a Xen environment, it has an explicit
run-time dependency on libxenctrl.so. This dependency occurs
during the configure stage and when building kexec-tools.

When kexec is utilized in a non-Xen environment (either bare
metal or KVM), the configure and build of kexec-tools omits
any reference to libxenctrl.so.

Thus today it is not currently possible to configure and build
a *single* kexec that will work in *both* Xen and non-Xen
environments, unless the libxenctrl.so is *always* present.

For example, a kexec configured for Xen in a Xen environment:

 # ldd build/sbin/kexec
        linux-vdso.so.1 =>  (0x00007ffdeba5c000)
        libxenctrl.so.4.4 => /usr/lib64/libxenctrl.so.4.4 (0x00000038d8000000)
        libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000)
        libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000)
        /lib64/ld-linux-x86-64.so.2 (0x000055e9f8c6c000)
 # build/sbin/kexec -v
 kexec-tools 2.0.16

However, the *same* kexec executable fails in a non-Xen environment:

 # copy xen kexec to .
 # ldd ./kexec
         linux-vdso.so.1 =>  (0x00007fffa9da7000)
         libxenctrl.so.4.4 => not found
         liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x0000003014e00000)
         libz.so.1 => /lib64/libz.so.1 (0x000000300ea00000)
         libc.so.6 => /lib64/libc.so.6 (0x000000300de00000)
         libpthread.so.0 => /lib64/libpthread.so.0 (0x000000300e200000)
         /lib64/ld-linux-x86-64.so.2 (0x0000558cc786c000)
 # ./kexec -v
 ./kexec: error while loading shared libraries:
 libxenctrl.so.4.4: cannot open shared object file: No such file or directory

At Oracle we "workaround" this by having two kexec-tools packages,
one for Xen and another for non-Xen environments. At Oracle, the
desire is to offer a single kexec-tools package that works in either
environment. To achieve this, kexec-tools would either have to ship
with libxenctrl.so (which we have deemed as unacceptable), or we can
make kexec perform run-time linking against libxenctrl.so.

This patch is one possible way to alleviate the explicit run-time
dependency on libxenctrl.so. This implementation utilizes a set of
macros to wrap calls into libxenctrl.so so that the library can
instead be dlopen() and obtain the function via dlsym() and then
make the call. The advantage of this implementation is that it
requires few changes to the existing kexec-tools code. The dis-
advantage is that it uses macros to remap libxenctrl functions
and do work under the hood.

Another possible implementation worth considering is the approach
taken by libvmi. Reference the following file:

https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/libxc_wrapper.h

The libxc_wrapper_t structure definition that starts at line ~33
has members that are function pointers into libxenctrl.so. This
structure is populated once and then later referenced/dereferenced
by the callers of libxenctrl.so members. The advantage of this
implementation is it is more explicit in managing the use of
libxenctrl.so and its versions, but the disadvantage is it would
require touching more of the kexec-tools code.

The following is a list libxenctrl members utilized by kexec:

Functions:
xc_interface_open
xc_kexec_get_range
xc_interface_close
xc_kexec_get_range
xc_interface_open
xc_get_max_cpus
xc_kexec_get_range
xc_version
xc_kexec_exec
xc_kexec_status
xc_kexec_unload
xc_hypercall_buffer_array_create
xc__hypercall_buffer_array_alloc
xc_hypercall_buffer_array_destroy
xc_kexec_load
xc_get_machine_memory_map

Data:
xc__hypercall_buffer_HYPERCALL_BUFFER_NULL

These were identified by configuring and building kexec-tools
with Xen support, but omitting the -lxenctrl from the LDFLAGS
in the Makefile for an x86_64 build.

The above libxenctrl members were referenced via these source
files.

kexec/crashdump-xen.c
kexec/kexec-xen.c
kexec/arch/i386/kexec-x86-common.c
kexec/arch/i386/crashdump-x86.c

This patch provides a wrapper around the calls to the above
functions in libxenctrl.so. Every libxenctrl call must pass a
xc_interface which it obtains from xc_interface_open().
So the existing code is already structured in a manner that
facilitates graceful dlopen()'ing of the libxenctrl.so and
the subsequent dlsym() of the required member.

The patch creates a wrapper function around xc_interface_open()
and xc_interface_close() to perform the dlopen() and dlclose().

For the remaining xc_ functions, this patch defines a macro
of the same name which performs the dlsym() and then invokes
the function. See the _xc_call() macro for details.

There was one data item in libxenctrl.so that presented a
unique problem, HYPERCALL_BUFFER_NULL. It was only utilized
once, as

    set_xen_guest_handle(xen_segs[s].buf.h, HYPERCALL_BUFFER_NULL);

I tried a variety of techniques but could not find a general
macro-type solution without modifying xenctrl.h. So the
solution was to declare a local HYPERCALL_BUFFER_NULL, and
this appears to work. I admit I am not familiar with libxenctrl
to state if this is a satisfactory workaround, so feedback
here welcome. I can state that this allows kexec to load/unload/kexec
on Xen and non-Xen environments that I've tested without issue.

With this patch applied, kexec-tools can be built with Xen
support and yet there is no explicit run-time dependency on
libxenctrl.so. Thus it can also be deployed in non-Xen
environments where libxenctrl.so is not installed.

 # ldd build/sbin/kexec
        linux-vdso.so.1 =>  (0x00007fff7dbcd000)
        liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x00000038d9000000)
        libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000)
        libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000)
        /lib64/ld-linux-x86-64.so.2 (0x0000562dc0c14000)
 # build/sbin/kexec -v
 kexec-tools 2.0.16

Currently this feature is enabled with the following:

 ./configure --with-xen-dl --with-xen=no

This is a bit clunky. I welcome feedback such as better names
and/or usage of --with, as well as if we might make this feature
the default.

Signed-off-by: Eric DeVolder <eric.devolder at oracle.com>
---
v1: 29nov2017
 - Daniel Kiper suggested Debian's libxen package of libraries,
   but I did not find similar package on most other systems.

v2: 14dec2017
 - Reposted to kexec and xen-devel mailing lists
---
 configure.ac                       | 18 ++++++++++++++
 kexec/Makefile                     |  1 +
 kexec/arch/i386/crashdump-x86.c    |  4 +---
 kexec/arch/i386/kexec-x86-common.c |  4 +---
 kexec/crashdump-xen.c              |  4 +---
 kexec/kexec-xen.c                  | 43 +++++++++++++++++++++++++++++++++-
 kexec/kexec-xen.h                  | 48 ++++++++++++++++++++++++++++++++++++++
 7 files changed, 112 insertions(+), 10 deletions(-)
 create mode 100644 kexec/kexec-xen.h

diff --git a/configure.ac b/configure.ac
index 208dc0a..4fc8aa0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -92,6 +92,9 @@ AC_ARG_WITH([lzma], AC_HELP_STRING([--without-lzma],[disable lzma support]),
 AC_ARG_WITH([xen], AC_HELP_STRING([--without-xen],
 	[disable extended xen support]), [ with_xen="$withval"], [ with_xen=yes ] )
 
+AC_ARG_WITH([xen-dl], AC_HELP_STRING([--without-xen-dl],
+	[link libxenctrl.so at run-time rather than build-time]), [ with_xen_dl="$withval"], [ with_xen_dl=no ] )
+
 AC_ARG_WITH([booke],
 		AC_HELP_STRING([--with-booke],[build for booke]),
 		AC_DEFINE(CONFIG_BOOKE,1,
@@ -174,6 +177,21 @@ if test "$with_xen" = yes ; then
 				AC_MSG_NOTICE([The kexec_status call is not available]))
 		fi
 fi
+if test "$with_xen_dl" = yes ; then
+	if test "$with_xen" = yes ; then
+		AC_MSG_ERROR([Options --with-xen and --with-xen-dl are mutually exclusive])
+	fi
+	AC_DEFINE(CONFIG_LIBXENCTRL_DL, 1, [Define to 1 to link libxenctrl.so at run-time rather than build-time])
+	AC_CHECK_HEADER(dlfcn.h, , AC_MSG_ERROR([Dynamic library linking not available]))
+	AC_CHECK_LIB(dl, dlopen, , AC_MSG_ERROR([Dynamic library linking not available]))
+	AC_CHECK_HEADER(xenctrl.h,
+		AC_DEFINE(HAVE_LIBXENCTRL, 1, [Define to 1 to enable run-time linking of libxenctrl.so]),
+		AC_MSG_ERROR([Xen support not available]))
+dnl NOTE: Explicitly *NOT* performing AC_CHECK_LIB(xenctrl) as only need the header file to build
+	AC_CHECK_HEADER(xenctrl.h,
+		AC_DEFINE(HAVE_KEXEC_CMD_STATUS, 1, [Define to 1 so kexec_status call is available]),
+		AC_MSG_NOTICE([The kexec_status call is not available]))
+fi
 
 dnl ---Sanity checks
 if test "$CC"      = "no"; then AC_MSG_ERROR([cc not found]); fi
diff --git a/kexec/Makefile b/kexec/Makefile
index 2b4fb3d..8871731 100644
--- a/kexec/Makefile
+++ b/kexec/Makefile
@@ -36,6 +36,7 @@ dist += kexec/Makefile						\
 	kexec/kexec-elf-boot.h					\
 	kexec/kexec-elf.h kexec/kexec-sha256.h			\
 	kexec/kexec-zlib.h kexec/kexec-lzma.h			\
+	kexec/kexec-xen.h 								\
 	kexec/kexec-syscall.h kexec/kexec.h kexec/kexec.8
 
 dist				+= kexec/proc_iomem.c
diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
index 69a063a..a948d9f 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -44,9 +44,7 @@
 #include "kexec-x86.h"
 #include "crashdump-x86.h"
 
-#ifdef HAVE_LIBXENCTRL
-#include <xenctrl.h>
-#endif /* HAVE_LIBXENCTRL */
+#include "../../kexec-xen.h"
 
 #include "x86-linux-setup.h"
 
diff --git a/kexec/arch/i386/kexec-x86-common.c b/kexec/arch/i386/kexec-x86-common.c
index be03618..b44c8b7 100644
--- a/kexec/arch/i386/kexec-x86-common.c
+++ b/kexec/arch/i386/kexec-x86-common.c
@@ -40,9 +40,7 @@
 #include "../../crashdump.h"
 #include "kexec-x86.h"
 
-#ifdef HAVE_LIBXENCTRL
-#include <xenctrl.h>
-#endif /* HAVE_LIBXENCTRL */
+#include "../../kexec-xen.h"
 
 /* Used below but not present in (older?) xenctrl.h */
 #ifndef E820_PMEM
diff --git a/kexec/crashdump-xen.c b/kexec/crashdump-xen.c
index 60594f6..2e4cbdc 100644
--- a/kexec/crashdump-xen.c
+++ b/kexec/crashdump-xen.c
@@ -18,9 +18,7 @@
 
 #include "config.h"
 
-#ifdef HAVE_LIBXENCTRL
-#include <xenctrl.h>
-#endif
+#include "kexec-xen.h"
 
 struct crash_note_info {
 	unsigned long base;
diff --git a/kexec/kexec-xen.c b/kexec/kexec-xen.c
index 2b448d3..2b01fee 100644
--- a/kexec/kexec-xen.c
+++ b/kexec/kexec-xen.c
@@ -10,10 +10,51 @@
 #include "config.h"
 
 #ifdef HAVE_LIBXENCTRL
-#include <xenctrl.h>
+#include "kexec-xen.h"
 
 #include "crashdump.h"
 
+#ifdef CONFIG_LIBXENCTRL_DL
+void *xc_dlhandle = NULL;
+xc_hypercall_buffer_t XC__HYPERCALL_BUFFER_NAME(HYPERCALL_BUFFER_NULL);
+xc_interface *_xc_interface_open(xentoollog_logger *logger,
+                                xentoollog_logger *dombuild_logger,
+                                unsigned open_flags)
+{
+    xc_interface *xch = NULL;
+
+    if (NULL == xc_dlhandle)
+        xc_dlhandle = dlopen("libxenctrl.so", RTLD_NOW | RTLD_NODELETE);
+
+    if (xc_dlhandle) {
+        typedef xc_interface *(*func_t)(xentoollog_logger *logger,
+                                xentoollog_logger *dombuild_logger,
+                                unsigned open_flags);
+        func_t func = (func_t)dlsym(xc_dlhandle, "xc_interface_open");
+        xch = func(logger, dombuild_logger, open_flags);
+    }
+
+    return xch;
+}
+
+int _xc_interface_close(xc_interface *xch)
+{
+    int rc = -1;
+
+/*
+    if (xc_dlhandle) {
+*/
+        typedef int (*func_t)(xc_interface *xch);
+        func_t func = (func_t)dlsym(xc_dlhandle, "xc_interface_close");
+        rc = func(xch);
+/*
+        xc_dlhandle = NULL;
+    }
+*/
+    return rc;
+}
+#endif /* CONFIG_LIBXENCTRL_DL */
+
 int xen_kexec_load(struct kexec_info *info)
 {
 	uint32_t nr_segments = info->nr_segments;
diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
new file mode 100644
index 0000000..6d2046e
--- /dev/null
+++ b/kexec/kexec-xen.h
@@ -0,0 +1,48 @@
+#ifndef KEXEC_XEN_H
+#define KEXEC_XEN_H
+
+#ifdef HAVE_LIBXENCTRL
+#include <xenctrl.h>
+
+#ifdef CONFIG_LIBXENCTRL_DL
+#include <dlfcn.h>
+
+/* The handle from dlopen(), needed by dlsym(), dlclose() */
+extern void *xc_dlhandle;
+
+/* Wrappers around xc_interface_open/close() to insert dlopen/dlclose() */
+xc_interface *_xc_interface_open(xentoollog_logger *logger,
+                                xentoollog_logger *dombuild_logger,
+                                unsigned open_flags);
+int _xc_interface_close(xc_interface *xch);
+
+/* GCC expression statements for evaluating dlsym() */
+#define _xc_call(DTYPE, NAME, ARGS...) ({ DTYPE value; typedef DTYPE (*func_t)(xc_interface *, ...); func_t func = dlsym(xc_dlhandle, #NAME); value = func(ARGS); value; } )
+#define _xc_data(DTYPE, NAME) ({ DTYPE *value = (DTYPE *)dlsym(xc_dlhandle, #NAME); value; } )
+
+/* The wrappers around utilized xenctrl.h functions */
+#define xc_interface_open(A,B,C)    _xc_interface_open(A,B,C)
+#define xc_interface_close(A)       _xc_interface_close(A)
+#define xc_version(ARGS...)         _xc_call(int, xc_version, ARGS)
+#define xc_get_max_cpus(ARGS...)    _xc_call(int, xc_get_max_cpus, ARGS)
+#define xc_get_machine_memory_map(ARGS...)  _xc_call(int, xc_get_machine_memory_map, ARGS)
+#define xc_kexec_get_range(ARGS...) _xc_call(int, xc_kexec_get_range, ARGS)
+#define xc_kexec_load(ARGS...)      _xc_call(int, xc_kexec_load, ARGS)
+#define xc_kexec_unload(ARGS...)    _xc_call(int, xc_kexec_unload, ARGS)
+#define xc_kexec_status(ARGS...)    _xc_call(int, xc_kexec_status, ARGS)
+#define xc_kexec_exec(ARGS...)      _xc_call(int, xc_kexec_exec, ARGS)
+#define xc_hypercall_buffer_array_create(ARGS...)   _xc_call(xc_hypercall_buffer_array_t *, xc_hypercall_buffer_array_create, ARGS)
+#define xc__hypercall_buffer_alloc(ARGS...) _xc_call(void *, xc__hypercall_buffer_alloc, ARGS)
+#define xc__hypercall_buffer_free(ARGS...)  _xc_call(void  , xc__hypercall_buffer_free, ARGS)
+#define xc__hypercall_buffer_alloc_pages(ARGS...)   _xc_call(void *, xc__hypercall_buffer_alloc_pages, ARGS)
+#define xc__hypercall_buffer_free_pages(ARGS...)    _xc_call(void  , xc__hypercall_buffer_free_pages, ARGS)
+#define xc__hypercall_buffer_array_alloc(ARGS...)   _xc_call(void *, xc__hypercall_buffer_array_alloc, ARGS)
+#define xc__hypercall_buffer_array_get(ARGS...)     _xc_call(void *, xc__hypercall_buffer_array_get, ARGS)
+#define xc_hypercall_buffer_array_destroy(ARGS...)  _xc_call(void *, xc_hypercall_buffer_array_destroy, ARGS)
+
+#endif /* CONFIG_LIBXENCTRL_DL */
+
+#endif /* HAVE_LIBXENCTRL */
+
+#endif /* KEXEC_XEN_H */
+
-- 
2.7.4



From jbohac at suse.cz  Fri Dec 15 16:15:14 2017
From: jbohac at suse.cz (Jiri Bohac)
Date: Sat, 16 Dec 2017 01:15:14 +0100
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
Message-ID: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>

On machines where the GART aperture is mapped over physical RAM
/proc/vmcore contains the remapped range and reading it may
cause hangs or reboots. 

In the past, the GART region was added into the resource map, implemented by
commit 56dd669a138c ("[PATCH] Insert GART region into resource map")

However, inserting the iomem_resource from the early GART code caused
resource conflicts with some AGP drivers (bko#72201), which got avoided by
reverting the patch in commit 707d4eefbdb3 ("Revert [PATCH] Insert GART
region into resource map"). This revert introduced the /proc/vmcore bug.

The vmcore ELF header is either prepared by the kernel (when using the
kexec_file_load syscall) or by the kexec userspace (when using the kexec_load
syscall). Since we no longer have the GART iomem resource, the userspace
kexec has no way of knowing which region to exclude from the ELF header.

Changes from v1 of this patch:
Instead of excluding the aperture from the ELF header, this patch
makes /proc/vmcore return zeroes in the second kernel when attempting to
read the aperture region. This is done by reusing the
gart_oldmem_pfn_is_ram infrastructure originally intended to exclude XEN
balooned memory. This works for both, the kexec_file_load and kexec_load
syscalls.

[Note that the GART region is the same in the first and second kernels:
regardless whether the first kernel fixed up the northbridge/bios setting
and mapped the aperture over physical memory, the second kernel finds the
northbridge properly configured by the first kernel and the aperture
never overlaps with e820 memory because the second kernel has a fake e820
map created from the crashkernel memory regions. Thus, the second kernel
keeps the aperture address/size as configured by the first kernel.]

register_oldmem_pfn_is_ram can only register one callback and returns an error
if the callback has been registered already. Since XEN used to be the only user
of this function, it never checks the return value. Now that we have more than
one user, I added a WARN_ON just in case agp, XEN, or any other future user of
register_oldmem_pfn_is_ram were to step on each other's toes.


Signed-off-by: Jiri Bohac <jbohac at suse.cz>
Fixes: 707d4eefbdb3 ("Revert [PATCH] Insert GART region into resource map")

diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c
index f5d92bc3b884..837efa32110c 100644
--- a/arch/x86/kernel/aperture_64.c
+++ b/arch/x86/kernel/aperture_64.c
@@ -30,6 +30,7 @@
 #include <asm/dma.h>
 #include <asm/amd_nb.h>
 #include <asm/x86_init.h>
+#include <linux/crash_dump.h>
 
 /*
  * Using 512M as goal, in case kexec will load kernel_big
@@ -56,6 +57,27 @@ int fallback_aper_force __initdata;
 
 int fix_aperture __initdata = 1;
 
+#ifdef CONFIG_PROC_VMCORE
+static unsigned long aperture_pfn_start, aperture_page_count;
+
+static int gart_oldmem_pfn_is_ram(unsigned long pfn)
+{
+	return likely((pfn < aperture_pfn_start) ||
+		      (pfn >= aperture_pfn_start + aperture_page_count));
+}
+
+static void exclude_from_vmcore(u64 aper_base, u32 aper_order)
+{
+	aperture_pfn_start = aper_base >> PAGE_SHIFT;
+	aperture_page_count = (32 * 1024 * 1024) << aper_order >> PAGE_SHIFT;
+	WARN_ON(register_oldmem_pfn_is_ram(&gart_oldmem_pfn_is_ram));
+}
+#else
+static void exclude_from_vmcore(u64 aper_base, u32 aper_order)
+{
+}
+#endif
+
 /* This code runs before the PCI subsystem is initialized, so just
    access the northbridge directly. */
 
@@ -435,8 +457,10 @@ int __init gart_iommu_hole_init(void)
 
 out:
 	if (!fix && !fallback_aper_force) {
-		if (last_aper_base)
+		if (last_aper_base) {
+			exclude_from_vmcore(last_aper_base, last_aper_order);
 			return 1;
+		}
 		return 0;
 	}
 
@@ -473,6 +497,8 @@ int __init gart_iommu_hole_init(void)
 		return 0;
 	}
 
+	exclude_from_vmcore(aper_alloc, aper_order);
+
 	/* Fix up the north bridges */
 	for (i = 0; i < amd_nb_bus_dev_ranges[i].dev_limit; i++) {
 		int bus, dev_base, dev_limit;
diff --git a/arch/x86/xen/mmu_hvm.c b/arch/x86/xen/mmu_hvm.c
index 2cfcfe4f6b2a..dd2ad82eee80 100644
--- a/arch/x86/xen/mmu_hvm.c
+++ b/arch/x86/xen/mmu_hvm.c
@@ -75,6 +75,6 @@ void __init xen_hvm_init_mmu_ops(void)
 	if (is_pagetable_dying_supported())
 		pv_mmu_ops.exit_mmap = xen_hvm_exit_mmap;
 #ifdef CONFIG_PROC_VMCORE
-	register_oldmem_pfn_is_ram(&xen_oldmem_pfn_is_ram);
+	WARN_ON(register_oldmem_pfn_is_ram(&xen_oldmem_pfn_is_ram));
 #endif
 }
-- 
Jiri Bohac <jbohac at suse.cz>
SUSE Labs, Prague, Czechia



From bhe at redhat.com  Fri Dec 15 17:01:42 2017
From: bhe at redhat.com (Baoquan He)
Date: Sat, 16 Dec 2017 09:01:42 +0800
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
In-Reply-To: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
References: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
Message-ID: <20171216010142.GK12442@x1>

On 12/16/17 at 01:15am, Jiri Bohac wrote:
> On machines where the GART aperture is mapped over physical RAM
> /proc/vmcore contains the remapped range and reading it may
> cause hangs or reboots. 
> 
> In the past, the GART region was added into the resource map, implemented by
> commit 56dd669a138c ("[PATCH] Insert GART region into resource map")
> 
> However, inserting the iomem_resource from the early GART code caused
> resource conflicts with some AGP drivers (bko#72201), which got avoided by
> reverting the patch in commit 707d4eefbdb3 ("Revert [PATCH] Insert GART
> region into resource map"). This revert introduced the /proc/vmcore bug.
> 
> The vmcore ELF header is either prepared by the kernel (when using the
> kexec_file_load syscall) or by the kexec userspace (when using the kexec_load
> syscall). Since we no longer have the GART iomem resource, the userspace
> kexec has no way of knowing which region to exclude from the ELF header.
> 
> Changes from v1 of this patch:
> Instead of excluding the aperture from the ELF header, this patch
> makes /proc/vmcore return zeroes in the second kernel when attempting to
> read the aperture region. This is done by reusing the
> gart_oldmem_pfn_is_ram infrastructure originally intended to exclude XEN
> balooned memory. This works for both, the kexec_file_load and kexec_load
> syscalls.

This is funny and smart way. While it may not be good, E.g what if
people add iommu=off in kdump kernel? You made an assumption on this,
which people must use iommu in kdump kernel and aperture_pfn_start can
be got naturally.

As I said in v1, I listed 3 options. In fact 1st option is doable. Just
need to figure out why peopld did the quirk to enable gart anyway when
not enable gart in bios. You replied that you guess there's firmware bug
so that people have to do this. I have to say guess is not convincing.

Well, in fact, I have several questions:

1) In your case, the HP machine with gart, when you enable gart support
in bios, is it OK to you?

2) If firmware is broken, you can't enable gart in firmware, will
firmware engineer fix this since it's a firmware bug?

If above two have answer 'yes', then it's not a blocker to you. And we
should remove the quirk code which enable gart in kernel when people
forget enableing gart support in firmware.

OR

Take the option 3 as I said. Take the gart region off from iomem. But
you said there will be conflict with pci issue. If I did, I would make
clear what's the pci issue and why they conflict. Can we do in another
way to dig it out from iomem, e.g in different stage, after pci bus
iteration. 

I didn't reply to you later since I think my comments have been clear,
and I am very busy on rhel regression bug fix. Kernel bug fix sometime
is not easy. I don't find a gart system in redhat's lab, otherwise I
will make a post to have a try.

Thanks
Baoquan

> 
> [Note that the GART region is the same in the first and second kernels:
> regardless whether the first kernel fixed up the northbridge/bios setting
> and mapped the aperture over physical memory, the second kernel finds the
> northbridge properly configured by the first kernel and the aperture
> never overlaps with e820 memory because the second kernel has a fake e820
> map created from the crashkernel memory regions. Thus, the second kernel
> keeps the aperture address/size as configured by the first kernel.]
> 
> register_oldmem_pfn_is_ram can only register one callback and returns an error
> if the callback has been registered already. Since XEN used to be the only user
> of this function, it never checks the return value. Now that we have more than
> one user, I added a WARN_ON just in case agp, XEN, or any other future user of
> register_oldmem_pfn_is_ram were to step on each other's toes.
> 
> 
> Signed-off-by: Jiri Bohac <jbohac at suse.cz>
> Fixes: 707d4eefbdb3 ("Revert [PATCH] Insert GART region into resource map")
> 
> diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c
> index f5d92bc3b884..837efa32110c 100644
> --- a/arch/x86/kernel/aperture_64.c
> +++ b/arch/x86/kernel/aperture_64.c
> @@ -30,6 +30,7 @@
>  #include <asm/dma.h>
>  #include <asm/amd_nb.h>
>  #include <asm/x86_init.h>
> +#include <linux/crash_dump.h>
>  
>  /*
>   * Using 512M as goal, in case kexec will load kernel_big
> @@ -56,6 +57,27 @@ int fallback_aper_force __initdata;
>  
>  int fix_aperture __initdata = 1;
>  
> +#ifdef CONFIG_PROC_VMCORE
> +static unsigned long aperture_pfn_start, aperture_page_count;
> +
> +static int gart_oldmem_pfn_is_ram(unsigned long pfn)
> +{
> +	return likely((pfn < aperture_pfn_start) ||
> +		      (pfn >= aperture_pfn_start + aperture_page_count));
> +}
> +
> +static void exclude_from_vmcore(u64 aper_base, u32 aper_order)
> +{
> +	aperture_pfn_start = aper_base >> PAGE_SHIFT;
> +	aperture_page_count = (32 * 1024 * 1024) << aper_order >> PAGE_SHIFT;
> +	WARN_ON(register_oldmem_pfn_is_ram(&gart_oldmem_pfn_is_ram));
> +}
> +#else
> +static void exclude_from_vmcore(u64 aper_base, u32 aper_order)
> +{
> +}
> +#endif
> +
>  /* This code runs before the PCI subsystem is initialized, so just
>     access the northbridge directly. */
>  
> @@ -435,8 +457,10 @@ int __init gart_iommu_hole_init(void)
>  
>  out:
>  	if (!fix && !fallback_aper_force) {
> -		if (last_aper_base)
> +		if (last_aper_base) {
> +			exclude_from_vmcore(last_aper_base, last_aper_order);
>  			return 1;
> +		}
>  		return 0;
>  	}
>  
> @@ -473,6 +497,8 @@ int __init gart_iommu_hole_init(void)
>  		return 0;
>  	}
>  
> +	exclude_from_vmcore(aper_alloc, aper_order);
> +
>  	/* Fix up the north bridges */
>  	for (i = 0; i < amd_nb_bus_dev_ranges[i].dev_limit; i++) {
>  		int bus, dev_base, dev_limit;
> diff --git a/arch/x86/xen/mmu_hvm.c b/arch/x86/xen/mmu_hvm.c
> index 2cfcfe4f6b2a..dd2ad82eee80 100644
> --- a/arch/x86/xen/mmu_hvm.c
> +++ b/arch/x86/xen/mmu_hvm.c
> @@ -75,6 +75,6 @@ void __init xen_hvm_init_mmu_ops(void)
>  	if (is_pagetable_dying_supported())
>  		pv_mmu_ops.exit_mmap = xen_hvm_exit_mmap;
>  #ifdef CONFIG_PROC_VMCORE
> -	register_oldmem_pfn_is_ram(&xen_oldmem_pfn_is_ram);
> +	WARN_ON(register_oldmem_pfn_is_ram(&xen_oldmem_pfn_is_ram));
>  #endif
>  }
> -- 
> Jiri Bohac <jbohac at suse.cz>
> SUSE Labs, Prague, Czechia
> 


From bp at alien8.de  Sun Dec 17 13:47:35 2017
From: bp at alien8.de (Borislav Petkov)
Date: Sun, 17 Dec 2017 22:47:35 +0100
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
In-Reply-To: <20171216010142.GK12442@x1>
References: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
 <20171216010142.GK12442@x1>
Message-ID: <20171217214735.nuxq5zo2eknqpbpi@pd.tnic>

On Sat, Dec 16, 2017 at 09:01:42AM +0800, Baoquan He wrote:
> 2) If firmware is broken, you can't enable gart in firmware, will
> firmware engineer fix this since it's a firmware bug?

Slow down and get a reality check first please!

A firmware engineer will fix a 10yr old BIOS?!? Yeah right. And I'll get
a pink pony for Christmas. Geez.

We need a reliable way to tell the second kernel not to access the gart
range. And frankly, the best thing to do would be to teach the *second*
kernel to simply avoid the gart range. Regardless of what it gets told
by the ELF header. Because there are some ranges which it shouldn't
touch. Maybe we can reuse the gart detection code to do that in the
second kernel too.

But I haven't looked at it, might be hairy. Need to deal with this PTI
madness first.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.


From dyoung at redhat.com  Sun Dec 17 21:16:57 2017
From: dyoung at redhat.com (Dave Young)
Date: Mon, 18 Dec 2017 13:16:57 +0800
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
References: <CAFTCetQHmpprAVu6uYO+rc5Xi4EUVhmovbmSaU6nM1n1mAH62w@mail.gmail.com>
 <CAKv+Gu9oda1Ee8AoXsCEw+Bjn-XF3wZA_CsxvqhjtT6_bmJ7uA@mail.gmail.com>
 <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
Message-ID: <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>

kexec at fedoraproject... is for Fedora kexec scripts discussion, changed it
to kexec at lists.infradead.org

Also add linux-acpi list
On 12/18/17 at 02:31am, Bhupesh Sharma wrote:
> On Fri, Dec 15, 2017 at 3:05 PM, Ard Biesheuvel
> <ard.biesheuvel at linaro.org> wrote:
> > On 15 December 2017 at 09:59, AKASHI Takahiro
> > <takahiro.akashi at linaro.org> wrote:
> >> On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> >>> On 13 December 2017 at 12:16, AKASHI Takahiro
> >>> <takahiro.akashi at linaro.org> wrote:
> >>> > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> >>> >> On 13 December 2017 at 10:26, AKASHI Takahiro
> >>> >> <takahiro.akashi at linaro.org> wrote:
> >>> >> > Bhupesh, Ard,
> >>> >> >
> >>> >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> >>> >> >> Hi Ard, Akashi
> >>> >> >>
> >>> >> > (snip)
> >>> >> >
> >>> >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> >>> >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> >>> >> >> identify its own usable memory and exclude, at its boot time, any
> >>> >> >> other memory areas that are part of the panicked kernel's memory.
> >>> >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> >>> >> >> , for details)
> >>> >> >
> >>> >> > Right.
> >>> >> >
> >>> >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> >>> >> >> with the crashkernel memory range:
> >>> >> >>
> >>> >> >>                 /* add linux,usable-memory-range */
> >>> >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> >>> >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> >>> >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> >>> >> >>                                 address_cells, size_cells);
> >>> >> >>
> >>> >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> >>> >> >> , for details)
> >>> >> >>
> >>> >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> >>> >> >> they are marked as System RAM or as RESERVED. As,
> >>> >> >> 'linux,usable-memory-range' dt node is patched up only with
> >>> >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> >>> >> >>
> >>> >> >> 3). As a result when the crashkernel boots up it doesn't find this
> >>> >> >> ACPI memory and crashes while trying to access the same:
> >>> >> >>
> >>> >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> >>> >> >> -r`.img --reuse-cmdline -d
> >>> >> >>
> >>> >> >> [snip..]
> >>> >> >>
> >>> >> >> Reserved memory range
> >>> >> >> 000000000e800000-000000002e7fffff (0)
> >>> >> >>
> >>> >> >> Coredump memory ranges
> >>> >> >> 0000000000000000-000000000e7fffff (0)
> >>> >> >> 000000002e800000-000000003961ffff (0)
> >>> >> >> 0000000039d40000-000000003ed2ffff (0)
> >>> >> >> 000000003ed60000-000000003fbfffff (0)
> >>> >> >> 0000001040000000-0000001ffbffffff (0)
> >>> >> >> 0000002000000000-0000002ffbffffff (0)
> >>> >> >> 0000009000000000-0000009ffbffffff (0)
> >>> >> >> 000000a000000000-000000affbffffff (0)
> >>> >> >>
> >>> >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> >>> >> >> memory cap'ing passed to the crash kernel inside
> >>> >> >> 'arch/arm64/mm/init.c' (see below):
> >>> >> >>
> >>> >> >> static void __init fdt_enforce_memory_region(void)
> >>> >> >> {
> >>> >> >>         struct memblock_region reg = {
> >>> >> >>                 .size = 0,
> >>> >> >>         };
> >>> >> >>
> >>> >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >>> >> >>
> >>> >> >>         if (reg.size)
> >>> >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> >>> >> >> comment this out */
> >>> >> >> }
> >>> >> >
> >>> >> > Please just don't do that. It can cause a fatal damage on
> >>> >> > memory contents of the *crashed* kernel.
> >>> >> >
> >>> >> >> 5). Both the above temporary solutions fix the problem.
> >>> >> >>
> >>> >> >> 6). However exposing all System RAM regions to the crashkernel is not
> >>> >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> >>> >> >> fail.
> >>> >> >>
> >>> >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> >>> >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> >>> >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> >>> >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> >>> >> >> dt node 'linux,usable-memory-range'
> >>> >> >
> >>> >> > I still don't understand why we need to carry over the information
> >>> >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> >>> >> > such regions are free to be reused by the kernel after some point of
> >>> >> > initialization. Why does crash dump kernel need to know about them?
> >>> >> >
> >>> >>
> >>> >> Not really. According to the UEFI spec, they can be reclaimed after
> >>> >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> >>> >> no longer needs them. Of course, in order to be able to boot a kexec
> >>> >> kernel, those regions needs to be preserved, which is why they are
> >>> >> memblock_reserve()'d now.
> >>> >
> >>> > For my better understandings, who is actually accessing such regions
> >>> > during boot time, uefi itself or efistub?
> >>> >
> >>>
> >>> No, only the kernel. This is where the ACPI tables are stored. For
> >>> instance, on QEMU we have
> >>>
> >>>  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> >>>  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> >>>   01000013)
> >>>  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> >>> BXPC 00000001)
> >>>  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> >>> BXPC 00000001)
> >>>  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> >>> BXPC 00000001)
> >>>  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> >>> BXPC 00000001)
> >>>  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> >>> BXPC 00000001)
> >>>  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> >>> BXPC 00000001)
> >>>  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> >>> BXPC 00000001)
> >>>
> >>> covered by
> >>>
> >>>  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> >>>  ...
> >>>  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> >>
> >> OK. I mistakenly understood those regions could be freed after exiting
> >> UEFI boot services.
> >>
> >>>
> >>> >> So it seems that kexec does not honour the memblock_reserve() table
> >>> >> when booting the next kernel.
> >>> >
> >>> > not really.
> >>> >
> >>> >> > (In other words, can or should we skip some part of ACPI-related init code
> >>> >> > on crash dump kernel?)
> >>> >> >
> >>> >>
> >>> >> I don't think so. And the change to the handling of ACPI reclaim
> >>> >> regions only revealed the bug, not created it (given that other
> >>> >> memblock_reserve regions may be affected as well)
> >>> >
> >>> > As whether we should honor such reserved regions over kexec'ing
> >>> > depends on each one's specific nature, we will have to take care one-by-one.
> >>> > As a matter of fact, no information about "reserved" memblocks is
> >>> > exposed to user space (via proc/iomem).
> >>> >
> >>>
> >>> That is why I suggested (somewhere in this thread?) to not expose them
> >>> as 'System RAM'. Do you think that could solve this?
> >>
> >> Memblock-reserv'ing them is necessary to prevent their corruption and
> >> marking them under another name in /proc/iomem would also be good in order
> >> not to allocate them as part of crash kernel's memory.
> >>
> >
> > I agree. However, this may not be entirely trivial, since iterating
> > over the memblock_reserved table and creating iomem entries may result
> > in collisions.
> 
> I found a method (using the patch I shared earlier in this thread) to mark these
> entries as 'ACPI reclaim memory' ranges rather than System RAM or
> reserved regions.
> 
> >> But I'm not still convinced that we should export them in useable-
> >> memory-range to crash dump kernel. They will be accessed through
> >> acpi_os_map_memory() and so won't be required to be part of system ram
> >> (or memblocks), I guess.
> >
> > Agreed. They will be covered by the linear mapping in the boot kernel,
> > and be mapped explicitly via ioremap_cache() in the kexec kernel,
> > which is exactly what we want in this case.
> 
> Now this is what is confusing me. I don't see the above happening.
> 
> I see that the primary kernel boots up and adds the ACPI regions via:
> acpi_os_ioremap
>     -> ioremap_cache
> 
> But during the crashkernel boot, ''acpi_os_ioremap' calls
> 'ioremap' for the ACPI Reclaim Memory regions and not the _cache
> variant.
> 
> And it fails while accessing the ACPI tables:
> 
> [    0.039205] ACPI: Core revision 20170728
> pud=000000002e7d0003, *pmd=000000002e7c0003, *pte=00e8000039710707
> [    0.095098] Internal error: Oops: 96000021 [#1] SMP
> [    0.100022] Modules linked in:
> [    0.103102] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc6 #1
> [    0.109432] task: ffff000008d05180 task.stack: ffff000008cc0000
> [    0.115414] PC is at acpi_ns_lookup+0x25c/0x3c0
> [    0.119987] LR is at acpi_ds_load1_begin_op+0xa4/0x294
> [    0.125175] pc : [<ffff0000084a6764>] lr : [<ffff00000849b4f8>]
> pstate: 60000045
> [    0.132647] sp : ffff000008ccfb40
> [    0.135989] x29: ffff000008ccfb40 x28: ffff000008a9f2a4
> [    0.141354] x27: ffff0000088be820 x26: 0000000000000000
> [    0.146718] x25: 000000000000001b x24: 0000000000000001
> [    0.152083] x23: 0000000000000001 x22: ffff000009710027
> [    0.157447] x21: ffff000008ccfc50 x20: 0000000000000001
> [    0.162812] x19: 000000000000001b x18: 0000000000000005
> [    0.168176] x17: 0000000000000000 x16: 0000000000000000
> [    0.173541] x15: 0000000000000000 x14: 000000000000038e
> [    0.178905] x13: ffffffff00000000 x12: ffffffffffffffff
> [    0.184270] x11: 0000000000000006 x10: 00000000ffffff76
> [    0.189634] x9 : 000000000000005f x8 : ffff8000126d0140
> [    0.194998] x7 : 0000000000000000 x6 : ffff000008ccfc50
> [    0.200362] x5 : ffff80000fe62c00 x4 : 0000000000000001
> [    0.205727] x3 : ffff000008ccfbe0 x2 : ffff0000095e3980
> [    0.211091] x1 : ffff000009710027 x0 : 0000000000000000
> [    0.216456] Process swapper/0 (pid: 0, stack limit = 0xffff000008cc0000)
> [    0.223224] Call trace:
> [    0.225688] Exception stack(0xffff000008ccfa00 to 0xffff000008ccfb40)
> [    0.232194] fa00: 0000000000000000 ffff000009710027
> ffff0000095e3980 ffff000008ccfbe0
> [    0.240106] fa20: 0000000000000001 ffff80000fe62c00
> ffff000008ccfc50 0000000000000000
> [    0.248018] fa40: ffff8000126d0140 000000000000005f
> 00000000ffffff76 0000000000000006
> [    0.255931] fa60: ffffffffffffffff ffffffff00000000
> 000000000000038e 0000000000000000
> [    0.263843] fa80: 0000000000000000 0000000000000000
> 0000000000000005 000000000000001b
> [    0.271754] faa0: 0000000000000001 ffff000008ccfc50
> ffff000009710027 0000000000000001
> [    0.279667] fac0: 0000000000000001 000000000000001b
> 0000000000000000 ffff0000088be820
> [    0.287579] fae0: ffff000008a9f2a4 ffff000008ccfb40
> ffff00000849b4f8 ffff000008ccfb40
> [    0.295491] fb00: ffff0000084a6764 0000000060000045
> ffff000008ccfb40 ffff000008260a18
> [    0.303403] fb20: ffffffffffffffff ffff0000087f3fb0
> ffff000008ccfb40 ffff0000084a6764
> [    0.311316] [<ffff0000084a6764>] acpi_ns_lookup+0x25c/0x3c0
> [    0.316943] [<ffff00000849b4f8>] acpi_ds_load1_begin_op+0xa4/0x294
> [    0.323186] [<ffff0000084ad4ac>] acpi_ps_build_named_op+0xc4/0x198
> [    0.329428] [<ffff0000084ad6cc>] acpi_ps_create_op+0x14c/0x270
> [    0.335319] [<ffff0000084acfa8>] acpi_ps_parse_loop+0x188/0x5c8
> [    0.341298] [<ffff0000084ae048>] acpi_ps_parse_aml+0xb0/0x2b8
> [    0.347101] [<ffff0000084a8e10>] acpi_ns_one_complete_parse+0x144/0x184
> [    0.353783] [<ffff0000084a8e98>] acpi_ns_parse_table+0x48/0x68
> [    0.359675] [<ffff0000084a82cc>] acpi_ns_load_table+0x4c/0xdc
> [    0.365479] [<ffff0000084b32f8>] acpi_tb_load_namespace+0xe4/0x264
> [    0.371723] [<ffff000008baf9b4>] acpi_load_tables+0x48/0xc0
> [    0.377350] [<ffff000008badc20>] acpi_early_init+0x9c/0xd0
> [    0.382891] [<ffff000008b70d50>] start_kernel+0x3b4/0x43c
> [    0.388343] Code: b9008fb9 2a000318 36380054 32190318 (b94002c0)
> [    0.394500] ---[ end trace c46ed37f9651c58e ]---
> [    0.399160] Kernel panic - not syncing: Fatal exception
> [    0.404437] Rebooting in 10 seconds.
> 
> So, I think the linear mapping done by the primary kernel does not
> make these accessible in the crash kernel directly.
> 
> Any pointers?

Can you get the code line number for acpi_ns_lookup+0x25c?

> 
> Regards,
> Bhupesh
> 
> >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >> via a kernel command line parameter, "memmap=".
> >>
> _______________________________________________
> kexec mailing list -- kexec at lists.fedoraproject.org
> To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From dyoung at redhat.com  Sun Dec 17 21:43:22 2017
From: dyoung at redhat.com (Dave Young)
Date: Mon, 18 Dec 2017 13:43:22 +0800
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
References: <20171116070005.GI29552@linaro.org>
 <CAFTCetQHmpprAVu6uYO+rc5Xi4EUVhmovbmSaU6nM1n1mAH62w@mail.gmail.com>
 <CAKv+Gu9oda1Ee8AoXsCEw+Bjn-XF3wZA_CsxvqhjtT6_bmJ7uA@mail.gmail.com>
 <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
Message-ID: <20171218054322.GA6458@dhcp-128-65.nay.redhat.com>

Fix the kexec list address.

On 12/18/17 at 01:40pm, Dave Young wrote:
> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> > > On 13 December 2017 at 12:16, AKASHI Takahiro
> > > <takahiro.akashi at linaro.org> wrote:
> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
> > > >> <takahiro.akashi at linaro.org> wrote:
> > > >> > Bhupesh, Ard,
> > > >> >
> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> > > >> >> Hi Ard, Akashi
> > > >> >>
> > > >> > (snip)
> > > >> >
> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> > > >> >> identify its own usable memory and exclude, at its boot time, any
> > > >> >> other memory areas that are part of the panicked kernel's memory.
> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> > > >> >> , for details)
> > > >> >
> > > >> > Right.
> > > >> >
> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> > > >> >> with the crashkernel memory range:
> > > >> >>
> > > >> >>                 /* add linux,usable-memory-range */
> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> > > >> >>                                 address_cells, size_cells);
> > > >> >>
> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> > > >> >> , for details)
> > > >> >>
> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> > > >> >> they are marked as System RAM or as RESERVED. As,
> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> > > >> >>
> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
> > > >> >> ACPI memory and crashes while trying to access the same:
> > > >> >>
> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> > > >> >> -r`.img --reuse-cmdline -d
> > > >> >>
> > > >> >> [snip..]
> > > >> >>
> > > >> >> Reserved memory range
> > > >> >> 000000000e800000-000000002e7fffff (0)
> > > >> >>
> > > >> >> Coredump memory ranges
> > > >> >> 0000000000000000-000000000e7fffff (0)
> > > >> >> 000000002e800000-000000003961ffff (0)
> > > >> >> 0000000039d40000-000000003ed2ffff (0)
> > > >> >> 000000003ed60000-000000003fbfffff (0)
> > > >> >> 0000001040000000-0000001ffbffffff (0)
> > > >> >> 0000002000000000-0000002ffbffffff (0)
> > > >> >> 0000009000000000-0000009ffbffffff (0)
> > > >> >> 000000a000000000-000000affbffffff (0)
> > > >> >>
> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> > > >> >> memory cap'ing passed to the crash kernel inside
> > > >> >> 'arch/arm64/mm/init.c' (see below):
> > > >> >>
> > > >> >> static void __init fdt_enforce_memory_region(void)
> > > >> >> {
> > > >> >>         struct memblock_region reg = {
> > > >> >>                 .size = 0,
> > > >> >>         };
> > > >> >>
> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> > > >> >>
> > > >> >>         if (reg.size)
> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> > > >> >> comment this out */
> > > >> >> }
> > > >> >
> > > >> > Please just don't do that. It can cause a fatal damage on
> > > >> > memory contents of the *crashed* kernel.
> > > >> >
> > > >> >> 5). Both the above temporary solutions fix the problem.
> > > >> >>
> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> > > >> >> fail.
> > > >> >>
> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> > > >> >> dt node 'linux,usable-memory-range'
> > > >> >
> > > >> > I still don't understand why we need to carry over the information
> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> > > >> > such regions are free to be reused by the kernel after some point of
> > > >> > initialization. Why does crash dump kernel need to know about them?
> > > >> >
> > > >>
> > > >> Not really. According to the UEFI spec, they can be reclaimed after
> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> > > >> no longer needs them. Of course, in order to be able to boot a kexec
> > > >> kernel, those regions needs to be preserved, which is why they are
> > > >> memblock_reserve()'d now.
> > > >
> > > > For my better understandings, who is actually accessing such regions
> > > > during boot time, uefi itself or efistub?
> > > >
> > > 
> > > No, only the kernel. This is where the ACPI tables are stored. For
> > > instance, on QEMU we have
> > > 
> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> > >   01000013)
> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> > > BXPC 00000001)
> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> > > BXPC 00000001)
> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> > > BXPC 00000001)
> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> > > BXPC 00000001)
> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> > > BXPC 00000001)
> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> > > BXPC 00000001)
> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> > > BXPC 00000001)
> > > 
> > > covered by
> > > 
> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> > >  ...
> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> > 
> > OK. I mistakenly understood those regions could be freed after exiting
> > UEFI boot services.
> > 
> > > 
> > > >> So it seems that kexec does not honour the memblock_reserve() table
> > > >> when booting the next kernel.
> > > >
> > > > not really.
> > > >
> > > >> > (In other words, can or should we skip some part of ACPI-related init code
> > > >> > on crash dump kernel?)
> > > >> >
> > > >>
> > > >> I don't think so. And the change to the handling of ACPI reclaim
> > > >> regions only revealed the bug, not created it (given that other
> > > >> memblock_reserve regions may be affected as well)
> > > >
> > > > As whether we should honor such reserved regions over kexec'ing
> > > > depends on each one's specific nature, we will have to take care one-by-one.
> > > > As a matter of fact, no information about "reserved" memblocks is
> > > > exposed to user space (via proc/iomem).
> > > >
> > > 
> > > That is why I suggested (somewhere in this thread?) to not expose them
> > > as 'System RAM'. Do you think that could solve this?
> > 
> > Memblock-reserv'ing them is necessary to prevent their corruption and
> > marking them under another name in /proc/iomem would also be good in order
> > not to allocate them as part of crash kernel's memory.
> > 
> > But I'm not still convinced that we should export them in useable-
> > memory-range to crash dump kernel. They will be accessed through
> > acpi_os_map_memory() and so won't be required to be part of system ram
> > (or memblocks), I guess.
> > 	-> Bhupesh?
> 
> I forgot how arm64 kernel retrieve the memory ranges and initialize
> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
> the memory according to the efi memmap?  For kdump kernel anything other
> than usable memory (which is from the dt node instead) should be
> reinitialized according to efi passed info, no?
> 
> > 
> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> > via a kernel command line parameter, "memmap=".
> 
> memmap= is only used in old kexec-tools, now we are passing them via
> e820 table.
> 
> [snip]
> 
> Thanks
> Dave


From takahiro.akashi at linaro.org  Sun Dec 17 21:54:00 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon, 18 Dec 2017 14:54:00 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
References: <CAKv+Gu9oda1Ee8AoXsCEw+Bjn-XF3wZA_CsxvqhjtT6_bmJ7uA@mail.gmail.com>
 <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
Message-ID: <20171218055356.mq4coiibh3frii54@fireball>

On Mon, Dec 18, 2017 at 01:16:57PM +0800, Dave Young wrote:
> kexec at fedoraproject... is for Fedora kexec scripts discussion, changed it
> to kexec at lists.infradead.org
> 
> Also add linux-acpi list

Thank you.

> On 12/18/17 at 02:31am, Bhupesh Sharma wrote:
> > On Fri, Dec 15, 2017 at 3:05 PM, Ard Biesheuvel
> > <ard.biesheuvel at linaro.org> wrote:
> > > On 15 December 2017 at 09:59, AKASHI Takahiro
> > > <takahiro.akashi at linaro.org> wrote:
> > >> On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> > >>> On 13 December 2017 at 12:16, AKASHI Takahiro
> > >>> <takahiro.akashi at linaro.org> wrote:
> > >>> > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> > >>> >> On 13 December 2017 at 10:26, AKASHI Takahiro
> > >>> >> <takahiro.akashi at linaro.org> wrote:
> > >>> >> > Bhupesh, Ard,
> > >>> >> >
> > >>> >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> > >>> >> >> Hi Ard, Akashi
> > >>> >> >>
> > >>> >> > (snip)
> > >>> >> >
> > >>> >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> > >>> >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> > >>> >> >> identify its own usable memory and exclude, at its boot time, any
> > >>> >> >> other memory areas that are part of the panicked kernel's memory.
> > >>> >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> > >>> >> >> , for details)
> > >>> >> >
> > >>> >> > Right.
> > >>> >> >
> > >>> >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> > >>> >> >> with the crashkernel memory range:
> > >>> >> >>
> > >>> >> >>                 /* add linux,usable-memory-range */
> > >>> >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> > >>> >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> > >>> >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> > >>> >> >>                                 address_cells, size_cells);
> > >>> >> >>
> > >>> >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> > >>> >> >> , for details)
> > >>> >> >>
> > >>> >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> > >>> >> >> they are marked as System RAM or as RESERVED. As,
> > >>> >> >> 'linux,usable-memory-range' dt node is patched up only with
> > >>> >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> > >>> >> >>
> > >>> >> >> 3). As a result when the crashkernel boots up it doesn't find this
> > >>> >> >> ACPI memory and crashes while trying to access the same:
> > >>> >> >>
> > >>> >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> > >>> >> >> -r`.img --reuse-cmdline -d
> > >>> >> >>
> > >>> >> >> [snip..]
> > >>> >> >>
> > >>> >> >> Reserved memory range
> > >>> >> >> 000000000e800000-000000002e7fffff (0)
> > >>> >> >>
> > >>> >> >> Coredump memory ranges
> > >>> >> >> 0000000000000000-000000000e7fffff (0)
> > >>> >> >> 000000002e800000-000000003961ffff (0)
> > >>> >> >> 0000000039d40000-000000003ed2ffff (0)
> > >>> >> >> 000000003ed60000-000000003fbfffff (0)
> > >>> >> >> 0000001040000000-0000001ffbffffff (0)
> > >>> >> >> 0000002000000000-0000002ffbffffff (0)
> > >>> >> >> 0000009000000000-0000009ffbffffff (0)
> > >>> >> >> 000000a000000000-000000affbffffff (0)
> > >>> >> >>
> > >>> >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> > >>> >> >> memory cap'ing passed to the crash kernel inside
> > >>> >> >> 'arch/arm64/mm/init.c' (see below):
> > >>> >> >>
> > >>> >> >> static void __init fdt_enforce_memory_region(void)
> > >>> >> >> {
> > >>> >> >>         struct memblock_region reg = {
> > >>> >> >>                 .size = 0,
> > >>> >> >>         };
> > >>> >> >>
> > >>> >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> > >>> >> >>
> > >>> >> >>         if (reg.size)
> > >>> >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> > >>> >> >> comment this out */
> > >>> >> >> }
> > >>> >> >
> > >>> >> > Please just don't do that. It can cause a fatal damage on
> > >>> >> > memory contents of the *crashed* kernel.
> > >>> >> >
> > >>> >> >> 5). Both the above temporary solutions fix the problem.
> > >>> >> >>
> > >>> >> >> 6). However exposing all System RAM regions to the crashkernel is not
> > >>> >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> > >>> >> >> fail.
> > >>> >> >>
> > >>> >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> > >>> >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> > >>> >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> > >>> >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> > >>> >> >> dt node 'linux,usable-memory-range'
> > >>> >> >
> > >>> >> > I still don't understand why we need to carry over the information
> > >>> >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> > >>> >> > such regions are free to be reused by the kernel after some point of
> > >>> >> > initialization. Why does crash dump kernel need to know about them?
> > >>> >> >
> > >>> >>
> > >>> >> Not really. According to the UEFI spec, they can be reclaimed after
> > >>> >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> > >>> >> no longer needs them. Of course, in order to be able to boot a kexec
> > >>> >> kernel, those regions needs to be preserved, which is why they are
> > >>> >> memblock_reserve()'d now.
> > >>> >
> > >>> > For my better understandings, who is actually accessing such regions
> > >>> > during boot time, uefi itself or efistub?
> > >>> >
> > >>>
> > >>> No, only the kernel. This is where the ACPI tables are stored. For
> > >>> instance, on QEMU we have
> > >>>
> > >>>  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> > >>>  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> > >>>   01000013)
> > >>>  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> > >>> BXPC 00000001)
> > >>>  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> > >>> BXPC 00000001)
> > >>>  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> > >>> BXPC 00000001)
> > >>>  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> > >>> BXPC 00000001)
> > >>>  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> > >>> BXPC 00000001)
> > >>>  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> > >>> BXPC 00000001)
> > >>>  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> > >>> BXPC 00000001)
> > >>>
> > >>> covered by
> > >>>
> > >>>  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> > >>>  ...
> > >>>  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> > >>
> > >> OK. I mistakenly understood those regions could be freed after exiting
> > >> UEFI boot services.
> > >>
> > >>>
> > >>> >> So it seems that kexec does not honour the memblock_reserve() table
> > >>> >> when booting the next kernel.
> > >>> >
> > >>> > not really.
> > >>> >
> > >>> >> > (In other words, can or should we skip some part of ACPI-related init code
> > >>> >> > on crash dump kernel?)
> > >>> >> >
> > >>> >>
> > >>> >> I don't think so. And the change to the handling of ACPI reclaim
> > >>> >> regions only revealed the bug, not created it (given that other
> > >>> >> memblock_reserve regions may be affected as well)
> > >>> >
> > >>> > As whether we should honor such reserved regions over kexec'ing
> > >>> > depends on each one's specific nature, we will have to take care one-by-one.
> > >>> > As a matter of fact, no information about "reserved" memblocks is
> > >>> > exposed to user space (via proc/iomem).
> > >>> >
> > >>>
> > >>> That is why I suggested (somewhere in this thread?) to not expose them
> > >>> as 'System RAM'. Do you think that could solve this?
> > >>
> > >> Memblock-reserv'ing them is necessary to prevent their corruption and
> > >> marking them under another name in /proc/iomem would also be good in order
> > >> not to allocate them as part of crash kernel's memory.
> > >>
> > >
> > > I agree. However, this may not be entirely trivial, since iterating
> > > over the memblock_reserved table and creating iomem entries may result
> > > in collisions.
> > 
> > I found a method (using the patch I shared earlier in this thread) to mark these
> > entries as 'ACPI reclaim memory' ranges rather than System RAM or
> > reserved regions.
> > 
> > >> But I'm not still convinced that we should export them in useable-
> > >> memory-range to crash dump kernel. They will be accessed through
> > >> acpi_os_map_memory() and so won't be required to be part of system ram
> > >> (or memblocks), I guess.
> > >
> > > Agreed. They will be covered by the linear mapping in the boot kernel,
> > > and be mapped explicitly via ioremap_cache() in the kexec kernel,
> > > which is exactly what we want in this case.
> > 
> > Now this is what is confusing me. I don't see the above happening.
> > 
> > I see that the primary kernel boots up and adds the ACPI regions via:
> > acpi_os_ioremap
> >     -> ioremap_cache
> > 
> > But during the crashkernel boot, ''acpi_os_ioremap' calls
> > 'ioremap' for the ACPI Reclaim Memory regions and not the _cache
> > variant.

It is natural if that region is out of memblocks.

> > And it fails while accessing the ACPI tables:
> > 
> > [    0.039205] ACPI: Core revision 20170728
> > pud=000000002e7d0003, *pmd=000000002e7c0003, *pte=00e8000039710707
> > [    0.095098] Internal error: Oops: 96000021 [#1] SMP

this (ESR = 0x96000021) means that Data Abort and Alignment fault happened.
As ioremap() makes the mapping as "Device memory", unaligned memory
access won't be allowed.

> > [    0.100022] Modules linked in:
> > [    0.103102] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc6 #1
> > [    0.109432] task: ffff000008d05180 task.stack: ffff000008cc0000
> > [    0.115414] PC is at acpi_ns_lookup+0x25c/0x3c0
> > [    0.119987] LR is at acpi_ds_load1_begin_op+0xa4/0x294
> > [    0.125175] pc : [<ffff0000084a6764>] lr : [<ffff00000849b4f8>]
> > pstate: 60000045
> > [    0.132647] sp : ffff000008ccfb40
> > [    0.135989] x29: ffff000008ccfb40 x28: ffff000008a9f2a4
> > [    0.141354] x27: ffff0000088be820 x26: 0000000000000000
> > [    0.146718] x25: 000000000000001b x24: 0000000000000001
> > [    0.152083] x23: 0000000000000001 x22: ffff000009710027
> > [    0.157447] x21: ffff000008ccfc50 x20: 0000000000000001
> > [    0.162812] x19: 000000000000001b x18: 0000000000000005
> > [    0.168176] x17: 0000000000000000 x16: 0000000000000000
> > [    0.173541] x15: 0000000000000000 x14: 000000000000038e
> > [    0.178905] x13: ffffffff00000000 x12: ffffffffffffffff
> > [    0.184270] x11: 0000000000000006 x10: 00000000ffffff76
> > [    0.189634] x9 : 000000000000005f x8 : ffff8000126d0140
> > [    0.194998] x7 : 0000000000000000 x6 : ffff000008ccfc50
> > [    0.200362] x5 : ffff80000fe62c00 x4 : 0000000000000001
> > [    0.205727] x3 : ffff000008ccfbe0 x2 : ffff0000095e3980
> > [    0.211091] x1 : ffff000009710027 x0 : 0000000000000000
> > [    0.216456] Process swapper/0 (pid: 0, stack limit = 0xffff000008cc0000)
> > [    0.223224] Call trace:
> > [    0.225688] Exception stack(0xffff000008ccfa00 to 0xffff000008ccfb40)
> > [    0.232194] fa00: 0000000000000000 ffff000009710027
> > ffff0000095e3980 ffff000008ccfbe0
> > [    0.240106] fa20: 0000000000000001 ffff80000fe62c00
> > ffff000008ccfc50 0000000000000000
> > [    0.248018] fa40: ffff8000126d0140 000000000000005f
> > 00000000ffffff76 0000000000000006
> > [    0.255931] fa60: ffffffffffffffff ffffffff00000000
> > 000000000000038e 0000000000000000
> > [    0.263843] fa80: 0000000000000000 0000000000000000
> > 0000000000000005 000000000000001b
> > [    0.271754] faa0: 0000000000000001 ffff000008ccfc50
> > ffff000009710027 0000000000000001
> > [    0.279667] fac0: 0000000000000001 000000000000001b
> > 0000000000000000 ffff0000088be820
> > [    0.287579] fae0: ffff000008a9f2a4 ffff000008ccfb40
> > ffff00000849b4f8 ffff000008ccfb40
> > [    0.295491] fb00: ffff0000084a6764 0000000060000045
> > ffff000008ccfb40 ffff000008260a18
> > [    0.303403] fb20: ffffffffffffffff ffff0000087f3fb0
> > ffff000008ccfb40 ffff0000084a6764
> > [    0.311316] [<ffff0000084a6764>] acpi_ns_lookup+0x25c/0x3c0
> > [    0.316943] [<ffff00000849b4f8>] acpi_ds_load1_begin_op+0xa4/0x294
> > [    0.323186] [<ffff0000084ad4ac>] acpi_ps_build_named_op+0xc4/0x198
> > [    0.329428] [<ffff0000084ad6cc>] acpi_ps_create_op+0x14c/0x270
> > [    0.335319] [<ffff0000084acfa8>] acpi_ps_parse_loop+0x188/0x5c8
> > [    0.341298] [<ffff0000084ae048>] acpi_ps_parse_aml+0xb0/0x2b8
> > [    0.347101] [<ffff0000084a8e10>] acpi_ns_one_complete_parse+0x144/0x184
> > [    0.353783] [<ffff0000084a8e98>] acpi_ns_parse_table+0x48/0x68
> > [    0.359675] [<ffff0000084a82cc>] acpi_ns_load_table+0x4c/0xdc
> > [    0.365479] [<ffff0000084b32f8>] acpi_tb_load_namespace+0xe4/0x264
> > [    0.371723] [<ffff000008baf9b4>] acpi_load_tables+0x48/0xc0
> > [    0.377350] [<ffff000008badc20>] acpi_early_init+0x9c/0xd0
> > [    0.382891] [<ffff000008b70d50>] start_kernel+0x3b4/0x43c
> > [    0.388343] Code: b9008fb9 2a000318 36380054 32190318 (b94002c0)
> > [    0.394500] ---[ end trace c46ed37f9651c58e ]---
> > [    0.399160] Kernel panic - not syncing: Fatal exception
> > [    0.404437] Rebooting in 10 seconds.
> > 
> > So, I think the linear mapping done by the primary kernel does not
> > make these accessible in the crash kernel directly.
> > 
> > Any pointers?
> 
> Can you get the code line number for acpi_ns_lookup+0x25c?

So should we always avoid ioremap() in acpi_os_ioremap() entirely, or
modify acpi_ns_lookup() (or any acpi functions') to prevent unaligned
accesses?
(I didn't find out how unaligned accesses could happen there.)

Thanks,
-Takahiro AKASHI

> > 
> > Regards,
> > Bhupesh
> > 
> > >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> > >> via a kernel command line parameter, "memmap=".
> > >>
> > _______________________________________________
> > kexec mailing list -- kexec at lists.fedoraproject.org
> > To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From bhupesh.linux at gmail.com  Mon Dec 18 00:59:05 2017
From: bhupesh.linux at gmail.com (Bhupesh SHARMA)
Date: Mon, 18 Dec 2017 14:29:05 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171218055356.mq4coiibh3frii54@fireball>
References: <CAKv+Gu9oda1Ee8AoXsCEw+Bjn-XF3wZA_CsxvqhjtT6_bmJ7uA@mail.gmail.com>
 <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
 <20171218055356.mq4coiibh3frii54@fireball>
Message-ID: <CAFTCetQ55zUKe25jSku0DHp8uVZA4hB32d5W6MSCNsTVpxu7Gw@mail.gmail.com>

On Mon, Dec 18, 2017 at 11:24 AM, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
> On Mon, Dec 18, 2017 at 01:16:57PM +0800, Dave Young wrote:
>> kexec at fedoraproject... is for Fedora kexec scripts discussion, changed it
>> to kexec at lists.infradead.org
>>
>> Also add linux-acpi list
>
> Thank you.
>
>> On 12/18/17 at 02:31am, Bhupesh Sharma wrote:
>> > On Fri, Dec 15, 2017 at 3:05 PM, Ard Biesheuvel
>> > <ard.biesheuvel at linaro.org> wrote:
>> > > On 15 December 2017 at 09:59, AKASHI Takahiro
>> > > <takahiro.akashi at linaro.org> wrote:
>> > >> On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
>> > >>> On 13 December 2017 at 12:16, AKASHI Takahiro
>> > >>> <takahiro.akashi at linaro.org> wrote:
>> > >>> > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
>> > >>> >> On 13 December 2017 at 10:26, AKASHI Takahiro
>> > >>> >> <takahiro.akashi at linaro.org> wrote:
>> > >>> >> > Bhupesh, Ard,
>> > >>> >> >
>> > >>> >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
>> > >>> >> >> Hi Ard, Akashi
>> > >>> >> >>
>> > >>> >> > (snip)
>> > >>> >> >
>> > >>> >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
>> > >>> >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
>> > >>> >> >> identify its own usable memory and exclude, at its boot time, any
>> > >>> >> >> other memory areas that are part of the panicked kernel's memory.
>> > >>> >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
>> > >>> >> >> , for details)
>> > >>> >> >
>> > >>> >> > Right.
>> > >>> >> >
>> > >>> >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
>> > >>> >> >> with the crashkernel memory range:
>> > >>> >> >>
>> > >>> >> >>                 /* add linux,usable-memory-range */
>> > >>> >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
>> > >>> >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
>> > >>> >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
>> > >>> >> >>                                 address_cells, size_cells);
>> > >>> >> >>
>> > >>> >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
>> > >>> >> >> , for details)
>> > >>> >> >>
>> > >>> >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
>> > >>> >> >> they are marked as System RAM or as RESERVED. As,
>> > >>> >> >> 'linux,usable-memory-range' dt node is patched up only with
>> > >>> >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
>> > >>> >> >>
>> > >>> >> >> 3). As a result when the crashkernel boots up it doesn't find this
>> > >>> >> >> ACPI memory and crashes while trying to access the same:
>> > >>> >> >>
>> > >>> >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
>> > >>> >> >> -r`.img --reuse-cmdline -d
>> > >>> >> >>
>> > >>> >> >> [snip..]
>> > >>> >> >>
>> > >>> >> >> Reserved memory range
>> > >>> >> >> 000000000e800000-000000002e7fffff (0)
>> > >>> >> >>
>> > >>> >> >> Coredump memory ranges
>> > >>> >> >> 0000000000000000-000000000e7fffff (0)
>> > >>> >> >> 000000002e800000-000000003961ffff (0)
>> > >>> >> >> 0000000039d40000-000000003ed2ffff (0)
>> > >>> >> >> 000000003ed60000-000000003fbfffff (0)
>> > >>> >> >> 0000001040000000-0000001ffbffffff (0)
>> > >>> >> >> 0000002000000000-0000002ffbffffff (0)
>> > >>> >> >> 0000009000000000-0000009ffbffffff (0)
>> > >>> >> >> 000000a000000000-000000affbffffff (0)
>> > >>> >> >>
>> > >>> >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
>> > >>> >> >> memory cap'ing passed to the crash kernel inside
>> > >>> >> >> 'arch/arm64/mm/init.c' (see below):
>> > >>> >> >>
>> > >>> >> >> static void __init fdt_enforce_memory_region(void)
>> > >>> >> >> {
>> > >>> >> >>         struct memblock_region reg = {
>> > >>> >> >>                 .size = 0,
>> > >>> >> >>         };
>> > >>> >> >>
>> > >>> >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> > >>> >> >>
>> > >>> >> >>         if (reg.size)
>> > >>> >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
>> > >>> >> >> comment this out */
>> > >>> >> >> }
>> > >>> >> >
>> > >>> >> > Please just don't do that. It can cause a fatal damage on
>> > >>> >> > memory contents of the *crashed* kernel.
>> > >>> >> >
>> > >>> >> >> 5). Both the above temporary solutions fix the problem.
>> > >>> >> >>
>> > >>> >> >> 6). However exposing all System RAM regions to the crashkernel is not
>> > >>> >> >> advisable and may cause the crashkernel or some crashkernel drivers to
>> > >>> >> >> fail.
>> > >>> >> >>
>> > >>> >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
>> > >>> >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
>> > >>> >> >> kernel code and on the other hand the user-space 'kexec-tools' will
>> > >>> >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
>> > >>> >> >> dt node 'linux,usable-memory-range'
>> > >>> >> >
>> > >>> >> > I still don't understand why we need to carry over the information
>> > >>> >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
>> > >>> >> > such regions are free to be reused by the kernel after some point of
>> > >>> >> > initialization. Why does crash dump kernel need to know about them?
>> > >>> >> >
>> > >>> >>
>> > >>> >> Not really. According to the UEFI spec, they can be reclaimed after
>> > >>> >> the OS has initialized, i.e., when it has consumed the ACPI tables and
>> > >>> >> no longer needs them. Of course, in order to be able to boot a kexec
>> > >>> >> kernel, those regions needs to be preserved, which is why they are
>> > >>> >> memblock_reserve()'d now.
>> > >>> >
>> > >>> > For my better understandings, who is actually accessing such regions
>> > >>> > during boot time, uefi itself or efistub?
>> > >>> >
>> > >>>
>> > >>> No, only the kernel. This is where the ACPI tables are stored. For
>> > >>> instance, on QEMU we have
>> > >>>
>> > >>>  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
>> > >>>  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
>> > >>>   01000013)
>> > >>>  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
>> > >>> BXPC 00000001)
>> > >>>  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
>> > >>> BXPC 00000001)
>> > >>>  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
>> > >>> BXPC 00000001)
>> > >>>  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
>> > >>> BXPC 00000001)
>> > >>>  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
>> > >>> BXPC 00000001)
>> > >>>  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
>> > >>> BXPC 00000001)
>> > >>>  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
>> > >>> BXPC 00000001)
>> > >>>
>> > >>> covered by
>> > >>>
>> > >>>  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
>> > >>>  ...
>> > >>>  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
>> > >>
>> > >> OK. I mistakenly understood those regions could be freed after exiting
>> > >> UEFI boot services.
>> > >>
>> > >>>
>> > >>> >> So it seems that kexec does not honour the memblock_reserve() table
>> > >>> >> when booting the next kernel.
>> > >>> >
>> > >>> > not really.
>> > >>> >
>> > >>> >> > (In other words, can or should we skip some part of ACPI-related init code
>> > >>> >> > on crash dump kernel?)
>> > >>> >> >
>> > >>> >>
>> > >>> >> I don't think so. And the change to the handling of ACPI reclaim
>> > >>> >> regions only revealed the bug, not created it (given that other
>> > >>> >> memblock_reserve regions may be affected as well)
>> > >>> >
>> > >>> > As whether we should honor such reserved regions over kexec'ing
>> > >>> > depends on each one's specific nature, we will have to take care one-by-one.
>> > >>> > As a matter of fact, no information about "reserved" memblocks is
>> > >>> > exposed to user space (via proc/iomem).
>> > >>> >
>> > >>>
>> > >>> That is why I suggested (somewhere in this thread?) to not expose them
>> > >>> as 'System RAM'. Do you think that could solve this?
>> > >>
>> > >> Memblock-reserv'ing them is necessary to prevent their corruption and
>> > >> marking them under another name in /proc/iomem would also be good in order
>> > >> not to allocate them as part of crash kernel's memory.
>> > >>
>> > >
>> > > I agree. However, this may not be entirely trivial, since iterating
>> > > over the memblock_reserved table and creating iomem entries may result
>> > > in collisions.
>> >
>> > I found a method (using the patch I shared earlier in this thread) to mark these
>> > entries as 'ACPI reclaim memory' ranges rather than System RAM or
>> > reserved regions.
>> >
>> > >> But I'm not still convinced that we should export them in useable-
>> > >> memory-range to crash dump kernel. They will be accessed through
>> > >> acpi_os_map_memory() and so won't be required to be part of system ram
>> > >> (or memblocks), I guess.
>> > >
>> > > Agreed. They will be covered by the linear mapping in the boot kernel,
>> > > and be mapped explicitly via ioremap_cache() in the kexec kernel,
>> > > which is exactly what we want in this case.
>> >
>> > Now this is what is confusing me. I don't see the above happening.
>> >
>> > I see that the primary kernel boots up and adds the ACPI regions via:
>> > acpi_os_ioremap
>> >     -> ioremap_cache
>> >
>> > But during the crashkernel boot, ''acpi_os_ioremap' calls
>> > 'ioremap' for the ACPI Reclaim Memory regions and not the _cache
>> > variant.
>
> It is natural if that region is out of memblocks.

Thanks for the confirmation. This was my understanding as well.

>> > And it fails while accessing the ACPI tables:
>> >
>> > [    0.039205] ACPI: Core revision 20170728
>> > pud=000000002e7d0003, *pmd=000000002e7c0003, *pte=00e8000039710707
>> > [    0.095098] Internal error: Oops: 96000021 [#1] SMP
>
> this (ESR = 0x96000021) means that Data Abort and Alignment fault happened.
> As ioremap() makes the mapping as "Device memory", unaligned memory
> access won't be allowed.
>
>> > [    0.100022] Modules linked in:
>> > [    0.103102] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc6 #1
>> > [    0.109432] task: ffff000008d05180 task.stack: ffff000008cc0000
>> > [    0.115414] PC is at acpi_ns_lookup+0x25c/0x3c0
>> > [    0.119987] LR is at acpi_ds_load1_begin_op+0xa4/0x294
>> > [    0.125175] pc : [<ffff0000084a6764>] lr : [<ffff00000849b4f8>]
>> > pstate: 60000045
>> > [    0.132647] sp : ffff000008ccfb40
>> > [    0.135989] x29: ffff000008ccfb40 x28: ffff000008a9f2a4
>> > [    0.141354] x27: ffff0000088be820 x26: 0000000000000000
>> > [    0.146718] x25: 000000000000001b x24: 0000000000000001
>> > [    0.152083] x23: 0000000000000001 x22: ffff000009710027
>> > [    0.157447] x21: ffff000008ccfc50 x20: 0000000000000001
>> > [    0.162812] x19: 000000000000001b x18: 0000000000000005
>> > [    0.168176] x17: 0000000000000000 x16: 0000000000000000
>> > [    0.173541] x15: 0000000000000000 x14: 000000000000038e
>> > [    0.178905] x13: ffffffff00000000 x12: ffffffffffffffff
>> > [    0.184270] x11: 0000000000000006 x10: 00000000ffffff76
>> > [    0.189634] x9 : 000000000000005f x8 : ffff8000126d0140
>> > [    0.194998] x7 : 0000000000000000 x6 : ffff000008ccfc50
>> > [    0.200362] x5 : ffff80000fe62c00 x4 : 0000000000000001
>> > [    0.205727] x3 : ffff000008ccfbe0 x2 : ffff0000095e3980
>> > [    0.211091] x1 : ffff000009710027 x0 : 0000000000000000
>> > [    0.216456] Process swapper/0 (pid: 0, stack limit = 0xffff000008cc0000)
>> > [    0.223224] Call trace:
>> > [    0.225688] Exception stack(0xffff000008ccfa00 to 0xffff000008ccfb40)
>> > [    0.232194] fa00: 0000000000000000 ffff000009710027
>> > ffff0000095e3980 ffff000008ccfbe0
>> > [    0.240106] fa20: 0000000000000001 ffff80000fe62c00
>> > ffff000008ccfc50 0000000000000000
>> > [    0.248018] fa40: ffff8000126d0140 000000000000005f
>> > 00000000ffffff76 0000000000000006
>> > [    0.255931] fa60: ffffffffffffffff ffffffff00000000
>> > 000000000000038e 0000000000000000
>> > [    0.263843] fa80: 0000000000000000 0000000000000000
>> > 0000000000000005 000000000000001b
>> > [    0.271754] faa0: 0000000000000001 ffff000008ccfc50
>> > ffff000009710027 0000000000000001
>> > [    0.279667] fac0: 0000000000000001 000000000000001b
>> > 0000000000000000 ffff0000088be820
>> > [    0.287579] fae0: ffff000008a9f2a4 ffff000008ccfb40
>> > ffff00000849b4f8 ffff000008ccfb40
>> > [    0.295491] fb00: ffff0000084a6764 0000000060000045
>> > ffff000008ccfb40 ffff000008260a18
>> > [    0.303403] fb20: ffffffffffffffff ffff0000087f3fb0
>> > ffff000008ccfb40 ffff0000084a6764
>> > [    0.311316] [<ffff0000084a6764>] acpi_ns_lookup+0x25c/0x3c0
>> > [    0.316943] [<ffff00000849b4f8>] acpi_ds_load1_begin_op+0xa4/0x294
>> > [    0.323186] [<ffff0000084ad4ac>] acpi_ps_build_named_op+0xc4/0x198
>> > [    0.329428] [<ffff0000084ad6cc>] acpi_ps_create_op+0x14c/0x270
>> > [    0.335319] [<ffff0000084acfa8>] acpi_ps_parse_loop+0x188/0x5c8
>> > [    0.341298] [<ffff0000084ae048>] acpi_ps_parse_aml+0xb0/0x2b8
>> > [    0.347101] [<ffff0000084a8e10>] acpi_ns_one_complete_parse+0x144/0x184
>> > [    0.353783] [<ffff0000084a8e98>] acpi_ns_parse_table+0x48/0x68
>> > [    0.359675] [<ffff0000084a82cc>] acpi_ns_load_table+0x4c/0xdc
>> > [    0.365479] [<ffff0000084b32f8>] acpi_tb_load_namespace+0xe4/0x264
>> > [    0.371723] [<ffff000008baf9b4>] acpi_load_tables+0x48/0xc0
>> > [    0.377350] [<ffff000008badc20>] acpi_early_init+0x9c/0xd0
>> > [    0.382891] [<ffff000008b70d50>] start_kernel+0x3b4/0x43c
>> > [    0.388343] Code: b9008fb9 2a000318 36380054 32190318 (b94002c0)
>> > [    0.394500] ---[ end trace c46ed37f9651c58e ]---
>> > [    0.399160] Kernel panic - not syncing: Fatal exception
>> > [    0.404437] Rebooting in 10 seconds.
>> >
>> > So, I think the linear mapping done by the primary kernel does not
>> > make these accessible in the crash kernel directly.
>> >
>> > Any pointers?
>>
>> Can you get the code line number for acpi_ns_lookup+0x25c?
>
> So should we always avoid ioremap() in acpi_os_ioremap() entirely, or
> modify acpi_ns_lookup() (or any acpi functions') to prevent unaligned
> accesses?
> (I didn't find out how unaligned accesses could happen there.)
>

Right. Like I captured somewhere in this thread (perhaps the first
email on this subject),
this is indeed an unaligned address access.

Now, modifying acpi_os_ioremap() to not ioremap() and thus avoiding
assigning this memory range
as device memory doesn't seem a neat solution as it means we are not
marking some thing with the right memory attribute and we can fall in
similar/related issues later.

Regarding the later suggestion, what I am seeing now is that the acpi
table access functions are perhaps reused from the earlier x86
implementation, but on the arm64 (or even arm) arch we should not be
allowing unaligned accesses which might cause UNDEFINED behaviour and
resultant crash.

So I can try going this approach and see if it works for me.

However, I am still not very sure as to why the crashkernel ranges
historically do not include the System RAM regions (which may include
the ACPI regions as well). These regions are available for the kernel
usage and perhaps should be exported to the crashkernel as well.

I am not fully aware of the previous discussions on capp'ing the
crashkernel memory being passed to the kdump kernel, but did we run
into any issues while doing so?

Also, even if I extend the kexec-tools to modify the
linux,usable-memory-range and add the ACPI regions to it, the
crashkernel fails to boot with the below message (I have added some
logic to print the DTB on the crash kernel boot start):

[    0.000000]     chosen {
[    0.000000]         linux,usable-memory-range
[    0.000000]  = <
[    0.000000] 0x00000000
[    0.000000] 0x0e800000
[    0.000000] 0x00000000
[    0.000000] 0x20000000
[    0.000000] 0x00000000
[    0.000000] 0x396c0000
[    0.000000] 0x00000000
[    0.000000] 0x000a0000
[    0.000000] 0x00000000
[    0.000000] 0x39770000
[    0.000000] 0x00000000
[    0.000000] 0x00040000
[    0.000000] 0x00000000
[    0.000000] 0x398a0000
[    0.000000] 0x00000000
[    0.000000] 0x00020000
[    0.000000] >
[    0.000000] ;

[snip..]

[    0.000000] linux,usable-memory-range base e800000, size 20000000
[    0.000000]  - e800000 ,  20000000
[    0.000000] linux,usable-memory-range base 396c0000, size a0000
[    0.000000]  - 396c0000 ,  a0000
[    0.000000] linux,usable-memory-range base 39770000, size 40000
[    0.000000]  - 39770000 ,  40000
[    0.000000] linux,usable-memory-range base 398a0000, size 20000
[    0.000000]  - 398a0000 ,  20000
[    0.000000] initrd not fully accessible via the linear mapping --
please check your bootloader ...
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at arch/arm64/mm/init.c:597
arm64_memblock_init+0x210/0x484
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0+ #7
[    0.000000] task: ffff000008d05580 task.stack: ffff000008cc0000
[    0.000000] PC is at arm64_memblock_init+0x210/0x484
[    0.000000] LR is at arm64_memblock_init+0x210/0x484
[    0.000000] pc : [<ffff000008b76984>] lr : [<ffff000008b76984>]
pstate: 600000c5
[    0.000000] sp : ffff000008ccfe80
[    0.000000] x29: ffff000008ccfe80 x28: 000000000f370018
[    0.000000] x27: 0000000011230000 x26: 00000000013b0000
[    0.000000] x25: 000000000fe80000 x24: ffff000008cf3000
[    0.000000] x23: ffff000008ec0000 x22: ffff000009680000
[    0.000000] x21: ffff000008afa000 x20: ffff000008080000
[    0.000000] x19: ffff000008afa000 x18: 000000000c283806
[    0.000000] x17: 0000000000000000 x16: ffff000008d05580
[    0.000000] x15: 000000002be00842 x14: 79206b6365686320
[    0.000000] x13: 657361656c70202d x12: 2d20676e69707061
[    0.000000] x11: 6d207261656e696c x10: 2065687420616976
[    0.000000] x9 : 00000000000000f4 x8 : ffff000008517414
[    0.000000] x7 : 746f6f622072756f x6 : 000000000000000d
[    0.000000] x5 : ffff000008c96360 x4 : 0000000000000001
[    0.000000] x3 : 0000000000000000 x2 : 0000000000000000
[    0.000000] x1 : 0000000000000000 x0 : 0000000000000056
[    0.000000] Call trace:
[    0.000000] Exception stack(0xffff000008ccfd40 to 0xffff000008ccfe80)
[    0.000000] fd40: 0000000000000056 0000000000000000
0000000000000000 0000000000000000
[    0.000000] fd60: 0000000000000001 ffff000008c96360
000000000000000d 746f6f622072756f
[    0.000000] fd80: ffff000008517414 00000000000000f4
2065687420616976 6d207261656e696c
[    0.000000] fda0: 2d20676e69707061 657361656c70202d
79206b6365686320 000000002be00842
[    0.000000] fdc0: ffff000008d05580 0000000000000000
000000000c283806 ffff000008afa000
[    0.000000] fde0: ffff000008080000 ffff000008afa000
ffff000009680000 ffff000008ec0000
[    0.000000] fe00: ffff000008cf3000 000000000fe80000
00000000013b0000 0000000011230000
[    0.000000] fe20: 000000000f370018 ffff000008ccfe80
ffff000008b76984 ffff000008ccfe80
[    0.000000] fe40: ffff000008b76984 00000000600000c5
ffff00000959b7a8 ffff000008ec0000
[    0.000000] fe60: ffffffffffffffff 0000000000000005
ffff000008ccfe80 ffff000008b76984
[    0.000000] [<ffff000008b76984>] arm64_memblock_init+0x210/0x484
[    0.000000] [<ffff000008b7398c>] setup_arch+0x1b8/0x5f4
[    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
[    0.000000] random: get_random_bytes called from
print_oops_end_marker+0x50/0x6c with crng_init=0
[    0.000000] ---[ end trace 0000000000000000 ]---
[    0.000000] Reserving 4KB of memory at 0x2e7f0000 for elfcorehdr
[    0.000000] cma: Failed to reserve 512 MiB
[    0.000000] Kernel panic - not syncing: ERROR: Failed to allocate
0x0000000000010000 bytes below 0x0000000000000000.
[    0.000000]
[    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G        W
------------   4.14.0+ #7
[    0.000000] Call trace:
[    0.000000] [<ffff000008088da8>] dump_backtrace+0x0/0x23c
[    0.000000] [<ffff000008089008>] show_stack+0x24/0x2c
[    0.000000] [<ffff0000087f647c>] dump_stack+0x84/0xa8
[    0.000000] [<ffff0000080cfd44>] panic+0x138/0x2a0
[    0.000000] [<ffff000008b95c88>] memblock_alloc_base+0x44/0x4c
[    0.000000] [<ffff000008b95cbc>] memblock_alloc+0x2c/0x38
[    0.000000] [<ffff000008b772dc>] early_pgtable_alloc+0x20/0x74
[    0.000000] [<ffff000008b7755c>] paging_init+0x28/0x544
[    0.000000] [<ffff000008b73990>] setup_arch+0x1bc/0x5f4
[    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
[    0.000000] ---[ end Kernel panic - not syncing: ERROR: Failed to
allocate 0x0000000000010000 bytes below 0x0000000000000000.
[    0.000000]

I guess it is because of the 1G alignment requirement between the
kernel image and the initrd and how we populate the holes between the
kernel image, segments (including dtb) and the initrd from the
kexec-tools.

Akashi, any pointers on this will be helpful as well.

Regards,
Bhupesh


>> >
>> > Regards,
>> > Bhupesh
>> >
>> > >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
>> > >> via a kernel command line parameter, "memmap=".
>> > >>
>> > _______________________________________________
>> > kexec mailing list -- kexec at lists.fedoraproject.org
>> > To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From takahiro.akashi at linaro.org  Mon Dec 18 03:18:50 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon, 18 Dec 2017 20:18:50 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CAFTCetQ55zUKe25jSku0DHp8uVZA4hB32d5W6MSCNsTVpxu7Gw@mail.gmail.com>
References: <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
 <20171218055356.mq4coiibh3frii54@fireball>
 <CAFTCetQ55zUKe25jSku0DHp8uVZA4hB32d5W6MSCNsTVpxu7Gw@mail.gmail.com>
Message-ID: <20171218111844.xyed3bel2wl6el3p@fireball>

Bhupesh,

On Mon, Dec 18, 2017 at 02:29:05PM +0530, Bhupesh SHARMA wrote:
> On Mon, Dec 18, 2017 at 11:24 AM, AKASHI Takahiro
> <takahiro.akashi at linaro.org> wrote:
> > On Mon, Dec 18, 2017 at 01:16:57PM +0800, Dave Young wrote:
> >> kexec at fedoraproject... is for Fedora kexec scripts discussion, changed it
> >> to kexec at lists.infradead.org
> >>
> >> Also add linux-acpi list
> >
> > Thank you.
> >
> >> On 12/18/17 at 02:31am, Bhupesh Sharma wrote:
> >> > On Fri, Dec 15, 2017 at 3:05 PM, Ard Biesheuvel
> >> > <ard.biesheuvel at linaro.org> wrote:
> >> > > On 15 December 2017 at 09:59, AKASHI Takahiro
> >> > > <takahiro.akashi at linaro.org> wrote:
> >> > >> On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> >> > >>> On 13 December 2017 at 12:16, AKASHI Takahiro
> >> > >>> <takahiro.akashi at linaro.org> wrote:
> >> > >>> > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> >> > >>> >> On 13 December 2017 at 10:26, AKASHI Takahiro
> >> > >>> >> <takahiro.akashi at linaro.org> wrote:
> >> > >>> >> > Bhupesh, Ard,
> >> > >>> >> >
> >> > >>> >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> >> > >>> >> >> Hi Ard, Akashi
> >> > >>> >> >>
> >> > >>> >> > (snip)
> >> > >>> >> >
> >> > >>> >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> >> > >>> >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> >> > >>> >> >> identify its own usable memory and exclude, at its boot time, any
> >> > >>> >> >> other memory areas that are part of the panicked kernel's memory.
> >> > >>> >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> >> > >>> >> >> , for details)
> >> > >>> >> >
> >> > >>> >> > Right.
> >> > >>> >> >
> >> > >>> >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> >> > >>> >> >> with the crashkernel memory range:
> >> > >>> >> >>
> >> > >>> >> >>                 /* add linux,usable-memory-range */
> >> > >>> >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> >> > >>> >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> >> > >>> >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> >> > >>> >> >>                                 address_cells, size_cells);
> >> > >>> >> >>
> >> > >>> >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> >> > >>> >> >> , for details)
> >> > >>> >> >>
> >> > >>> >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> >> > >>> >> >> they are marked as System RAM or as RESERVED. As,
> >> > >>> >> >> 'linux,usable-memory-range' dt node is patched up only with
> >> > >>> >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> >> > >>> >> >>
> >> > >>> >> >> 3). As a result when the crashkernel boots up it doesn't find this
> >> > >>> >> >> ACPI memory and crashes while trying to access the same:
> >> > >>> >> >>
> >> > >>> >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> >> > >>> >> >> -r`.img --reuse-cmdline -d
> >> > >>> >> >>
> >> > >>> >> >> [snip..]
> >> > >>> >> >>
> >> > >>> >> >> Reserved memory range
> >> > >>> >> >> 000000000e800000-000000002e7fffff (0)
> >> > >>> >> >>
> >> > >>> >> >> Coredump memory ranges
> >> > >>> >> >> 0000000000000000-000000000e7fffff (0)
> >> > >>> >> >> 000000002e800000-000000003961ffff (0)
> >> > >>> >> >> 0000000039d40000-000000003ed2ffff (0)
> >> > >>> >> >> 000000003ed60000-000000003fbfffff (0)
> >> > >>> >> >> 0000001040000000-0000001ffbffffff (0)
> >> > >>> >> >> 0000002000000000-0000002ffbffffff (0)
> >> > >>> >> >> 0000009000000000-0000009ffbffffff (0)
> >> > >>> >> >> 000000a000000000-000000affbffffff (0)
> >> > >>> >> >>
> >> > >>> >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> >> > >>> >> >> memory cap'ing passed to the crash kernel inside
> >> > >>> >> >> 'arch/arm64/mm/init.c' (see below):
> >> > >>> >> >>
> >> > >>> >> >> static void __init fdt_enforce_memory_region(void)
> >> > >>> >> >> {
> >> > >>> >> >>         struct memblock_region reg = {
> >> > >>> >> >>                 .size = 0,
> >> > >>> >> >>         };
> >> > >>> >> >>
> >> > >>> >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >> > >>> >> >>
> >> > >>> >> >>         if (reg.size)
> >> > >>> >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> >> > >>> >> >> comment this out */
> >> > >>> >> >> }
> >> > >>> >> >
> >> > >>> >> > Please just don't do that. It can cause a fatal damage on
> >> > >>> >> > memory contents of the *crashed* kernel.
> >> > >>> >> >
> >> > >>> >> >> 5). Both the above temporary solutions fix the problem.
> >> > >>> >> >>
> >> > >>> >> >> 6). However exposing all System RAM regions to the crashkernel is not
> >> > >>> >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> >> > >>> >> >> fail.
> >> > >>> >> >>
> >> > >>> >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> >> > >>> >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> >> > >>> >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> >> > >>> >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> >> > >>> >> >> dt node 'linux,usable-memory-range'
> >> > >>> >> >
> >> > >>> >> > I still don't understand why we need to carry over the information
> >> > >>> >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> >> > >>> >> > such regions are free to be reused by the kernel after some point of
> >> > >>> >> > initialization. Why does crash dump kernel need to know about them?
> >> > >>> >> >
> >> > >>> >>
> >> > >>> >> Not really. According to the UEFI spec, they can be reclaimed after
> >> > >>> >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> >> > >>> >> no longer needs them. Of course, in order to be able to boot a kexec
> >> > >>> >> kernel, those regions needs to be preserved, which is why they are
> >> > >>> >> memblock_reserve()'d now.
> >> > >>> >
> >> > >>> > For my better understandings, who is actually accessing such regions
> >> > >>> > during boot time, uefi itself or efistub?
> >> > >>> >
> >> > >>>
> >> > >>> No, only the kernel. This is where the ACPI tables are stored. For
> >> > >>> instance, on QEMU we have
> >> > >>>
> >> > >>>  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> >> > >>>  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> >> > >>>   01000013)
> >> > >>>  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> >> > >>> BXPC 00000001)
> >> > >>>  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> >> > >>> BXPC 00000001)
> >> > >>>  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> >> > >>> BXPC 00000001)
> >> > >>>  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> >> > >>> BXPC 00000001)
> >> > >>>  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> >> > >>> BXPC 00000001)
> >> > >>>  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> >> > >>> BXPC 00000001)
> >> > >>>  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> >> > >>> BXPC 00000001)
> >> > >>>
> >> > >>> covered by
> >> > >>>
> >> > >>>  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> >> > >>>  ...
> >> > >>>  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> >> > >>
> >> > >> OK. I mistakenly understood those regions could be freed after exiting
> >> > >> UEFI boot services.
> >> > >>
> >> > >>>
> >> > >>> >> So it seems that kexec does not honour the memblock_reserve() table
> >> > >>> >> when booting the next kernel.
> >> > >>> >
> >> > >>> > not really.
> >> > >>> >
> >> > >>> >> > (In other words, can or should we skip some part of ACPI-related init code
> >> > >>> >> > on crash dump kernel?)
> >> > >>> >> >
> >> > >>> >>
> >> > >>> >> I don't think so. And the change to the handling of ACPI reclaim
> >> > >>> >> regions only revealed the bug, not created it (given that other
> >> > >>> >> memblock_reserve regions may be affected as well)
> >> > >>> >
> >> > >>> > As whether we should honor such reserved regions over kexec'ing
> >> > >>> > depends on each one's specific nature, we will have to take care one-by-one.
> >> > >>> > As a matter of fact, no information about "reserved" memblocks is
> >> > >>> > exposed to user space (via proc/iomem).
> >> > >>> >
> >> > >>>
> >> > >>> That is why I suggested (somewhere in this thread?) to not expose them
> >> > >>> as 'System RAM'. Do you think that could solve this?
> >> > >>
> >> > >> Memblock-reserv'ing them is necessary to prevent their corruption and
> >> > >> marking them under another name in /proc/iomem would also be good in order
> >> > >> not to allocate them as part of crash kernel's memory.
> >> > >>
> >> > >
> >> > > I agree. However, this may not be entirely trivial, since iterating
> >> > > over the memblock_reserved table and creating iomem entries may result
> >> > > in collisions.
> >> >
> >> > I found a method (using the patch I shared earlier in this thread) to mark these
> >> > entries as 'ACPI reclaim memory' ranges rather than System RAM or
> >> > reserved regions.
> >> >
> >> > >> But I'm not still convinced that we should export them in useable-
> >> > >> memory-range to crash dump kernel. They will be accessed through
> >> > >> acpi_os_map_memory() and so won't be required to be part of system ram
> >> > >> (or memblocks), I guess.
> >> > >
> >> > > Agreed. They will be covered by the linear mapping in the boot kernel,
> >> > > and be mapped explicitly via ioremap_cache() in the kexec kernel,
> >> > > which is exactly what we want in this case.
> >> >
> >> > Now this is what is confusing me. I don't see the above happening.
> >> >
> >> > I see that the primary kernel boots up and adds the ACPI regions via:
> >> > acpi_os_ioremap
> >> >     -> ioremap_cache
> >> >
> >> > But during the crashkernel boot, ''acpi_os_ioremap' calls
> >> > 'ioremap' for the ACPI Reclaim Memory regions and not the _cache
> >> > variant.
> >
> > It is natural if that region is out of memblocks.
> 
> Thanks for the confirmation. This was my understanding as well.
> 
> >> > And it fails while accessing the ACPI tables:
> >> >
> >> > [    0.039205] ACPI: Core revision 20170728
> >> > pud=000000002e7d0003, *pmd=000000002e7c0003, *pte=00e8000039710707
> >> > [    0.095098] Internal error: Oops: 96000021 [#1] SMP
> >
> > this (ESR = 0x96000021) means that Data Abort and Alignment fault happened.
> > As ioremap() makes the mapping as "Device memory", unaligned memory
> > access won't be allowed.
> >
> >> > [    0.100022] Modules linked in:
> >> > [    0.103102] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc6 #1
> >> > [    0.109432] task: ffff000008d05180 task.stack: ffff000008cc0000
> >> > [    0.115414] PC is at acpi_ns_lookup+0x25c/0x3c0
> >> > [    0.119987] LR is at acpi_ds_load1_begin_op+0xa4/0x294
> >> > [    0.125175] pc : [<ffff0000084a6764>] lr : [<ffff00000849b4f8>]
> >> > pstate: 60000045
> >> > [    0.132647] sp : ffff000008ccfb40
> >> > [    0.135989] x29: ffff000008ccfb40 x28: ffff000008a9f2a4
> >> > [    0.141354] x27: ffff0000088be820 x26: 0000000000000000
> >> > [    0.146718] x25: 000000000000001b x24: 0000000000000001
> >> > [    0.152083] x23: 0000000000000001 x22: ffff000009710027
> >> > [    0.157447] x21: ffff000008ccfc50 x20: 0000000000000001
> >> > [    0.162812] x19: 000000000000001b x18: 0000000000000005
> >> > [    0.168176] x17: 0000000000000000 x16: 0000000000000000
> >> > [    0.173541] x15: 0000000000000000 x14: 000000000000038e
> >> > [    0.178905] x13: ffffffff00000000 x12: ffffffffffffffff
> >> > [    0.184270] x11: 0000000000000006 x10: 00000000ffffff76
> >> > [    0.189634] x9 : 000000000000005f x8 : ffff8000126d0140
> >> > [    0.194998] x7 : 0000000000000000 x6 : ffff000008ccfc50
> >> > [    0.200362] x5 : ffff80000fe62c00 x4 : 0000000000000001
> >> > [    0.205727] x3 : ffff000008ccfbe0 x2 : ffff0000095e3980
> >> > [    0.211091] x1 : ffff000009710027 x0 : 0000000000000000
> >> > [    0.216456] Process swapper/0 (pid: 0, stack limit = 0xffff000008cc0000)
> >> > [    0.223224] Call trace:
> >> > [    0.225688] Exception stack(0xffff000008ccfa00 to 0xffff000008ccfb40)
> >> > [    0.232194] fa00: 0000000000000000 ffff000009710027
> >> > ffff0000095e3980 ffff000008ccfbe0
> >> > [    0.240106] fa20: 0000000000000001 ffff80000fe62c00
> >> > ffff000008ccfc50 0000000000000000
> >> > [    0.248018] fa40: ffff8000126d0140 000000000000005f
> >> > 00000000ffffff76 0000000000000006
> >> > [    0.255931] fa60: ffffffffffffffff ffffffff00000000
> >> > 000000000000038e 0000000000000000
> >> > [    0.263843] fa80: 0000000000000000 0000000000000000
> >> > 0000000000000005 000000000000001b
> >> > [    0.271754] faa0: 0000000000000001 ffff000008ccfc50
> >> > ffff000009710027 0000000000000001
> >> > [    0.279667] fac0: 0000000000000001 000000000000001b
> >> > 0000000000000000 ffff0000088be820
> >> > [    0.287579] fae0: ffff000008a9f2a4 ffff000008ccfb40
> >> > ffff00000849b4f8 ffff000008ccfb40
> >> > [    0.295491] fb00: ffff0000084a6764 0000000060000045
> >> > ffff000008ccfb40 ffff000008260a18
> >> > [    0.303403] fb20: ffffffffffffffff ffff0000087f3fb0
> >> > ffff000008ccfb40 ffff0000084a6764
> >> > [    0.311316] [<ffff0000084a6764>] acpi_ns_lookup+0x25c/0x3c0
> >> > [    0.316943] [<ffff00000849b4f8>] acpi_ds_load1_begin_op+0xa4/0x294
> >> > [    0.323186] [<ffff0000084ad4ac>] acpi_ps_build_named_op+0xc4/0x198
> >> > [    0.329428] [<ffff0000084ad6cc>] acpi_ps_create_op+0x14c/0x270
> >> > [    0.335319] [<ffff0000084acfa8>] acpi_ps_parse_loop+0x188/0x5c8
> >> > [    0.341298] [<ffff0000084ae048>] acpi_ps_parse_aml+0xb0/0x2b8
> >> > [    0.347101] [<ffff0000084a8e10>] acpi_ns_one_complete_parse+0x144/0x184
> >> > [    0.353783] [<ffff0000084a8e98>] acpi_ns_parse_table+0x48/0x68
> >> > [    0.359675] [<ffff0000084a82cc>] acpi_ns_load_table+0x4c/0xdc
> >> > [    0.365479] [<ffff0000084b32f8>] acpi_tb_load_namespace+0xe4/0x264
> >> > [    0.371723] [<ffff000008baf9b4>] acpi_load_tables+0x48/0xc0
> >> > [    0.377350] [<ffff000008badc20>] acpi_early_init+0x9c/0xd0
> >> > [    0.382891] [<ffff000008b70d50>] start_kernel+0x3b4/0x43c
> >> > [    0.388343] Code: b9008fb9 2a000318 36380054 32190318 (b94002c0)
> >> > [    0.394500] ---[ end trace c46ed37f9651c58e ]---
> >> > [    0.399160] Kernel panic - not syncing: Fatal exception
> >> > [    0.404437] Rebooting in 10 seconds.
> >> >
> >> > So, I think the linear mapping done by the primary kernel does not
> >> > make these accessible in the crash kernel directly.
> >> >
> >> > Any pointers?
> >>
> >> Can you get the code line number for acpi_ns_lookup+0x25c?
> >
> > So should we always avoid ioremap() in acpi_os_ioremap() entirely, or
> > modify acpi_ns_lookup() (or any acpi functions') to prevent unaligned
> > accesses?
> > (I didn't find out how unaligned accesses could happen there.)
> >
> 
> Right. Like I captured somewhere in this thread (perhaps the first
> email on this subject),
> this is indeed an unaligned address access.
> 
> Now, modifying acpi_os_ioremap() to not ioremap() and thus avoiding
> assigning this memory range
> as device memory doesn't seem a neat solution as it means we are not
> marking some thing with the right memory attribute and we can fall in
> similar/related issues later.
> 
> Regarding the later suggestion, what I am seeing now is that the acpi
> table access functions are perhaps reused from the earlier x86
> implementation, but on the arm64 (or even arm) arch we should not be
> allowing unaligned accesses which might cause UNDEFINED behaviour and
> resultant crash.
> 
> So I can try going this approach and see if it works for me.
> 
> However, I am still not very sure as to why the crashkernel ranges
> historically do not include the System RAM regions (which may include
> the ACPI regions as well). These regions are available for the kernel
> usage and perhaps should be exported to the crashkernel as well.
> 
> I am not fully aware of the previous discussions on capp'ing the
> crashkernel memory being passed to the kdump kernel, but did we run
> into any issues while doing so?
> 
> Also, even if I extend the kexec-tools to modify the
> linux,usable-memory-range and add the ACPI regions to it, the
> crashkernel fails to boot with the below message (I have added some
> logic to print the DTB on the crash kernel boot start):
> 
> [    0.000000]     chosen {
> [    0.000000]         linux,usable-memory-range
> [    0.000000]  = <
> [    0.000000] 0x00000000
> [    0.000000] 0x0e800000
> [    0.000000] 0x00000000
> [    0.000000] 0x20000000
> [    0.000000] 0x00000000
> [    0.000000] 0x396c0000
> [    0.000000] 0x00000000
> [    0.000000] 0x000a0000
> [    0.000000] 0x00000000
> [    0.000000] 0x39770000
> [    0.000000] 0x00000000
> [    0.000000] 0x00040000
> [    0.000000] 0x00000000
> [    0.000000] 0x398a0000
> [    0.000000] 0x00000000
> [    0.000000] 0x00020000
> [    0.000000] >
> [    0.000000] ;
> 
> [snip..]
> 
> [    0.000000] linux,usable-memory-range base e800000, size 20000000
> [    0.000000]  - e800000 ,  20000000
> [    0.000000] linux,usable-memory-range base 396c0000, size a0000
> [    0.000000]  - 396c0000 ,  a0000
> [    0.000000] linux,usable-memory-range base 39770000, size 40000
> [    0.000000]  - 39770000 ,  40000
> [    0.000000] linux,usable-memory-range base 398a0000, size 20000
> [    0.000000]  - 398a0000 ,  20000
> [    0.000000] initrd not fully accessible via the linear mapping --
> please check your bootloader ...
> [    0.000000] ------------[ cut here ]------------
> [    0.000000] WARNING: CPU: 0 PID: 0 at arch/arm64/mm/init.c:597
> arm64_memblock_init+0x210/0x484
> [    0.000000] Modules linked in:
> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0+ #7
> [    0.000000] task: ffff000008d05580 task.stack: ffff000008cc0000
> [    0.000000] PC is at arm64_memblock_init+0x210/0x484
> [    0.000000] LR is at arm64_memblock_init+0x210/0x484
> [    0.000000] pc : [<ffff000008b76984>] lr : [<ffff000008b76984>]
> pstate: 600000c5
> [    0.000000] sp : ffff000008ccfe80
> [    0.000000] x29: ffff000008ccfe80 x28: 000000000f370018
> [    0.000000] x27: 0000000011230000 x26: 00000000013b0000
> [    0.000000] x25: 000000000fe80000 x24: ffff000008cf3000
> [    0.000000] x23: ffff000008ec0000 x22: ffff000009680000
> [    0.000000] x21: ffff000008afa000 x20: ffff000008080000
> [    0.000000] x19: ffff000008afa000 x18: 000000000c283806
> [    0.000000] x17: 0000000000000000 x16: ffff000008d05580
> [    0.000000] x15: 000000002be00842 x14: 79206b6365686320
> [    0.000000] x13: 657361656c70202d x12: 2d20676e69707061
> [    0.000000] x11: 6d207261656e696c x10: 2065687420616976
> [    0.000000] x9 : 00000000000000f4 x8 : ffff000008517414
> [    0.000000] x7 : 746f6f622072756f x6 : 000000000000000d
> [    0.000000] x5 : ffff000008c96360 x4 : 0000000000000001
> [    0.000000] x3 : 0000000000000000 x2 : 0000000000000000
> [    0.000000] x1 : 0000000000000000 x0 : 0000000000000056
> [    0.000000] Call trace:
> [    0.000000] Exception stack(0xffff000008ccfd40 to 0xffff000008ccfe80)
> [    0.000000] fd40: 0000000000000056 0000000000000000
> 0000000000000000 0000000000000000
> [    0.000000] fd60: 0000000000000001 ffff000008c96360
> 000000000000000d 746f6f622072756f
> [    0.000000] fd80: ffff000008517414 00000000000000f4
> 2065687420616976 6d207261656e696c
> [    0.000000] fda0: 2d20676e69707061 657361656c70202d
> 79206b6365686320 000000002be00842
> [    0.000000] fdc0: ffff000008d05580 0000000000000000
> 000000000c283806 ffff000008afa000
> [    0.000000] fde0: ffff000008080000 ffff000008afa000
> ffff000009680000 ffff000008ec0000
> [    0.000000] fe00: ffff000008cf3000 000000000fe80000
> 00000000013b0000 0000000011230000
> [    0.000000] fe20: 000000000f370018 ffff000008ccfe80
> ffff000008b76984 ffff000008ccfe80
> [    0.000000] fe40: ffff000008b76984 00000000600000c5
> ffff00000959b7a8 ffff000008ec0000
> [    0.000000] fe60: ffffffffffffffff 0000000000000005
> ffff000008ccfe80 ffff000008b76984
> [    0.000000] [<ffff000008b76984>] arm64_memblock_init+0x210/0x484
> [    0.000000] [<ffff000008b7398c>] setup_arch+0x1b8/0x5f4
> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
> [    0.000000] random: get_random_bytes called from
> print_oops_end_marker+0x50/0x6c with crng_init=0
> [    0.000000] ---[ end trace 0000000000000000 ]---
> [    0.000000] Reserving 4KB of memory at 0x2e7f0000 for elfcorehdr
> [    0.000000] cma: Failed to reserve 512 MiB
> [    0.000000] Kernel panic - not syncing: ERROR: Failed to allocate
> 0x0000000000010000 bytes below 0x0000000000000000.
> [    0.000000]
> [    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G        W
> ------------   4.14.0+ #7
> [    0.000000] Call trace:
> [    0.000000] [<ffff000008088da8>] dump_backtrace+0x0/0x23c
> [    0.000000] [<ffff000008089008>] show_stack+0x24/0x2c
> [    0.000000] [<ffff0000087f647c>] dump_stack+0x84/0xa8
> [    0.000000] [<ffff0000080cfd44>] panic+0x138/0x2a0
> [    0.000000] [<ffff000008b95c88>] memblock_alloc_base+0x44/0x4c
> [    0.000000] [<ffff000008b95cbc>] memblock_alloc+0x2c/0x38
> [    0.000000] [<ffff000008b772dc>] early_pgtable_alloc+0x20/0x74
> [    0.000000] [<ffff000008b7755c>] paging_init+0x28/0x544
> [    0.000000] [<ffff000008b73990>] setup_arch+0x1bc/0x5f4
> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
> [    0.000000] ---[ end Kernel panic - not syncing: ERROR: Failed to
> allocate 0x0000000000010000 bytes below 0x0000000000000000.
> [    0.000000]
> 
> I guess it is because of the 1G alignment requirement between the
> kernel image and the initrd and how we populate the holes between the
> kernel image, segments (including dtb) and the initrd from the
> kexec-tools.
> 
> Akashi, any pointers on this will be helpful as well.

Please show me:
 * "Virtual kernel memory layout" in dmesg
 * /proc/iomem
 * debug messages from kexec-tools (kexec -d)

-Takahiro AKASHI


> Regards,
> Bhupesh
> 
> 
> >> >
> >> > Regards,
> >> > Bhupesh
> >> >
> >> > >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >> > >> via a kernel command line parameter, "memmap=".
> >> > >>
> >> > _______________________________________________
> >> > kexec mailing list -- kexec at lists.fedoraproject.org
> >> > To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From bhe at redhat.com  Mon Dec 18 05:47:36 2017
From: bhe at redhat.com (Baoquan He)
Date: Mon, 18 Dec 2017 21:47:36 +0800
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
In-Reply-To: <20171217214735.nuxq5zo2eknqpbpi@pd.tnic>
References: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
 <20171216010142.GK12442@x1>
 <20171217214735.nuxq5zo2eknqpbpi@pd.tnic>
Message-ID: <20171218134736.GA4035@x1>

On 12/17/17 at 10:47pm, Borislav Petkov wrote:
> On Sat, Dec 16, 2017 at 09:01:42AM +0800, Baoquan He wrote:
> > 2) If firmware is broken, you can't enable gart in firmware, will
> > firmware engineer fix this since it's a firmware bug?
> 
> Slow down and get a reality check first please!
> 
> A firmware engineer will fix a 10yr old BIOS?!? Yeah right. And I'll get
> a pink pony for Christmas. Geez.

The code is too old. I tried to find out the original commit, many files
moving commits make it not easy to track. In the current code, can only
see the pr_info telling people to "enable the IOMMU option in the BIOS
setup". No even one word to mention that it's for borken firmware.

>From Jiri's replying, he used 'guess', means the bug he is trying to fix
is not broken firmware case, but not enabling gart iommu support in bios.


gart_iommu_hole_init() {

	...

        } else if ((!no_iommu && max_pfn > MAX_DMA32_PFN) ||                                                                                     
                   force_iommu ||
                   valid_agp ||
                   fallback_aper_force) {
                pr_info("Your BIOS doesn't leave an aperture memory hole\n");
                pr_info("Please enable the IOMMU option in the BIOS setup\n");
                pr_info("This costs you %dMB of RAM\n",
                        32 << fallback_aper_order);
	...
}

> 
> We need a reliable way to tell the second kernel not to access the gart
> range. And frankly, the best thing to do would be to teach the *second*
> kernel to simply avoid the gart range. Regardless of what it gets told
> by the ELF header. Because there are some ranges which it shouldn't
> touch. Maybe we can reuse the gart detection code to do that in the
> second kernel too.

Previously people added gart region to iomem to notice that even though
there's ram mapped, while it's occupied by gart, please don't dump it.
Later it's reverted commit 707d4eefbdb3 ("Revert [PATCH] Insert GART
region into resource map"). 

In fact, there are two ways to fix this bug. One is to tell kdump kernel
not to dump the region of gart even though there are ram mapped to that
region and added to vmemmap and direct mapping. This was done before and
reverted later.

The other is not to tell kdump kernel that there's ram mapped into the
region. In the mail I replied to Jiri's v1 post, I meant the 2nd way.
Remove the ram region occupied by gart from iomem, then kdump kernel
won't see it and won't dump it.

And note that when we talk about this gart issue, we only mean the case
that gart support is not enabled in bios. In this case, gart will find a
region of ram and occupy it as gart aperture. And this is done during
gart iommu init, and after that ram region has been added to memory
subsystem.

> 
> But I haven't looked at it, might be hairy. Need to deal with this PTI
> madness first.
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> Good mailing practices for 400: avoid top-posting and trim the reply.


From daniel.kiper at oracle.com  Mon Dec 18 05:43:52 2017
From: daniel.kiper at oracle.com (Daniel Kiper)
Date: Mon, 18 Dec 2017 14:43:52 +0100
Subject: [PATCH v2] kexec-tools: Perform run-time linking of libxenctrl.so
In-Reply-To: <1513291681-497-1-git-send-email-eric.devolder@oracle.com>
References: <1513291681-497-1-git-send-email-eric.devolder@oracle.com>
Message-ID: <20171218134352.GX4531@olila.local.net-space.pl>

On Thu, Dec 14, 2017 at 04:48:01PM -0600, Eric DeVolder wrote:
> When kexec is utilized in a Xen environment, it has an explicit
> run-time dependency on libxenctrl.so. This dependency occurs
> during the configure stage and when building kexec-tools.
>
> When kexec is utilized in a non-Xen environment (either bare
> metal or KVM), the configure and build of kexec-tools omits
> any reference to libxenctrl.so.
>
> Thus today it is not currently possible to configure and build
> a *single* kexec that will work in *both* Xen and non-Xen
> environments, unless the libxenctrl.so is *always* present.
>
> For example, a kexec configured for Xen in a Xen environment:
>
>  # ldd build/sbin/kexec
>         linux-vdso.so.1 =>  (0x00007ffdeba5c000)
>         libxenctrl.so.4.4 => /usr/lib64/libxenctrl.so.4.4 (0x00000038d8000000)
>         libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000)
>         libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000)
>         /lib64/ld-linux-x86-64.so.2 (0x000055e9f8c6c000)
>  # build/sbin/kexec -v
>  kexec-tools 2.0.16
>
> However, the *same* kexec executable fails in a non-Xen environment:
>
>  # copy xen kexec to .
>  # ldd ./kexec
>          linux-vdso.so.1 =>  (0x00007fffa9da7000)
>          libxenctrl.so.4.4 => not found
>          liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x0000003014e00000)
>          libz.so.1 => /lib64/libz.so.1 (0x000000300ea00000)
>          libc.so.6 => /lib64/libc.so.6 (0x000000300de00000)
>          libpthread.so.0 => /lib64/libpthread.so.0 (0x000000300e200000)
>          /lib64/ld-linux-x86-64.so.2 (0x0000558cc786c000)
>  # ./kexec -v
>  ./kexec: error while loading shared libraries:
>  libxenctrl.so.4.4: cannot open shared object file: No such file or directory
>
> At Oracle we "workaround" this by having two kexec-tools packages,
> one for Xen and another for non-Xen environments. At Oracle, the
> desire is to offer a single kexec-tools package that works in either
> environment. To achieve this, kexec-tools would either have to ship
> with libxenctrl.so (which we have deemed as unacceptable), or we can
> make kexec perform run-time linking against libxenctrl.so.
>
> This patch is one possible way to alleviate the explicit run-time
> dependency on libxenctrl.so. This implementation utilizes a set of
> macros to wrap calls into libxenctrl.so so that the library can
> instead be dlopen() and obtain the function via dlsym() and then
> make the call. The advantage of this implementation is that it
> requires few changes to the existing kexec-tools code. The dis-
> advantage is that it uses macros to remap libxenctrl functions
> and do work under the hood.
>
> Another possible implementation worth considering is the approach
> taken by libvmi. Reference the following file:
>
> https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/libxc_wrapper.h
>
> The libxc_wrapper_t structure definition that starts at line ~33
> has members that are function pointers into libxenctrl.so. This
> structure is populated once and then later referenced/dereferenced
> by the callers of libxenctrl.so members. The advantage of this
> implementation is it is more explicit in managing the use of
> libxenctrl.so and its versions, but the disadvantage is it would
> require touching more of the kexec-tools code.
>
> The following is a list libxenctrl members utilized by kexec:
>
> Functions:
> xc_interface_open
> xc_kexec_get_range
> xc_interface_close
> xc_kexec_get_range
> xc_interface_open
> xc_get_max_cpus
> xc_kexec_get_range
> xc_version
> xc_kexec_exec
> xc_kexec_status
> xc_kexec_unload
> xc_hypercall_buffer_array_create
> xc__hypercall_buffer_array_alloc
> xc_hypercall_buffer_array_destroy
> xc_kexec_load
> xc_get_machine_memory_map
>
> Data:
> xc__hypercall_buffer_HYPERCALL_BUFFER_NULL
>
> These were identified by configuring and building kexec-tools
> with Xen support, but omitting the -lxenctrl from the LDFLAGS
> in the Makefile for an x86_64 build.
>
> The above libxenctrl members were referenced via these source
> files.
>
> kexec/crashdump-xen.c
> kexec/kexec-xen.c
> kexec/arch/i386/kexec-x86-common.c
> kexec/arch/i386/crashdump-x86.c
>
> This patch provides a wrapper around the calls to the above
> functions in libxenctrl.so. Every libxenctrl call must pass a
> xc_interface which it obtains from xc_interface_open().
> So the existing code is already structured in a manner that
> facilitates graceful dlopen()'ing of the libxenctrl.so and
> the subsequent dlsym() of the required member.
>
> The patch creates a wrapper function around xc_interface_open()
> and xc_interface_close() to perform the dlopen() and dlclose().
>
> For the remaining xc_ functions, this patch defines a macro
> of the same name which performs the dlsym() and then invokes
> the function. See the _xc_call() macro for details.
>
> There was one data item in libxenctrl.so that presented a
> unique problem, HYPERCALL_BUFFER_NULL. It was only utilized
> once, as
>
>     set_xen_guest_handle(xen_segs[s].buf.h, HYPERCALL_BUFFER_NULL);
>
> I tried a variety of techniques but could not find a general
> macro-type solution without modifying xenctrl.h. So the
> solution was to declare a local HYPERCALL_BUFFER_NULL, and
> this appears to work. I admit I am not familiar with libxenctrl
> to state if this is a satisfactory workaround, so feedback
> here welcome. I can state that this allows kexec to load/unload/kexec
> on Xen and non-Xen environments that I've tested without issue.
>
> With this patch applied, kexec-tools can be built with Xen
> support and yet there is no explicit run-time dependency on
> libxenctrl.so. Thus it can also be deployed in non-Xen
> environments where libxenctrl.so is not installed.
>
>  # ldd build/sbin/kexec
>         linux-vdso.so.1 =>  (0x00007fff7dbcd000)
>         liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x00000038d9000000)
>         libz.so.1 => /lib64/libz.so.1 (0x00000038d6c00000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00000038d6400000)
>         libc.so.6 => /lib64/libc.so.6 (0x00000038d6000000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00000038d6800000)
>         /lib64/ld-linux-x86-64.so.2 (0x0000562dc0c14000)
>  # build/sbin/kexec -v
>  kexec-tools 2.0.16
>
> Currently this feature is enabled with the following:
>
>  ./configure --with-xen-dl --with-xen=no
>
> This is a bit clunky. I welcome feedback such as better names
> and/or usage of --with, as well as if we might make this feature
> the default.

I would do it in a bit different way. If somebody specifies --with-xen
then kexec-tools should build as usual. However, if somebody specifies
with-xen-dl itself or --with-xen-dl and --with-xen then --with-xen-dl
should have a precedence.

> Signed-off-by: Eric DeVolder <eric.devolder at oracle.com>
> ---
> v1: 29nov2017
>  - Daniel Kiper suggested Debian's libxen package of libraries,
>    but I did not find similar package on most other systems.
>
> v2: 14dec2017
>  - Reposted to kexec and xen-devel mailing lists
> ---
>  configure.ac                       | 18 ++++++++++++++
>  kexec/Makefile                     |  1 +
>  kexec/arch/i386/crashdump-x86.c    |  4 +---
>  kexec/arch/i386/kexec-x86-common.c |  4 +---
>  kexec/crashdump-xen.c              |  4 +---
>  kexec/kexec-xen.c                  | 43 +++++++++++++++++++++++++++++++++-
>  kexec/kexec-xen.h                  | 48 ++++++++++++++++++++++++++++++++++++++
>  7 files changed, 112 insertions(+), 10 deletions(-)
>  create mode 100644 kexec/kexec-xen.h
>
> diff --git a/configure.ac b/configure.ac
> index 208dc0a..4fc8aa0 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -92,6 +92,9 @@ AC_ARG_WITH([lzma], AC_HELP_STRING([--without-lzma],[disable lzma support]),
>  AC_ARG_WITH([xen], AC_HELP_STRING([--without-xen],
>  	[disable extended xen support]), [ with_xen="$withval"], [ with_xen=yes ] )
>
> +AC_ARG_WITH([xen-dl], AC_HELP_STRING([--without-xen-dl],
> +	[link libxenctrl.so at run-time rather than build-time]), [ with_xen_dl="$withval"], [ with_xen_dl=no ] )
> +
>  AC_ARG_WITH([booke],
>  		AC_HELP_STRING([--with-booke],[build for booke]),
>  		AC_DEFINE(CONFIG_BOOKE,1,
> @@ -174,6 +177,21 @@ if test "$with_xen" = yes ; then
>  				AC_MSG_NOTICE([The kexec_status call is not available]))
>  		fi
>  fi
> +if test "$with_xen_dl" = yes ; then
> +	if test "$with_xen" = yes ; then
> +		AC_MSG_ERROR([Options --with-xen and --with-xen-dl are mutually exclusive])
> +	fi
> +	AC_DEFINE(CONFIG_LIBXENCTRL_DL, 1, [Define to 1 to link libxenctrl.so at run-time rather than build-time])
> +	AC_CHECK_HEADER(dlfcn.h, , AC_MSG_ERROR([Dynamic library linking not available]))
> +	AC_CHECK_LIB(dl, dlopen, , AC_MSG_ERROR([Dynamic library linking not available]))

Please call AC_CHECK_LIB() from AC_CHECK_HEADER(). You can find more details
if you take a look at zlib, lzma and even xenctrl config in configure.ac.

> +	AC_CHECK_HEADER(xenctrl.h,
> +		AC_DEFINE(HAVE_LIBXENCTRL, 1, [Define to 1 to enable run-time linking of libxenctrl.so]),
> +		AC_MSG_ERROR([Xen support not available]))
> +dnl NOTE: Explicitly *NOT* performing AC_CHECK_LIB(xenctrl) as only need the header file to build
> +	AC_CHECK_HEADER(xenctrl.h,
> +		AC_DEFINE(HAVE_KEXEC_CMD_STATUS, 1, [Define to 1 so kexec_status call is available]),
> +		AC_MSG_NOTICE([The kexec_status call is not available]))
> +fi
>
>  dnl ---Sanity checks
>  if test "$CC"      = "no"; then AC_MSG_ERROR([cc not found]); fi
> diff --git a/kexec/Makefile b/kexec/Makefile
> index 2b4fb3d..8871731 100644
> --- a/kexec/Makefile
> +++ b/kexec/Makefile
> @@ -36,6 +36,7 @@ dist += kexec/Makefile						\
>  	kexec/kexec-elf-boot.h					\
>  	kexec/kexec-elf.h kexec/kexec-sha256.h			\
>  	kexec/kexec-zlib.h kexec/kexec-lzma.h			\
> +	kexec/kexec-xen.h 								\
>  	kexec/kexec-syscall.h kexec/kexec.h kexec/kexec.8
>
>  dist				+= kexec/proc_iomem.c
> diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
> index 69a063a..a948d9f 100644
> --- a/kexec/arch/i386/crashdump-x86.c
> +++ b/kexec/arch/i386/crashdump-x86.c
> @@ -44,9 +44,7 @@
>  #include "kexec-x86.h"
>  #include "crashdump-x86.h"
>
> -#ifdef HAVE_LIBXENCTRL
> -#include <xenctrl.h>
> -#endif /* HAVE_LIBXENCTRL */
> +#include "../../kexec-xen.h"
>
>  #include "x86-linux-setup.h"
>
> diff --git a/kexec/arch/i386/kexec-x86-common.c b/kexec/arch/i386/kexec-x86-common.c
> index be03618..b44c8b7 100644
> --- a/kexec/arch/i386/kexec-x86-common.c
> +++ b/kexec/arch/i386/kexec-x86-common.c
> @@ -40,9 +40,7 @@
>  #include "../../crashdump.h"
>  #include "kexec-x86.h"
>
> -#ifdef HAVE_LIBXENCTRL
> -#include <xenctrl.h>
> -#endif /* HAVE_LIBXENCTRL */
> +#include "../../kexec-xen.h"
>
>  /* Used below but not present in (older?) xenctrl.h */
>  #ifndef E820_PMEM
> diff --git a/kexec/crashdump-xen.c b/kexec/crashdump-xen.c
> index 60594f6..2e4cbdc 100644
> --- a/kexec/crashdump-xen.c
> +++ b/kexec/crashdump-xen.c
> @@ -18,9 +18,7 @@
>
>  #include "config.h"
>
> -#ifdef HAVE_LIBXENCTRL
> -#include <xenctrl.h>
> -#endif
> +#include "kexec-xen.h"
>
>  struct crash_note_info {
>  	unsigned long base;
> diff --git a/kexec/kexec-xen.c b/kexec/kexec-xen.c
> index 2b448d3..2b01fee 100644
> --- a/kexec/kexec-xen.c
> +++ b/kexec/kexec-xen.c
> @@ -10,10 +10,51 @@
>  #include "config.h"
>
>  #ifdef HAVE_LIBXENCTRL
> -#include <xenctrl.h>
> +#include "kexec-xen.h"
>
>  #include "crashdump.h"
>
> +#ifdef CONFIG_LIBXENCTRL_DL
> +void *xc_dlhandle = NULL;

Just "void *xc_dlhandle;". Compiler will do work for you.

> +xc_hypercall_buffer_t XC__HYPERCALL_BUFFER_NAME(HYPERCALL_BUFFER_NULL);

static?

> +xc_interface *_xc_interface_open(xentoollog_logger *logger,

I would prefer __xc_interface_open() instead of _xc_interface_open()
(two underscores at the beginning of the function). Same applies to
the functions/variables below.

> +                                xentoollog_logger *dombuild_logger,
> +                                unsigned open_flags)
> +{
> +    xc_interface *xch = NULL;

xc_interface *xch;

> +    if (NULL == xc_dlhandle)

if (!xc_dlhandle)

> +        xc_dlhandle = dlopen("libxenctrl.so", RTLD_NOW | RTLD_NODELETE);

if (!xc_dlhandle)
  return NULL;

> +    if (xc_dlhandle) {

... then you do not need this condition.

> +        typedef xc_interface *(*func_t)(xentoollog_logger *logger,
> +                                xentoollog_logger *dombuild_logger,
> +                                unsigned open_flags);

Please define type(s) just behind the includes.

> +        func_t func = (func_t)dlsym(xc_dlhandle, "xc_interface_open");
> +        xch = func(logger, dombuild_logger, open_flags);
> +    }
> +
> +    return xch;
> +}
> +
> +int _xc_interface_close(xc_interface *xch)
> +{
> +    int rc = -1;
> +
> +/*
> +    if (xc_dlhandle) {
> +*/
> +        typedef int (*func_t)(xc_interface *xch);
> +        func_t func = (func_t)dlsym(xc_dlhandle, "xc_interface_close");
> +        rc = func(xch);
> +/*

If you are not sure please provide the comment why it is
commented out or drop these lines entirely.

> +        xc_dlhandle = NULL;
> +    }
> +*/
> +    return rc;
> +}
> +#endif /* CONFIG_LIBXENCTRL_DL */
> +
>  int xen_kexec_load(struct kexec_info *info)
>  {
>  	uint32_t nr_segments = info->nr_segments;
> diff --git a/kexec/kexec-xen.h b/kexec/kexec-xen.h
> new file mode 100644
> index 0000000..6d2046e
> --- /dev/null
> +++ b/kexec/kexec-xen.h
> @@ -0,0 +1,48 @@
> +#ifndef KEXEC_XEN_H
> +#define KEXEC_XEN_H
> +
> +#ifdef HAVE_LIBXENCTRL
> +#include <xenctrl.h>
> +
> +#ifdef CONFIG_LIBXENCTRL_DL
> +#include <dlfcn.h>
> +
> +/* The handle from dlopen(), needed by dlsym(), dlclose() */
> +extern void *xc_dlhandle;
> +
> +/* Wrappers around xc_interface_open/close() to insert dlopen/dlclose() */
> +xc_interface *_xc_interface_open(xentoollog_logger *logger,
> +                                xentoollog_logger *dombuild_logger,
> +                                unsigned open_flags);
> +int _xc_interface_close(xc_interface *xch);
> +
> +/* GCC expression statements for evaluating dlsym() */
> +#define _xc_call(DTYPE, NAME, ARGS...) ({ DTYPE value; typedef DTYPE (*func_t)(xc_interface *, ...); func_t func = dlsym(xc_dlhandle, #NAME); value = func(ARGS); value; } )

Too long line. Please try to not exceed 80 characters.

> +#define _xc_data(DTYPE, NAME) ({ DTYPE *value = (DTYPE *)dlsym(xc_dlhandle, #NAME); value; } )

I would export e.g. __xc_dlsym(const char *symbol) which calls
dlsym(xc_dlhandle, symbol). Then you can make xc_dlhandle static.
Hmmm... xc_dlhandle -> xc_dl_handle -> xc_dlh or even xdlh?

> +/* The wrappers around utilized xenctrl.h functions */
> +#define xc_interface_open(A,B,C)    _xc_interface_open(A,B,C)

Lack of space after commas... And please use lowercase letters here.

> +#define xc_interface_close(A)       _xc_interface_close(A)
> +#define xc_version(ARGS...)         _xc_call(int, xc_version, ARGS)
> +#define xc_get_max_cpus(ARGS...)    _xc_call(int, xc_get_max_cpus, ARGS)
> +#define xc_get_machine_memory_map(ARGS...)  _xc_call(int, xc_get_machine_memory_map, ARGS)
> +#define xc_kexec_get_range(ARGS...) _xc_call(int, xc_kexec_get_range, ARGS)
> +#define xc_kexec_load(ARGS...)      _xc_call(int, xc_kexec_load, ARGS)
> +#define xc_kexec_unload(ARGS...)    _xc_call(int, xc_kexec_unload, ARGS)
> +#define xc_kexec_status(ARGS...)    _xc_call(int, xc_kexec_status, ARGS)
> +#define xc_kexec_exec(ARGS...)      _xc_call(int, xc_kexec_exec, ARGS)
> +#define xc_hypercall_buffer_array_create(ARGS...)   _xc_call(xc_hypercall_buffer_array_t *, xc_hypercall_buffer_array_create, ARGS)
> +#define xc__hypercall_buffer_alloc(ARGS...) _xc_call(void *, xc__hypercall_buffer_alloc, ARGS)
> +#define xc__hypercall_buffer_free(ARGS...)  _xc_call(void  , xc__hypercall_buffer_free, ARGS)
> +#define xc__hypercall_buffer_alloc_pages(ARGS...)   _xc_call(void *, xc__hypercall_buffer_alloc_pages, ARGS)
> +#define xc__hypercall_buffer_free_pages(ARGS...)    _xc_call(void  , xc__hypercall_buffer_free_pages, ARGS)
> +#define xc__hypercall_buffer_array_alloc(ARGS...)   _xc_call(void *, xc__hypercall_buffer_array_alloc, ARGS)
> +#define xc__hypercall_buffer_array_get(ARGS...)     _xc_call(void *, xc__hypercall_buffer_array_get, ARGS)
> +#define xc_hypercall_buffer_array_destroy(ARGS...)  _xc_call(void *, xc_hypercall_buffer_array_destroy, ARGS)

Could you try to make this more readable? Alignment, line wraps
if needed, etc. And of course too long lines in many places...

Really kexec-tools use all of them?

Daniel


From bhsharma at redhat.com  Mon Dec 18 13:28:20 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Tue, 19 Dec 2017 02:58:20 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
References: <CAFTCetQHmpprAVu6uYO+rc5Xi4EUVhmovbmSaU6nM1n1mAH62w@mail.gmail.com>
 <CAKv+Gu9oda1Ee8AoXsCEw+Bjn-XF3wZA_CsxvqhjtT6_bmJ7uA@mail.gmail.com>
 <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
Message-ID: <CACi5LpOmeEMuoCkTC7MrBDaA1J5a4vZ_7bh3HSC0G5GoAMUCjw@mail.gmail.com>

Hi Dave,

On Mon, Dec 18, 2017 at 10:46 AM, Dave Young <dyoung at redhat.com> wrote:
> kexec at fedoraproject... is for Fedora kexec scripts discussion, changed it
> to kexec at lists.infradead.org
>
> Also add linux-acpi list
> On 12/18/17 at 02:31am, Bhupesh Sharma wrote:
>> On Fri, Dec 15, 2017 at 3:05 PM, Ard Biesheuvel
>> <ard.biesheuvel at linaro.org> wrote:
>> > On 15 December 2017 at 09:59, AKASHI Takahiro
>> > <takahiro.akashi at linaro.org> wrote:
>> >> On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
>> >>> On 13 December 2017 at 12:16, AKASHI Takahiro
>> >>> <takahiro.akashi at linaro.org> wrote:
>> >>> > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
>> >>> >> On 13 December 2017 at 10:26, AKASHI Takahiro
>> >>> >> <takahiro.akashi at linaro.org> wrote:
>> >>> >> > Bhupesh, Ard,
>> >>> >> >
>> >>> >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
>> >>> >> >> Hi Ard, Akashi
>> >>> >> >>
>> >>> >> > (snip)
>> >>> >> >
>> >>> >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
>> >>> >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
>> >>> >> >> identify its own usable memory and exclude, at its boot time, any
>> >>> >> >> other memory areas that are part of the panicked kernel's memory.
>> >>> >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
>> >>> >> >> , for details)
>> >>> >> >
>> >>> >> > Right.
>> >>> >> >
>> >>> >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
>> >>> >> >> with the crashkernel memory range:
>> >>> >> >>
>> >>> >> >>                 /* add linux,usable-memory-range */
>> >>> >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
>> >>> >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
>> >>> >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
>> >>> >> >>                                 address_cells, size_cells);
>> >>> >> >>
>> >>> >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
>> >>> >> >> , for details)
>> >>> >> >>
>> >>> >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
>> >>> >> >> they are marked as System RAM or as RESERVED. As,
>> >>> >> >> 'linux,usable-memory-range' dt node is patched up only with
>> >>> >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
>> >>> >> >>
>> >>> >> >> 3). As a result when the crashkernel boots up it doesn't find this
>> >>> >> >> ACPI memory and crashes while trying to access the same:
>> >>> >> >>
>> >>> >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
>> >>> >> >> -r`.img --reuse-cmdline -d
>> >>> >> >>
>> >>> >> >> [snip..]
>> >>> >> >>
>> >>> >> >> Reserved memory range
>> >>> >> >> 000000000e800000-000000002e7fffff (0)
>> >>> >> >>
>> >>> >> >> Coredump memory ranges
>> >>> >> >> 0000000000000000-000000000e7fffff (0)
>> >>> >> >> 000000002e800000-000000003961ffff (0)
>> >>> >> >> 0000000039d40000-000000003ed2ffff (0)
>> >>> >> >> 000000003ed60000-000000003fbfffff (0)
>> >>> >> >> 0000001040000000-0000001ffbffffff (0)
>> >>> >> >> 0000002000000000-0000002ffbffffff (0)
>> >>> >> >> 0000009000000000-0000009ffbffffff (0)
>> >>> >> >> 000000a000000000-000000affbffffff (0)
>> >>> >> >>
>> >>> >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
>> >>> >> >> memory cap'ing passed to the crash kernel inside
>> >>> >> >> 'arch/arm64/mm/init.c' (see below):
>> >>> >> >>
>> >>> >> >> static void __init fdt_enforce_memory_region(void)
>> >>> >> >> {
>> >>> >> >>         struct memblock_region reg = {
>> >>> >> >>                 .size = 0,
>> >>> >> >>         };
>> >>> >> >>
>> >>> >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> >>> >> >>
>> >>> >> >>         if (reg.size)
>> >>> >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
>> >>> >> >> comment this out */
>> >>> >> >> }
>> >>> >> >
>> >>> >> > Please just don't do that. It can cause a fatal damage on
>> >>> >> > memory contents of the *crashed* kernel.
>> >>> >> >
>> >>> >> >> 5). Both the above temporary solutions fix the problem.
>> >>> >> >>
>> >>> >> >> 6). However exposing all System RAM regions to the crashkernel is not
>> >>> >> >> advisable and may cause the crashkernel or some crashkernel drivers to
>> >>> >> >> fail.
>> >>> >> >>
>> >>> >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
>> >>> >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
>> >>> >> >> kernel code and on the other hand the user-space 'kexec-tools' will
>> >>> >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
>> >>> >> >> dt node 'linux,usable-memory-range'
>> >>> >> >
>> >>> >> > I still don't understand why we need to carry over the information
>> >>> >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
>> >>> >> > such regions are free to be reused by the kernel after some point of
>> >>> >> > initialization. Why does crash dump kernel need to know about them?
>> >>> >> >
>> >>> >>
>> >>> >> Not really. According to the UEFI spec, they can be reclaimed after
>> >>> >> the OS has initialized, i.e., when it has consumed the ACPI tables and
>> >>> >> no longer needs them. Of course, in order to be able to boot a kexec
>> >>> >> kernel, those regions needs to be preserved, which is why they are
>> >>> >> memblock_reserve()'d now.
>> >>> >
>> >>> > For my better understandings, who is actually accessing such regions
>> >>> > during boot time, uefi itself or efistub?
>> >>> >
>> >>>
>> >>> No, only the kernel. This is where the ACPI tables are stored. For
>> >>> instance, on QEMU we have
>> >>>
>> >>>  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
>> >>>  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
>> >>>   01000013)
>> >>>  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
>> >>> BXPC 00000001)
>> >>>  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
>> >>> BXPC 00000001)
>> >>>  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
>> >>> BXPC 00000001)
>> >>>  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
>> >>> BXPC 00000001)
>> >>>  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
>> >>> BXPC 00000001)
>> >>>  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
>> >>> BXPC 00000001)
>> >>>  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
>> >>> BXPC 00000001)
>> >>>
>> >>> covered by
>> >>>
>> >>>  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
>> >>>  ...
>> >>>  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
>> >>
>> >> OK. I mistakenly understood those regions could be freed after exiting
>> >> UEFI boot services.
>> >>
>> >>>
>> >>> >> So it seems that kexec does not honour the memblock_reserve() table
>> >>> >> when booting the next kernel.
>> >>> >
>> >>> > not really.
>> >>> >
>> >>> >> > (In other words, can or should we skip some part of ACPI-related init code
>> >>> >> > on crash dump kernel?)
>> >>> >> >
>> >>> >>
>> >>> >> I don't think so. And the change to the handling of ACPI reclaim
>> >>> >> regions only revealed the bug, not created it (given that other
>> >>> >> memblock_reserve regions may be affected as well)
>> >>> >
>> >>> > As whether we should honor such reserved regions over kexec'ing
>> >>> > depends on each one's specific nature, we will have to take care one-by-one.
>> >>> > As a matter of fact, no information about "reserved" memblocks is
>> >>> > exposed to user space (via proc/iomem).
>> >>> >
>> >>>
>> >>> That is why I suggested (somewhere in this thread?) to not expose them
>> >>> as 'System RAM'. Do you think that could solve this?
>> >>
>> >> Memblock-reserv'ing them is necessary to prevent their corruption and
>> >> marking them under another name in /proc/iomem would also be good in order
>> >> not to allocate them as part of crash kernel's memory.
>> >>
>> >
>> > I agree. However, this may not be entirely trivial, since iterating
>> > over the memblock_reserved table and creating iomem entries may result
>> > in collisions.
>>
>> I found a method (using the patch I shared earlier in this thread) to mark these
>> entries as 'ACPI reclaim memory' ranges rather than System RAM or
>> reserved regions.
>>
>> >> But I'm not still convinced that we should export them in useable-
>> >> memory-range to crash dump kernel. They will be accessed through
>> >> acpi_os_map_memory() and so won't be required to be part of system ram
>> >> (or memblocks), I guess.
>> >
>> > Agreed. They will be covered by the linear mapping in the boot kernel,
>> > and be mapped explicitly via ioremap_cache() in the kexec kernel,
>> > which is exactly what we want in this case.
>>
>> Now this is what is confusing me. I don't see the above happening.
>>
>> I see that the primary kernel boots up and adds the ACPI regions via:
>> acpi_os_ioremap
>>     -> ioremap_cache
>>
>> But during the crashkernel boot, ''acpi_os_ioremap' calls
>> 'ioremap' for the ACPI Reclaim Memory regions and not the _cache
>> variant.
>>
>> And it fails while accessing the ACPI tables:
>>
>> [    0.039205] ACPI: Core revision 20170728
>> pud=000000002e7d0003, *pmd=000000002e7c0003, *pte=00e8000039710707
>> [    0.095098] Internal error: Oops: 96000021 [#1] SMP
>> [    0.100022] Modules linked in:
>> [    0.103102] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc6 #1
>> [    0.109432] task: ffff000008d05180 task.stack: ffff000008cc0000
>> [    0.115414] PC is at acpi_ns_lookup+0x25c/0x3c0
>> [    0.119987] LR is at acpi_ds_load1_begin_op+0xa4/0x294
>> [    0.125175] pc : [<ffff0000084a6764>] lr : [<ffff00000849b4f8>]
>> pstate: 60000045
>> [    0.132647] sp : ffff000008ccfb40
>> [    0.135989] x29: ffff000008ccfb40 x28: ffff000008a9f2a4
>> [    0.141354] x27: ffff0000088be820 x26: 0000000000000000
>> [    0.146718] x25: 000000000000001b x24: 0000000000000001
>> [    0.152083] x23: 0000000000000001 x22: ffff000009710027
>> [    0.157447] x21: ffff000008ccfc50 x20: 0000000000000001
>> [    0.162812] x19: 000000000000001b x18: 0000000000000005
>> [    0.168176] x17: 0000000000000000 x16: 0000000000000000
>> [    0.173541] x15: 0000000000000000 x14: 000000000000038e
>> [    0.178905] x13: ffffffff00000000 x12: ffffffffffffffff
>> [    0.184270] x11: 0000000000000006 x10: 00000000ffffff76
>> [    0.189634] x9 : 000000000000005f x8 : ffff8000126d0140
>> [    0.194998] x7 : 0000000000000000 x6 : ffff000008ccfc50
>> [    0.200362] x5 : ffff80000fe62c00 x4 : 0000000000000001
>> [    0.205727] x3 : ffff000008ccfbe0 x2 : ffff0000095e3980
>> [    0.211091] x1 : ffff000009710027 x0 : 0000000000000000
>> [    0.216456] Process swapper/0 (pid: 0, stack limit = 0xffff000008cc0000)
>> [    0.223224] Call trace:
>> [    0.225688] Exception stack(0xffff000008ccfa00 to 0xffff000008ccfb40)
>> [    0.232194] fa00: 0000000000000000 ffff000009710027
>> ffff0000095e3980 ffff000008ccfbe0
>> [    0.240106] fa20: 0000000000000001 ffff80000fe62c00
>> ffff000008ccfc50 0000000000000000
>> [    0.248018] fa40: ffff8000126d0140 000000000000005f
>> 00000000ffffff76 0000000000000006
>> [    0.255931] fa60: ffffffffffffffff ffffffff00000000
>> 000000000000038e 0000000000000000
>> [    0.263843] fa80: 0000000000000000 0000000000000000
>> 0000000000000005 000000000000001b
>> [    0.271754] faa0: 0000000000000001 ffff000008ccfc50
>> ffff000009710027 0000000000000001
>> [    0.279667] fac0: 0000000000000001 000000000000001b
>> 0000000000000000 ffff0000088be820
>> [    0.287579] fae0: ffff000008a9f2a4 ffff000008ccfb40
>> ffff00000849b4f8 ffff000008ccfb40
>> [    0.295491] fb00: ffff0000084a6764 0000000060000045
>> ffff000008ccfb40 ffff000008260a18
>> [    0.303403] fb20: ffffffffffffffff ffff0000087f3fb0
>> ffff000008ccfb40 ffff0000084a6764
>> [    0.311316] [<ffff0000084a6764>] acpi_ns_lookup+0x25c/0x3c0
>> [    0.316943] [<ffff00000849b4f8>] acpi_ds_load1_begin_op+0xa4/0x294
>> [    0.323186] [<ffff0000084ad4ac>] acpi_ps_build_named_op+0xc4/0x198
>> [    0.329428] [<ffff0000084ad6cc>] acpi_ps_create_op+0x14c/0x270
>> [    0.335319] [<ffff0000084acfa8>] acpi_ps_parse_loop+0x188/0x5c8
>> [    0.341298] [<ffff0000084ae048>] acpi_ps_parse_aml+0xb0/0x2b8
>> [    0.347101] [<ffff0000084a8e10>] acpi_ns_one_complete_parse+0x144/0x184
>> [    0.353783] [<ffff0000084a8e98>] acpi_ns_parse_table+0x48/0x68
>> [    0.359675] [<ffff0000084a82cc>] acpi_ns_load_table+0x4c/0xdc
>> [    0.365479] [<ffff0000084b32f8>] acpi_tb_load_namespace+0xe4/0x264
>> [    0.371723] [<ffff000008baf9b4>] acpi_load_tables+0x48/0xc0
>> [    0.377350] [<ffff000008badc20>] acpi_early_init+0x9c/0xd0
>> [    0.382891] [<ffff000008b70d50>] start_kernel+0x3b4/0x43c
>> [    0.388343] Code: b9008fb9 2a000318 36380054 32190318 (b94002c0)
>> [    0.394500] ---[ end trace c46ed37f9651c58e ]---
>> [    0.399160] Kernel panic - not syncing: Fatal exception
>> [    0.404437] Rebooting in 10 seconds.
>>
>> So, I think the linear mapping done by the primary kernel does not
>> make these accessible in the crash kernel directly.
>>
>> Any pointers?
>
> Can you get the code line number for acpi_ns_lookup+0x25c?

gdb points to the following code line number:

(gdb) list *(acpi_ns_lookup+0x25c)
0xffff0000084aa250 is in acpi_ns_lookup (drivers/acpi/acpica/nsaccess.c:577).
572                }
573            }
574
575            /* Extract one ACPI name from the front of the pathname */
576
577            ACPI_MOVE_32_TO_32(&simple_name, path);
578
579            /* Try to find the single (4 character) ACPI name */
580
581            status =
(gdb)

i.e. ACPI_MOVE_32_TO_32(&simple_name, path);

addr2line also confirms the same:

# addr2line -e  vmlinux ffff0000084aa250
/root/git/kernel-alt/drivers/acpi/acpica/nsaccess.c:577


Regards,
Bhupesh


>>
>> Regards,
>> Bhupesh
>>
>> >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
>> >> via a kernel command line parameter, "memmap=".
>> >>
>> _______________________________________________
>> kexec mailing list -- kexec at lists.fedoraproject.org
>> To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From bhe at redhat.com  Mon Dec 18 17:58:04 2017
From: bhe at redhat.com (Baoquan He)
Date: Tue, 19 Dec 2017 09:58:04 +0800
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
In-Reply-To: <20171218143753.k7xyq6yiyjisnonh@pd.tnic>
References: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
 <20171216010142.GK12442@x1>
 <20171217214735.nuxq5zo2eknqpbpi@pd.tnic>
 <20171218134736.GA4035@x1>
 <20171218143753.k7xyq6yiyjisnonh@pd.tnic>
Message-ID: <20171219015804.GC4035@x1>

On 12/18/17 at 03:37pm, Borislav Petkov wrote:
> On Mon, Dec 18, 2017 at 09:47:36PM +0800, Baoquan He wrote:
> >                 pr_info("Your BIOS doesn't leave an aperture memory hole\n");
> >                 pr_info("Please enable the IOMMU option in the BIOS setup\n");
> >                 pr_info("This costs you %dMB of RAM\n",
> >                         32 << fallback_aper_order);
> > 	...
> > }
> 
> There are BIOSen where there's not even an IOMMU option to enable in the
> first place. So forget fixing the firmware.

Yes, while GART might not be this case. Because in code there isn't any
information telling that ram region stealing is for borken firmware. And
even the pr_info is telling people to enable GART in bios.

In fact we at least should change the pr_info to pr_warn since this kind
of operation is not encouraged if firmware is not broken.


diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c
index f5d92bc3b884..281ba2b595aa 100644
--- a/arch/x86/kernel/aperture_64.c
+++ b/arch/x86/kernel/aperture_64.c
@@ -451,9 +451,9 @@ int __init gart_iommu_hole_init(void)
 		   force_iommu ||
 		   valid_agp ||
 		   fallback_aper_force) {
-		pr_info("Your BIOS doesn't leave an aperture memory hole\n");
-		pr_info("Please enable the IOMMU option in the BIOS setup\n");
-		pr_info("This costs you %dMB of RAM\n",
+		pr_warn("Your BIOS doesn't leave an aperture memory hole\n");
+		pr_warn("Please enable the IOMMU option in the BIOS setup\n");
+		pr_warn("This costs you %dMB of RAM\n",
 			32 << fallback_aper_order);
 
 		aper_order = fallback_aper_order;

> 
> > Previously people added gart region to iomem to notice that even though
> > there's ram mapped, while it's occupied by gart, please don't dump it.
> > Later it's reverted commit 707d4eefbdb3 ("Revert [PATCH] Insert GART
> > region into resource map").
> 
> Yes, I read Jiri's commit message, TYVM.
> 
> > The other is not to tell kdump kernel that there's ram mapped into the
> > region. In the mail I replied to Jiri's v1 post, I meant the 2nd way.
> > Remove the ram region occupied by gart from iomem, then kdump kernel
> > won't see it and won't dump it.
> 
> That's the wrong approach. Because this way you're lying in iomem about
> the layout by hiding the gart range.
> 
> What needs to happen is to *exclude* the region from the dumping side
> only, so that it doesn't touch it. Because the second kernel still needs
> to show a *correct* iomem ranges list. Imagine someone looks at it
> during debugging...

With dmesg of 1st kernel, people should know the situation. In 1st
kernel, the region of ram memory is reserved in memblock, then no anyone
will touch it. It may not make sense to debug this region.

As the pr_info is saying, that region of ram is stolen by GART, just
GART won't use it. not sure if anyone will try to debug it on purpose.

> 
> So I think Jiri's approach is the right thing to do.

Hmm, as I have said in the first replying mail, the v2 will introduce
issues:

1) If 'iommu=off' is specified in 1st kernel but not in kdump kernel, it
will ignore the ram we need dump.

2) If 'iommu=off' is specified in kdump kernel, but not in 1st kernel,
it won't get the GART region, this patch does't work.

3) If people enable GART in bios, there's a ram memory hole for GART.
Nothing need to do while kdump kernel doesn't know GART is enabled or
not in bios, will try to avoid it anyway. It won't hurt anythig though,
in logic it's not suggested since confusion will be brought in.

Thanks
Baoquan


From takahiro.akashi at linaro.org  Mon Dec 18 21:01:15 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Tue, 19 Dec 2017 14:01:15 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CAFTCetQ55zUKe25jSku0DHp8uVZA4hB32d5W6MSCNsTVpxu7Gw@mail.gmail.com>
References: <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
 <20171218055356.mq4coiibh3frii54@fireball>
 <CAFTCetQ55zUKe25jSku0DHp8uVZA4hB32d5W6MSCNsTVpxu7Gw@mail.gmail.com>
Message-ID: <20171219050113.GF28046@linaro.org>

On Mon, Dec 18, 2017 at 02:29:05PM +0530, Bhupesh SHARMA wrote:
> 
> [snip..]
> 
> [    0.000000] linux,usable-memory-range base e800000, size 20000000
> [    0.000000]  - e800000 ,  20000000
> [    0.000000] linux,usable-memory-range base 396c0000, size a0000
> [    0.000000]  - 396c0000 ,  a0000
> [    0.000000] linux,usable-memory-range base 39770000, size 40000
> [    0.000000]  - 39770000 ,  40000
> [    0.000000] linux,usable-memory-range base 398a0000, size 20000
> [    0.000000]  - 398a0000 ,  20000
> [    0.000000] initrd not fully accessible via the linear mapping --
> please check your bootloader ...

This is an odd message coming from:
|void __init arm64_memblock_init(void)
|...
|
|                if (WARN(base < memblock_start_of_DRAM() ||
|                         base + size > memblock_start_of_DRAM() +
|                                       linear_region_size,
|                        "initrd not fully accessible via the linear mapping -- please check your bootloader ...\n")) {

Can you confirm how the condition breaks here?
I suppose
    base: 0xfe70000
    size: 0x13c0000
    memblock_start_of_DRAM(): 0xe800000
according to the information you gave me.

Thanks,
-Takahiro AKASHI


> [    0.000000] ------------[ cut here ]------------
> [    0.000000] WARNING: CPU: 0 PID: 0 at arch/arm64/mm/init.c:597
> arm64_memblock_init+0x210/0x484
> [    0.000000] Modules linked in:
> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0+ #7
> [    0.000000] task: ffff000008d05580 task.stack: ffff000008cc0000
> [    0.000000] PC is at arm64_memblock_init+0x210/0x484
> [    0.000000] LR is at arm64_memblock_init+0x210/0x484
> [    0.000000] pc : [<ffff000008b76984>] lr : [<ffff000008b76984>]
> pstate: 600000c5
> [    0.000000] sp : ffff000008ccfe80
> [    0.000000] x29: ffff000008ccfe80 x28: 000000000f370018
> [    0.000000] x27: 0000000011230000 x26: 00000000013b0000
> [    0.000000] x25: 000000000fe80000 x24: ffff000008cf3000
> [    0.000000] x23: ffff000008ec0000 x22: ffff000009680000
> [    0.000000] x21: ffff000008afa000 x20: ffff000008080000
> [    0.000000] x19: ffff000008afa000 x18: 000000000c283806
> [    0.000000] x17: 0000000000000000 x16: ffff000008d05580
> [    0.000000] x15: 000000002be00842 x14: 79206b6365686320
> [    0.000000] x13: 657361656c70202d x12: 2d20676e69707061
> [    0.000000] x11: 6d207261656e696c x10: 2065687420616976
> [    0.000000] x9 : 00000000000000f4 x8 : ffff000008517414
> [    0.000000] x7 : 746f6f622072756f x6 : 000000000000000d
> [    0.000000] x5 : ffff000008c96360 x4 : 0000000000000001
> [    0.000000] x3 : 0000000000000000 x2 : 0000000000000000
> [    0.000000] x1 : 0000000000000000 x0 : 0000000000000056
> [    0.000000] Call trace:
> [    0.000000] Exception stack(0xffff000008ccfd40 to 0xffff000008ccfe80)
> [    0.000000] fd40: 0000000000000056 0000000000000000
> 0000000000000000 0000000000000000
> [    0.000000] fd60: 0000000000000001 ffff000008c96360
> 000000000000000d 746f6f622072756f
> [    0.000000] fd80: ffff000008517414 00000000000000f4
> 2065687420616976 6d207261656e696c
> [    0.000000] fda0: 2d20676e69707061 657361656c70202d
> 79206b6365686320 000000002be00842
> [    0.000000] fdc0: ffff000008d05580 0000000000000000
> 000000000c283806 ffff000008afa000
> [    0.000000] fde0: ffff000008080000 ffff000008afa000
> ffff000009680000 ffff000008ec0000
> [    0.000000] fe00: ffff000008cf3000 000000000fe80000
> 00000000013b0000 0000000011230000
> [    0.000000] fe20: 000000000f370018 ffff000008ccfe80
> ffff000008b76984 ffff000008ccfe80
> [    0.000000] fe40: ffff000008b76984 00000000600000c5
> ffff00000959b7a8 ffff000008ec0000
> [    0.000000] fe60: ffffffffffffffff 0000000000000005
> ffff000008ccfe80 ffff000008b76984
> [    0.000000] [<ffff000008b76984>] arm64_memblock_init+0x210/0x484
> [    0.000000] [<ffff000008b7398c>] setup_arch+0x1b8/0x5f4
> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
> [    0.000000] random: get_random_bytes called from
> print_oops_end_marker+0x50/0x6c with crng_init=0
> [    0.000000] ---[ end trace 0000000000000000 ]---
> [    0.000000] Reserving 4KB of memory at 0x2e7f0000 for elfcorehdr
> [    0.000000] cma: Failed to reserve 512 MiB
> [    0.000000] Kernel panic - not syncing: ERROR: Failed to allocate
> 0x0000000000010000 bytes below 0x0000000000000000.
> [    0.000000]
> [    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G        W
> ------------   4.14.0+ #7
> [    0.000000] Call trace:
> [    0.000000] [<ffff000008088da8>] dump_backtrace+0x0/0x23c
> [    0.000000] [<ffff000008089008>] show_stack+0x24/0x2c
> [    0.000000] [<ffff0000087f647c>] dump_stack+0x84/0xa8
> [    0.000000] [<ffff0000080cfd44>] panic+0x138/0x2a0
> [    0.000000] [<ffff000008b95c88>] memblock_alloc_base+0x44/0x4c
> [    0.000000] [<ffff000008b95cbc>] memblock_alloc+0x2c/0x38
> [    0.000000] [<ffff000008b772dc>] early_pgtable_alloc+0x20/0x74
> [    0.000000] [<ffff000008b7755c>] paging_init+0x28/0x544
> [    0.000000] [<ffff000008b73990>] setup_arch+0x1bc/0x5f4
> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
> [    0.000000] ---[ end Kernel panic - not syncing: ERROR: Failed to
> allocate 0x0000000000010000 bytes below 0x0000000000000000.
> [    0.000000]
> 
> I guess it is because of the 1G alignment requirement between the
> kernel image and the initrd and how we populate the holes between the
> kernel image, segments (including dtb) and the initrd from the
> kexec-tools.
> 
> Akashi, any pointers on this will be helpful as well.
> 
> Regards,
> Bhupesh
> 
> 
> >> >
> >> > Regards,
> >> > Bhupesh
> >> >
> >> > >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >> > >> via a kernel command line parameter, "memmap=".
> >> > >>
> >> > _______________________________________________
> >> > kexec mailing list -- kexec at lists.fedoraproject.org
> >> > To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From takahiro.akashi at linaro.org  Mon Dec 18 21:25:49 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Tue, 19 Dec 2017 14:25:49 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CACi5LpOmeEMuoCkTC7MrBDaA1J5a4vZ_7bh3HSC0G5GoAMUCjw@mail.gmail.com>
References: <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
 <CACi5LpOmeEMuoCkTC7MrBDaA1J5a4vZ_7bh3HSC0G5GoAMUCjw@mail.gmail.com>
Message-ID: <20171219052548.GG28046@linaro.org>

On Tue, Dec 19, 2017 at 02:58:20AM +0530, Bhupesh Sharma wrote:
> Hi Dave,
> 
> On Mon, Dec 18, 2017 at 10:46 AM, Dave Young <dyoung at redhat.com> wrote:
> > kexec at fedoraproject... is for Fedora kexec scripts discussion, changed it
> > to kexec at lists.infradead.org
> >
> > Also add linux-acpi list
> > On 12/18/17 at 02:31am, Bhupesh Sharma wrote:
> >> On Fri, Dec 15, 2017 at 3:05 PM, Ard Biesheuvel
> >> <ard.biesheuvel at linaro.org> wrote:
> >> > On 15 December 2017 at 09:59, AKASHI Takahiro
> >> > <takahiro.akashi at linaro.org> wrote:
> >> >> On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> >> >>> On 13 December 2017 at 12:16, AKASHI Takahiro
> >> >>> <takahiro.akashi at linaro.org> wrote:
> >> >>> > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> >> >>> >> On 13 December 2017 at 10:26, AKASHI Takahiro
> >> >>> >> <takahiro.akashi at linaro.org> wrote:
> >> >>> >> > Bhupesh, Ard,
> >> >>> >> >
> >> >>> >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> >> >>> >> >> Hi Ard, Akashi
> >> >>> >> >>
> >> >>> >> > (snip)
> >> >>> >> >
> >> >>> >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> >> >>> >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> >> >>> >> >> identify its own usable memory and exclude, at its boot time, any
> >> >>> >> >> other memory areas that are part of the panicked kernel's memory.
> >> >>> >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> >> >>> >> >> , for details)
> >> >>> >> >
> >> >>> >> > Right.
> >> >>> >> >
> >> >>> >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> >> >>> >> >> with the crashkernel memory range:
> >> >>> >> >>
> >> >>> >> >>                 /* add linux,usable-memory-range */
> >> >>> >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> >> >>> >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> >> >>> >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> >> >>> >> >>                                 address_cells, size_cells);
> >> >>> >> >>
> >> >>> >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> >> >>> >> >> , for details)
> >> >>> >> >>
> >> >>> >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> >> >>> >> >> they are marked as System RAM or as RESERVED. As,
> >> >>> >> >> 'linux,usable-memory-range' dt node is patched up only with
> >> >>> >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> >> >>> >> >>
> >> >>> >> >> 3). As a result when the crashkernel boots up it doesn't find this
> >> >>> >> >> ACPI memory and crashes while trying to access the same:
> >> >>> >> >>
> >> >>> >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> >> >>> >> >> -r`.img --reuse-cmdline -d
> >> >>> >> >>
> >> >>> >> >> [snip..]
> >> >>> >> >>
> >> >>> >> >> Reserved memory range
> >> >>> >> >> 000000000e800000-000000002e7fffff (0)
> >> >>> >> >>
> >> >>> >> >> Coredump memory ranges
> >> >>> >> >> 0000000000000000-000000000e7fffff (0)
> >> >>> >> >> 000000002e800000-000000003961ffff (0)
> >> >>> >> >> 0000000039d40000-000000003ed2ffff (0)
> >> >>> >> >> 000000003ed60000-000000003fbfffff (0)
> >> >>> >> >> 0000001040000000-0000001ffbffffff (0)
> >> >>> >> >> 0000002000000000-0000002ffbffffff (0)
> >> >>> >> >> 0000009000000000-0000009ffbffffff (0)
> >> >>> >> >> 000000a000000000-000000affbffffff (0)
> >> >>> >> >>
> >> >>> >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> >> >>> >> >> memory cap'ing passed to the crash kernel inside
> >> >>> >> >> 'arch/arm64/mm/init.c' (see below):
> >> >>> >> >>
> >> >>> >> >> static void __init fdt_enforce_memory_region(void)
> >> >>> >> >> {
> >> >>> >> >>         struct memblock_region reg = {
> >> >>> >> >>                 .size = 0,
> >> >>> >> >>         };
> >> >>> >> >>
> >> >>> >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >> >>> >> >>
> >> >>> >> >>         if (reg.size)
> >> >>> >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> >> >>> >> >> comment this out */
> >> >>> >> >> }
> >> >>> >> >
> >> >>> >> > Please just don't do that. It can cause a fatal damage on
> >> >>> >> > memory contents of the *crashed* kernel.
> >> >>> >> >
> >> >>> >> >> 5). Both the above temporary solutions fix the problem.
> >> >>> >> >>
> >> >>> >> >> 6). However exposing all System RAM regions to the crashkernel is not
> >> >>> >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> >> >>> >> >> fail.
> >> >>> >> >>
> >> >>> >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> >> >>> >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> >> >>> >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> >> >>> >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> >> >>> >> >> dt node 'linux,usable-memory-range'
> >> >>> >> >
> >> >>> >> > I still don't understand why we need to carry over the information
> >> >>> >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> >> >>> >> > such regions are free to be reused by the kernel after some point of
> >> >>> >> > initialization. Why does crash dump kernel need to know about them?
> >> >>> >> >
> >> >>> >>
> >> >>> >> Not really. According to the UEFI spec, they can be reclaimed after
> >> >>> >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> >> >>> >> no longer needs them. Of course, in order to be able to boot a kexec
> >> >>> >> kernel, those regions needs to be preserved, which is why they are
> >> >>> >> memblock_reserve()'d now.
> >> >>> >
> >> >>> > For my better understandings, who is actually accessing such regions
> >> >>> > during boot time, uefi itself or efistub?
> >> >>> >
> >> >>>
> >> >>> No, only the kernel. This is where the ACPI tables are stored. For
> >> >>> instance, on QEMU we have
> >> >>>
> >> >>>  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> >> >>>  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> >> >>>   01000013)
> >> >>>  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> >> >>> BXPC 00000001)
> >> >>>  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> >> >>> BXPC 00000001)
> >> >>>  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> >> >>> BXPC 00000001)
> >> >>>  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> >> >>> BXPC 00000001)
> >> >>>  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> >> >>> BXPC 00000001)
> >> >>>  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> >> >>> BXPC 00000001)
> >> >>>  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> >> >>> BXPC 00000001)
> >> >>>
> >> >>> covered by
> >> >>>
> >> >>>  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> >> >>>  ...
> >> >>>  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> >> >>
> >> >> OK. I mistakenly understood those regions could be freed after exiting
> >> >> UEFI boot services.
> >> >>
> >> >>>
> >> >>> >> So it seems that kexec does not honour the memblock_reserve() table
> >> >>> >> when booting the next kernel.
> >> >>> >
> >> >>> > not really.
> >> >>> >
> >> >>> >> > (In other words, can or should we skip some part of ACPI-related init code
> >> >>> >> > on crash dump kernel?)
> >> >>> >> >
> >> >>> >>
> >> >>> >> I don't think so. And the change to the handling of ACPI reclaim
> >> >>> >> regions only revealed the bug, not created it (given that other
> >> >>> >> memblock_reserve regions may be affected as well)
> >> >>> >
> >> >>> > As whether we should honor such reserved regions over kexec'ing
> >> >>> > depends on each one's specific nature, we will have to take care one-by-one.
> >> >>> > As a matter of fact, no information about "reserved" memblocks is
> >> >>> > exposed to user space (via proc/iomem).
> >> >>> >
> >> >>>
> >> >>> That is why I suggested (somewhere in this thread?) to not expose them
> >> >>> as 'System RAM'. Do you think that could solve this?
> >> >>
> >> >> Memblock-reserv'ing them is necessary to prevent their corruption and
> >> >> marking them under another name in /proc/iomem would also be good in order
> >> >> not to allocate them as part of crash kernel's memory.
> >> >>
> >> >
> >> > I agree. However, this may not be entirely trivial, since iterating
> >> > over the memblock_reserved table and creating iomem entries may result
> >> > in collisions.
> >>
> >> I found a method (using the patch I shared earlier in this thread) to mark these
> >> entries as 'ACPI reclaim memory' ranges rather than System RAM or
> >> reserved regions.
> >>
> >> >> But I'm not still convinced that we should export them in useable-
> >> >> memory-range to crash dump kernel. They will be accessed through
> >> >> acpi_os_map_memory() and so won't be required to be part of system ram
> >> >> (or memblocks), I guess.
> >> >
> >> > Agreed. They will be covered by the linear mapping in the boot kernel,
> >> > and be mapped explicitly via ioremap_cache() in the kexec kernel,
> >> > which is exactly what we want in this case.
> >>
> >> Now this is what is confusing me. I don't see the above happening.
> >>
> >> I see that the primary kernel boots up and adds the ACPI regions via:
> >> acpi_os_ioremap
> >>     -> ioremap_cache
> >>
> >> But during the crashkernel boot, ''acpi_os_ioremap' calls
> >> 'ioremap' for the ACPI Reclaim Memory regions and not the _cache
> >> variant.
> >>
> >> And it fails while accessing the ACPI tables:
> >>
> >> [    0.039205] ACPI: Core revision 20170728
> >> pud=000000002e7d0003, *pmd=000000002e7c0003, *pte=00e8000039710707
> >> [    0.095098] Internal error: Oops: 96000021 [#1] SMP
> >> [    0.100022] Modules linked in:
> >> [    0.103102] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc6 #1
> >> [    0.109432] task: ffff000008d05180 task.stack: ffff000008cc0000
> >> [    0.115414] PC is at acpi_ns_lookup+0x25c/0x3c0
> >> [    0.119987] LR is at acpi_ds_load1_begin_op+0xa4/0x294
> >> [    0.125175] pc : [<ffff0000084a6764>] lr : [<ffff00000849b4f8>]
> >> pstate: 60000045
> >> [    0.132647] sp : ffff000008ccfb40
> >> [    0.135989] x29: ffff000008ccfb40 x28: ffff000008a9f2a4
> >> [    0.141354] x27: ffff0000088be820 x26: 0000000000000000
> >> [    0.146718] x25: 000000000000001b x24: 0000000000000001
> >> [    0.152083] x23: 0000000000000001 x22: ffff000009710027
> >> [    0.157447] x21: ffff000008ccfc50 x20: 0000000000000001
> >> [    0.162812] x19: 000000000000001b x18: 0000000000000005
> >> [    0.168176] x17: 0000000000000000 x16: 0000000000000000
> >> [    0.173541] x15: 0000000000000000 x14: 000000000000038e
> >> [    0.178905] x13: ffffffff00000000 x12: ffffffffffffffff
> >> [    0.184270] x11: 0000000000000006 x10: 00000000ffffff76
> >> [    0.189634] x9 : 000000000000005f x8 : ffff8000126d0140
> >> [    0.194998] x7 : 0000000000000000 x6 : ffff000008ccfc50
> >> [    0.200362] x5 : ffff80000fe62c00 x4 : 0000000000000001
> >> [    0.205727] x3 : ffff000008ccfbe0 x2 : ffff0000095e3980
> >> [    0.211091] x1 : ffff000009710027 x0 : 0000000000000000
> >> [    0.216456] Process swapper/0 (pid: 0, stack limit = 0xffff000008cc0000)
> >> [    0.223224] Call trace:
> >> [    0.225688] Exception stack(0xffff000008ccfa00 to 0xffff000008ccfb40)
> >> [    0.232194] fa00: 0000000000000000 ffff000009710027
> >> ffff0000095e3980 ffff000008ccfbe0
> >> [    0.240106] fa20: 0000000000000001 ffff80000fe62c00
> >> ffff000008ccfc50 0000000000000000
> >> [    0.248018] fa40: ffff8000126d0140 000000000000005f
> >> 00000000ffffff76 0000000000000006
> >> [    0.255931] fa60: ffffffffffffffff ffffffff00000000
> >> 000000000000038e 0000000000000000
> >> [    0.263843] fa80: 0000000000000000 0000000000000000
> >> 0000000000000005 000000000000001b
> >> [    0.271754] faa0: 0000000000000001 ffff000008ccfc50
> >> ffff000009710027 0000000000000001
> >> [    0.279667] fac0: 0000000000000001 000000000000001b
> >> 0000000000000000 ffff0000088be820
> >> [    0.287579] fae0: ffff000008a9f2a4 ffff000008ccfb40
> >> ffff00000849b4f8 ffff000008ccfb40
> >> [    0.295491] fb00: ffff0000084a6764 0000000060000045
> >> ffff000008ccfb40 ffff000008260a18
> >> [    0.303403] fb20: ffffffffffffffff ffff0000087f3fb0
> >> ffff000008ccfb40 ffff0000084a6764
> >> [    0.311316] [<ffff0000084a6764>] acpi_ns_lookup+0x25c/0x3c0
> >> [    0.316943] [<ffff00000849b4f8>] acpi_ds_load1_begin_op+0xa4/0x294
> >> [    0.323186] [<ffff0000084ad4ac>] acpi_ps_build_named_op+0xc4/0x198
> >> [    0.329428] [<ffff0000084ad6cc>] acpi_ps_create_op+0x14c/0x270
> >> [    0.335319] [<ffff0000084acfa8>] acpi_ps_parse_loop+0x188/0x5c8
> >> [    0.341298] [<ffff0000084ae048>] acpi_ps_parse_aml+0xb0/0x2b8
> >> [    0.347101] [<ffff0000084a8e10>] acpi_ns_one_complete_parse+0x144/0x184
> >> [    0.353783] [<ffff0000084a8e98>] acpi_ns_parse_table+0x48/0x68
> >> [    0.359675] [<ffff0000084a82cc>] acpi_ns_load_table+0x4c/0xdc
> >> [    0.365479] [<ffff0000084b32f8>] acpi_tb_load_namespace+0xe4/0x264
> >> [    0.371723] [<ffff000008baf9b4>] acpi_load_tables+0x48/0xc0
> >> [    0.377350] [<ffff000008badc20>] acpi_early_init+0x9c/0xd0
> >> [    0.382891] [<ffff000008b70d50>] start_kernel+0x3b4/0x43c
> >> [    0.388343] Code: b9008fb9 2a000318 36380054 32190318 (b94002c0)
> >> [    0.394500] ---[ end trace c46ed37f9651c58e ]---
> >> [    0.399160] Kernel panic - not syncing: Fatal exception
> >> [    0.404437] Rebooting in 10 seconds.
> >>
> >> So, I think the linear mapping done by the primary kernel does not
> >> make these accessible in the crash kernel directly.
> >>
> >> Any pointers?
> >
> > Can you get the code line number for acpi_ns_lookup+0x25c?
> 
> gdb points to the following code line number:
> 
> (gdb) list *(acpi_ns_lookup+0x25c)
> 0xffff0000084aa250 is in acpi_ns_lookup (drivers/acpi/acpica/nsaccess.c:577).
> 572                }
> 573            }
> 574
> 575            /* Extract one ACPI name from the front of the pathname */
> 576
> 577            ACPI_MOVE_32_TO_32(&simple_name, path);
> 578
> 579            /* Try to find the single (4 character) ACPI name */
> 580
> 581            status =
> (gdb)
> 
> i.e. ACPI_MOVE_32_TO_32(&simple_name, path);

This macro can be defined in two ways depending on
ACPI_MISALIGNMENT_NOT_SUPPORTED in drivers/acpi/acpica/acmarcos.h.
So, in principle, any use of ioremap() in acpi_os_ioremap() may be
in conflict with those definitions here.

This suggests that, under the current code base, we must expose
ACPI reclaim regions as memblocks (i.e. via usable-memory-range)
in order to avoid the reported issue.

Thanks,
-Takahiro AKASHI

> addr2line also confirms the same:
> 
> # addr2line -e  vmlinux ffff0000084aa250
> /root/git/kernel-alt/drivers/acpi/acpica/nsaccess.c:577
> 
> 
> Regards,
> Bhupesh
> 
> 
> >>
> >> Regards,
> >> Bhupesh
> >>
> >> >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >> >> via a kernel command line parameter, "memmap=".
> >> >>
> >> _______________________________________________
> >> kexec mailing list -- kexec at lists.fedoraproject.org
> >> To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From jbohac at suse.cz  Tue Dec 19 09:58:27 2017
From: jbohac at suse.cz (Jiri Bohac)
Date: Tue, 19 Dec 2017 18:58:27 +0100
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
In-Reply-To: <20171219015804.GC4035@x1>
References: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
 <20171216010142.GK12442@x1>
 <20171217214735.nuxq5zo2eknqpbpi@pd.tnic>
 <20171218134736.GA4035@x1>
 <20171218143753.k7xyq6yiyjisnonh@pd.tnic>
 <20171219015804.GC4035@x1>
Message-ID: <20171219175827.oqfskuax6zzm2ljq@dwarf.suse.cz>


On Tue, Dec 19, 2017 at 09:58:04AM +0800, Baoquan He wrote:
> Hmm, as I have said in the first replying mail, the v2 will introduce
> issues:
> 
> 1) If 'iommu=off' is specified in 1st kernel but not in kdump kernel, it
> will ignore the ram we need dump.

yes, instead of crashing the machine (because GART may be initialized in the
2nd kernel, overlapping the 1st kernel memory, which the 2nd kernel with its
fake e820 map sees as unused).

I'd say this is an improvement.

> 2) If 'iommu=off' is specified in kdump kernel, but not in 1st kernel,
> it won't get the GART region, this patch does't work.

No. It will work:

First kernel initializes the GART (either in a hole properly provided by the
BIOS or overlapping e820 RAM).

Second kernel will start with the GART initialized.  In gart_iommu_hole_init()
the setting is read from the northbridge registers and verified as valid. It
does not overlap e820 memory, because the second kernel has a fake e820 map
only spanning the crashkernel= reserved range. "fix" is never set to 1, so it
will exclude GART from vmcore in this path:

out:
        if (!fix && !fallback_aper_force) {
                if (last_aper_base) {
                        exclude_from_vmcore(last_aper_base, last_aper_order);
                        return 1;

(fix is never set to 1)
no_iommu is only checked after that.


> 3) If people enable GART in bios, there's a ram memory hole for GART.
> Nothing need to do while kdump kernel doesn't know GART is enabled or
> not in bios, will try to avoid it anyway. It won't hurt anythig though,
> in logic it's not suggested since confusion will be brought in.

I don't have easy access to the HP machines. I have a machine right here in our
lab that has this issue. It has no "enable GART" setting in BIOS.  It has a
"enable IOMMU" setting. The bug stays there regardless of the setting.
It's old. Noone will fix the firmware. The patch fixes it.

Thanks,

-- 
Jiri Bohac <jbohac at suse.cz>
SUSE Labs, Prague, Czechia



From dyoung at redhat.com  Tue Dec 19 23:24:43 2017
From: dyoung at redhat.com (Dave Young)
Date: Wed, 20 Dec 2017 15:24:43 +0800
Subject: [PATCH] handle renamed init_level4_pgt -> init_top_pgt
In-Reply-To: <0910DD04CBD6DE4193FCF86B9C00BE9701F2A617@BPXM01GP.gisp.nec.co.jp>
References: <0910DD04CBD6DE4193FCF86B9C00BE9701F22E63@BPXM01GP.gisp.nec.co.jp>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2A617@BPXM01GP.gisp.nec.co.jp>
Message-ID: <20171220072443.GA7413@dhcp-128-65.nay.redhat.com>

On 12/08/17 at 08:15am, Atsushi Kumagai wrote:
> >>Linux 4.13 renamed init_level4_pgt to init_top_pgt in preparation for
> >>introducing 5-level page tables.  This patch follows the rename if
> >>the lookup for init_level4_pgt fails.  It also checks to see if
> >>5-level page tables are enabled and bails if it discovers they are.
> >
> >Thanks Jeff, but could you rebase it on the current devel branch ?
> >vtop4 for x86_64 was modified in the commit below:
> >
> >commit 8c89727155f4994b4e75a659e28e5eff16ff6cbc
> >Author: Takao Indoh <indou.takao at jp.fujitsu.com>
> >Date:   Thu Oct 26 20:32:54 2017 +0900
> >
> >    [PATCH v3 2/4] Introduce vtop4_x86_64_pagetable
> >
> >Regards,
> >Atsushi Kumagai
> 
> I rebased that and updated the devel branch:
> 
> https://sourceforge.net/p/makedumpfile/code/ci/97e156b493c46fabfc4b136ab834a6495166ac66/
> 
> I'll merge it into v1.6.3 if you have no comment.

[dyoung at dhcp-*-* makedumpfile]$ sudo ./makedumpfile -l -d 31 /mnt/vmcore/vmcore /tmp/vmcore.1
The kernel version is not supported.
The makedumpfile operation may be incomplete.
Checking for memory holes                         : [100.0 %] |                  __vtop4_x86_64: Can't get a valid pte.
readmem: Can't convert a virtual address(ffff88007ebb1000) to physical address.
readmem: type_addr: 0, addr:ffff88007ebb1000, size:32768
__exclude_unnecessary_pages: Can't read the buffer of struct page.
create_2nd_bitmap: Can't exclude unnecessary pages.

makedumpfile Failed.

If you need the vmcore for debugging please let me know, my test is just
a normal test in kvm guest.

> 
> Thanks,
> Atsushi Kumagai
> 
> >>Signed-off-by: Jeff Mahoney <jeffm at suse.com>
> >>---
> >> arch/x86_64.c  | 24 +++++++++++++++++++++---
> >> makedumpfile.c |  6 ++++++
> >> makedumpfile.h |  2 ++
> >> 3 files changed, 29 insertions(+), 3 deletions(-)
> >>
> >>diff --git a/arch/x86_64.c b/arch/x86_64.c
> >>index 08dd6b2..9b09035 100644
> >>--- a/arch/x86_64.c
> >>+++ b/arch/x86_64.c
> >>@@ -259,16 +259,26 @@ vtop4_x86_64(unsigned long vaddr)
> >> {
> >> 	unsigned long page_dir, pml4, pgd_paddr, pgd_pte, pmd_paddr, pmd_pte;
> >> 	unsigned long pte_paddr, pte;
> >>+	unsigned long init_level4_pgt;
> >>
> >>-	if (SYMBOL(init_level4_pgt) == NOT_FOUND_SYMBOL) {
> >>+	if (SYMBOL(init_level4_pgt) != NOT_FOUND_SYMBOL)
> >>+		init_level4_pgt = SYMBOL(init_level4_pgt);
> >>+	else if (SYMBOL(init_top_pgt) != NOT_FOUND_SYMBOL)
> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
> >>+	else {
> >> 		ERRMSG("Can't get the symbol of init_level4_pgt.\n");
> >> 		return NOT_PADDR;
> >> 	}
> >>
> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
> >>+		ERRMSG("Kernel is built with 5-level page tables\n");
> >>+		return NOT_PADDR;
> >>+	}
> >>+
> >> 	/*
> >> 	 * Get PGD.
> >> 	 */
> >>-	page_dir = SYMBOL(init_level4_pgt) - __START_KERNEL_map + info->phys_base;
> >>+	page_dir = init_level4_pgt - __START_KERNEL_map + info->phys_base;
> >> 	if (is_xen_memory()) {
> >> 		page_dir = ptom_xen(page_dir);
> >> 		if (page_dir == NOT_PADDR)
> >>@@ -549,8 +559,16 @@ find_vmemmap_x86_64()
> >> 	struct vmap_pfns *vmapp, *vmaphead = NULL, *cur, *tail;
> >>
> >> 	init_level4_pgt = SYMBOL(init_level4_pgt);
> >>+	if (init_level4_pgt == NOT_FOUND_SYMBOL)
> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
> >>+
> >> 	if (init_level4_pgt == NOT_FOUND_SYMBOL) {
> >>-		ERRMSG("init_level4_pgt not found\n");
> >>+		ERRMSG("init_level4_pgt/init_top_pgt not found\n");
> >>+		return FAILED;
> >>+	}
> >>+
> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
> >>+		ERRMSG("kernel is configured for 5-level page tables\n");
> >> 		return FAILED;
> >> 	}
> >> 	pagestructsize = size_table.page;
> >>diff --git a/makedumpfile.c b/makedumpfile.c
> >>index f85003a..6e5ec34 100644
> >>--- a/makedumpfile.c
> >>+++ b/makedumpfile.c
> >>@@ -1486,6 +1486,8 @@ get_symbol_info(void)
> >> 	SYMBOL_INIT(_stext, "_stext");
> >> 	SYMBOL_INIT(swapper_pg_dir, "swapper_pg_dir");
> >> 	SYMBOL_INIT(init_level4_pgt, "init_level4_pgt");
> >>+	SYMBOL_INIT(level4_kernel_pgt, "level4_kernel_pgt");
> >>+	SYMBOL_INIT(init_top_pgt, "init_top_pgt");
> >> 	SYMBOL_INIT(vmlist, "vmlist");
> >> 	SYMBOL_INIT(vmap_area_list, "vmap_area_list");
> >> 	SYMBOL_INIT(node_online_map, "node_online_map");
> >>@@ -2105,6 +2107,8 @@ write_vmcoreinfo_data(void)
> >> 	WRITE_SYMBOL("_stext", _stext);
> >> 	WRITE_SYMBOL("swapper_pg_dir", swapper_pg_dir);
> >> 	WRITE_SYMBOL("init_level4_pgt", init_level4_pgt);
> >>+	WRITE_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
> >>+	WRITE_SYMBOL("init_top_pgt", init_top_pgt);
> >> 	WRITE_SYMBOL("vmlist", vmlist);
> >> 	WRITE_SYMBOL("vmap_area_list", vmap_area_list);
> >> 	WRITE_SYMBOL("node_online_map", node_online_map);
> >>@@ -2500,6 +2504,8 @@ read_vmcoreinfo(void)
> >> 	READ_SYMBOL("_stext", _stext);
> >> 	READ_SYMBOL("swapper_pg_dir", swapper_pg_dir);
> >> 	READ_SYMBOL("init_level4_pgt", init_level4_pgt);
> >>+	READ_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
> >>+	READ_SYMBOL("init_top_pgt", init_top_pgt);
> >> 	READ_SYMBOL("vmlist", vmlist);
> >> 	READ_SYMBOL("vmap_area_list", vmap_area_list);
> >> 	READ_SYMBOL("node_online_map", node_online_map);
> >>diff --git a/makedumpfile.h b/makedumpfile.h
> >>index 8a05794..9357e47 100644
> >>--- a/makedumpfile.h
> >>+++ b/makedumpfile.h
> >>@@ -1517,6 +1517,8 @@ struct symbol_table {
> >> 	unsigned long long	_stext;
> >> 	unsigned long long	swapper_pg_dir;
> >> 	unsigned long long	init_level4_pgt;
> >>+	unsigned long long	level4_kernel_pgt;
> >>+	unsigned long long	init_top_pgt;
> >> 	unsigned long long	vmlist;
> >> 	unsigned long long	vmap_area_list;
> >> 	unsigned long long	phys_base;
> >>
> >>
> >>--
> >>Jeff Mahoney
> >>SUSE Labs
> >
> >
> >
> >_______________________________________________
> >kexec mailing list
> >kexec at lists.infradead.org
> >http://lists.infradead.org/mailman/listinfo/kexec
> 
> 
> _______________________________________________
> kexec mailing list
> kexec at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

Thanks
Dave


From bhsharma at redhat.com  Mon Dec 18 14:28:28 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Tue, 19 Dec 2017 03:58:28 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171218111844.xyed3bel2wl6el3p@fireball>
References: <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
 <20171218055356.mq4coiibh3frii54@fireball>
 <CAFTCetQ55zUKe25jSku0DHp8uVZA4hB32d5W6MSCNsTVpxu7Gw@mail.gmail.com>
 <20171218111844.xyed3bel2wl6el3p@fireball>
Message-ID: <CACi5LpP_acpMT8B7xAHVJF31b8PrSLUovdXRLMn2+OuOHv8TgQ@mail.gmail.com>

On Mon, Dec 18, 2017 at 4:48 PM, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
> Bhupesh,
>
> On Mon, Dec 18, 2017 at 02:29:05PM +0530, Bhupesh SHARMA wrote:
>> On Mon, Dec 18, 2017 at 11:24 AM, AKASHI Takahiro
>> <takahiro.akashi at linaro.org> wrote:
>> > On Mon, Dec 18, 2017 at 01:16:57PM +0800, Dave Young wrote:
>> >> kexec at fedoraproject... is for Fedora kexec scripts discussion, changed it
>> >> to kexec at lists.infradead.org
>> >>
>> >> Also add linux-acpi list
>> >
>> > Thank you.
>> >
>> >> On 12/18/17 at 02:31am, Bhupesh Sharma wrote:
>> >> > On Fri, Dec 15, 2017 at 3:05 PM, Ard Biesheuvel
>> >> > <ard.biesheuvel at linaro.org> wrote:
>> >> > > On 15 December 2017 at 09:59, AKASHI Takahiro
>> >> > > <takahiro.akashi at linaro.org> wrote:
>> >> > >> On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
>> >> > >>> On 13 December 2017 at 12:16, AKASHI Takahiro
>> >> > >>> <takahiro.akashi at linaro.org> wrote:
>> >> > >>> > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
>> >> > >>> >> On 13 December 2017 at 10:26, AKASHI Takahiro
>> >> > >>> >> <takahiro.akashi at linaro.org> wrote:
>> >> > >>> >> > Bhupesh, Ard,
>> >> > >>> >> >
>> >> > >>> >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
>> >> > >>> >> >> Hi Ard, Akashi
>> >> > >>> >> >>
>> >> > >>> >> > (snip)
>> >> > >>> >> >
>> >> > >>> >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
>> >> > >>> >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
>> >> > >>> >> >> identify its own usable memory and exclude, at its boot time, any
>> >> > >>> >> >> other memory areas that are part of the panicked kernel's memory.
>> >> > >>> >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
>> >> > >>> >> >> , for details)
>> >> > >>> >> >
>> >> > >>> >> > Right.
>> >> > >>> >> >
>> >> > >>> >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
>> >> > >>> >> >> with the crashkernel memory range:
>> >> > >>> >> >>
>> >> > >>> >> >>                 /* add linux,usable-memory-range */
>> >> > >>> >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
>> >> > >>> >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
>> >> > >>> >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
>> >> > >>> >> >>                                 address_cells, size_cells);
>> >> > >>> >> >>
>> >> > >>> >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
>> >> > >>> >> >> , for details)
>> >> > >>> >> >>
>> >> > >>> >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
>> >> > >>> >> >> they are marked as System RAM or as RESERVED. As,
>> >> > >>> >> >> 'linux,usable-memory-range' dt node is patched up only with
>> >> > >>> >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
>> >> > >>> >> >>
>> >> > >>> >> >> 3). As a result when the crashkernel boots up it doesn't find this
>> >> > >>> >> >> ACPI memory and crashes while trying to access the same:
>> >> > >>> >> >>
>> >> > >>> >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
>> >> > >>> >> >> -r`.img --reuse-cmdline -d
>> >> > >>> >> >>
>> >> > >>> >> >> [snip..]
>> >> > >>> >> >>
>> >> > >>> >> >> Reserved memory range
>> >> > >>> >> >> 000000000e800000-000000002e7fffff (0)
>> >> > >>> >> >>
>> >> > >>> >> >> Coredump memory ranges
>> >> > >>> >> >> 0000000000000000-000000000e7fffff (0)
>> >> > >>> >> >> 000000002e800000-000000003961ffff (0)
>> >> > >>> >> >> 0000000039d40000-000000003ed2ffff (0)
>> >> > >>> >> >> 000000003ed60000-000000003fbfffff (0)
>> >> > >>> >> >> 0000001040000000-0000001ffbffffff (0)
>> >> > >>> >> >> 0000002000000000-0000002ffbffffff (0)
>> >> > >>> >> >> 0000009000000000-0000009ffbffffff (0)
>> >> > >>> >> >> 000000a000000000-000000affbffffff (0)
>> >> > >>> >> >>
>> >> > >>> >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
>> >> > >>> >> >> memory cap'ing passed to the crash kernel inside
>> >> > >>> >> >> 'arch/arm64/mm/init.c' (see below):
>> >> > >>> >> >>
>> >> > >>> >> >> static void __init fdt_enforce_memory_region(void)
>> >> > >>> >> >> {
>> >> > >>> >> >>         struct memblock_region reg = {
>> >> > >>> >> >>                 .size = 0,
>> >> > >>> >> >>         };
>> >> > >>> >> >>
>> >> > >>> >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> >> > >>> >> >>
>> >> > >>> >> >>         if (reg.size)
>> >> > >>> >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
>> >> > >>> >> >> comment this out */
>> >> > >>> >> >> }
>> >> > >>> >> >
>> >> > >>> >> > Please just don't do that. It can cause a fatal damage on
>> >> > >>> >> > memory contents of the *crashed* kernel.
>> >> > >>> >> >
>> >> > >>> >> >> 5). Both the above temporary solutions fix the problem.
>> >> > >>> >> >>
>> >> > >>> >> >> 6). However exposing all System RAM regions to the crashkernel is not
>> >> > >>> >> >> advisable and may cause the crashkernel or some crashkernel drivers to
>> >> > >>> >> >> fail.
>> >> > >>> >> >>
>> >> > >>> >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
>> >> > >>> >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
>> >> > >>> >> >> kernel code and on the other hand the user-space 'kexec-tools' will
>> >> > >>> >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
>> >> > >>> >> >> dt node 'linux,usable-memory-range'
>> >> > >>> >> >
>> >> > >>> >> > I still don't understand why we need to carry over the information
>> >> > >>> >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
>> >> > >>> >> > such regions are free to be reused by the kernel after some point of
>> >> > >>> >> > initialization. Why does crash dump kernel need to know about them?
>> >> > >>> >> >
>> >> > >>> >>
>> >> > >>> >> Not really. According to the UEFI spec, they can be reclaimed after
>> >> > >>> >> the OS has initialized, i.e., when it has consumed the ACPI tables and
>> >> > >>> >> no longer needs them. Of course, in order to be able to boot a kexec
>> >> > >>> >> kernel, those regions needs to be preserved, which is why they are
>> >> > >>> >> memblock_reserve()'d now.
>> >> > >>> >
>> >> > >>> > For my better understandings, who is actually accessing such regions
>> >> > >>> > during boot time, uefi itself or efistub?
>> >> > >>> >
>> >> > >>>
>> >> > >>> No, only the kernel. This is where the ACPI tables are stored. For
>> >> > >>> instance, on QEMU we have
>> >> > >>>
>> >> > >>>  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
>> >> > >>>  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
>> >> > >>>   01000013)
>> >> > >>>  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
>> >> > >>> BXPC 00000001)
>> >> > >>>  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
>> >> > >>> BXPC 00000001)
>> >> > >>>  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
>> >> > >>> BXPC 00000001)
>> >> > >>>  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
>> >> > >>> BXPC 00000001)
>> >> > >>>  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
>> >> > >>> BXPC 00000001)
>> >> > >>>  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
>> >> > >>> BXPC 00000001)
>> >> > >>>  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
>> >> > >>> BXPC 00000001)
>> >> > >>>
>> >> > >>> covered by
>> >> > >>>
>> >> > >>>  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
>> >> > >>>  ...
>> >> > >>>  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
>> >> > >>
>> >> > >> OK. I mistakenly understood those regions could be freed after exiting
>> >> > >> UEFI boot services.
>> >> > >>
>> >> > >>>
>> >> > >>> >> So it seems that kexec does not honour the memblock_reserve() table
>> >> > >>> >> when booting the next kernel.
>> >> > >>> >
>> >> > >>> > not really.
>> >> > >>> >
>> >> > >>> >> > (In other words, can or should we skip some part of ACPI-related init code
>> >> > >>> >> > on crash dump kernel?)
>> >> > >>> >> >
>> >> > >>> >>
>> >> > >>> >> I don't think so. And the change to the handling of ACPI reclaim
>> >> > >>> >> regions only revealed the bug, not created it (given that other
>> >> > >>> >> memblock_reserve regions may be affected as well)
>> >> > >>> >
>> >> > >>> > As whether we should honor such reserved regions over kexec'ing
>> >> > >>> > depends on each one's specific nature, we will have to take care one-by-one.
>> >> > >>> > As a matter of fact, no information about "reserved" memblocks is
>> >> > >>> > exposed to user space (via proc/iomem).
>> >> > >>> >
>> >> > >>>
>> >> > >>> That is why I suggested (somewhere in this thread?) to not expose them
>> >> > >>> as 'System RAM'. Do you think that could solve this?
>> >> > >>
>> >> > >> Memblock-reserv'ing them is necessary to prevent their corruption and
>> >> > >> marking them under another name in /proc/iomem would also be good in order
>> >> > >> not to allocate them as part of crash kernel's memory.
>> >> > >>
>> >> > >
>> >> > > I agree. However, this may not be entirely trivial, since iterating
>> >> > > over the memblock_reserved table and creating iomem entries may result
>> >> > > in collisions.
>> >> >
>> >> > I found a method (using the patch I shared earlier in this thread) to mark these
>> >> > entries as 'ACPI reclaim memory' ranges rather than System RAM or
>> >> > reserved regions.
>> >> >
>> >> > >> But I'm not still convinced that we should export them in useable-
>> >> > >> memory-range to crash dump kernel. They will be accessed through
>> >> > >> acpi_os_map_memory() and so won't be required to be part of system ram
>> >> > >> (or memblocks), I guess.
>> >> > >
>> >> > > Agreed. They will be covered by the linear mapping in the boot kernel,
>> >> > > and be mapped explicitly via ioremap_cache() in the kexec kernel,
>> >> > > which is exactly what we want in this case.
>> >> >
>> >> > Now this is what is confusing me. I don't see the above happening.
>> >> >
>> >> > I see that the primary kernel boots up and adds the ACPI regions via:
>> >> > acpi_os_ioremap
>> >> >     -> ioremap_cache
>> >> >
>> >> > But during the crashkernel boot, ''acpi_os_ioremap' calls
>> >> > 'ioremap' for the ACPI Reclaim Memory regions and not the _cache
>> >> > variant.
>> >
>> > It is natural if that region is out of memblocks.
>>
>> Thanks for the confirmation. This was my understanding as well.
>>
>> >> > And it fails while accessing the ACPI tables:
>> >> >
>> >> > [    0.039205] ACPI: Core revision 20170728
>> >> > pud=000000002e7d0003, *pmd=000000002e7c0003, *pte=00e8000039710707
>> >> > [    0.095098] Internal error: Oops: 96000021 [#1] SMP
>> >
>> > this (ESR = 0x96000021) means that Data Abort and Alignment fault happened.
>> > As ioremap() makes the mapping as "Device memory", unaligned memory
>> > access won't be allowed.
>> >
>> >> > [    0.100022] Modules linked in:
>> >> > [    0.103102] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-rc6 #1
>> >> > [    0.109432] task: ffff000008d05180 task.stack: ffff000008cc0000
>> >> > [    0.115414] PC is at acpi_ns_lookup+0x25c/0x3c0
>> >> > [    0.119987] LR is at acpi_ds_load1_begin_op+0xa4/0x294
>> >> > [    0.125175] pc : [<ffff0000084a6764>] lr : [<ffff00000849b4f8>]
>> >> > pstate: 60000045
>> >> > [    0.132647] sp : ffff000008ccfb40
>> >> > [    0.135989] x29: ffff000008ccfb40 x28: ffff000008a9f2a4
>> >> > [    0.141354] x27: ffff0000088be820 x26: 0000000000000000
>> >> > [    0.146718] x25: 000000000000001b x24: 0000000000000001
>> >> > [    0.152083] x23: 0000000000000001 x22: ffff000009710027
>> >> > [    0.157447] x21: ffff000008ccfc50 x20: 0000000000000001
>> >> > [    0.162812] x19: 000000000000001b x18: 0000000000000005
>> >> > [    0.168176] x17: 0000000000000000 x16: 0000000000000000
>> >> > [    0.173541] x15: 0000000000000000 x14: 000000000000038e
>> >> > [    0.178905] x13: ffffffff00000000 x12: ffffffffffffffff
>> >> > [    0.184270] x11: 0000000000000006 x10: 00000000ffffff76
>> >> > [    0.189634] x9 : 000000000000005f x8 : ffff8000126d0140
>> >> > [    0.194998] x7 : 0000000000000000 x6 : ffff000008ccfc50
>> >> > [    0.200362] x5 : ffff80000fe62c00 x4 : 0000000000000001
>> >> > [    0.205727] x3 : ffff000008ccfbe0 x2 : ffff0000095e3980
>> >> > [    0.211091] x1 : ffff000009710027 x0 : 0000000000000000
>> >> > [    0.216456] Process swapper/0 (pid: 0, stack limit = 0xffff000008cc0000)
>> >> > [    0.223224] Call trace:
>> >> > [    0.225688] Exception stack(0xffff000008ccfa00 to 0xffff000008ccfb40)
>> >> > [    0.232194] fa00: 0000000000000000 ffff000009710027
>> >> > ffff0000095e3980 ffff000008ccfbe0
>> >> > [    0.240106] fa20: 0000000000000001 ffff80000fe62c00
>> >> > ffff000008ccfc50 0000000000000000
>> >> > [    0.248018] fa40: ffff8000126d0140 000000000000005f
>> >> > 00000000ffffff76 0000000000000006
>> >> > [    0.255931] fa60: ffffffffffffffff ffffffff00000000
>> >> > 000000000000038e 0000000000000000
>> >> > [    0.263843] fa80: 0000000000000000 0000000000000000
>> >> > 0000000000000005 000000000000001b
>> >> > [    0.271754] faa0: 0000000000000001 ffff000008ccfc50
>> >> > ffff000009710027 0000000000000001
>> >> > [    0.279667] fac0: 0000000000000001 000000000000001b
>> >> > 0000000000000000 ffff0000088be820
>> >> > [    0.287579] fae0: ffff000008a9f2a4 ffff000008ccfb40
>> >> > ffff00000849b4f8 ffff000008ccfb40
>> >> > [    0.295491] fb00: ffff0000084a6764 0000000060000045
>> >> > ffff000008ccfb40 ffff000008260a18
>> >> > [    0.303403] fb20: ffffffffffffffff ffff0000087f3fb0
>> >> > ffff000008ccfb40 ffff0000084a6764
>> >> > [    0.311316] [<ffff0000084a6764>] acpi_ns_lookup+0x25c/0x3c0
>> >> > [    0.316943] [<ffff00000849b4f8>] acpi_ds_load1_begin_op+0xa4/0x294
>> >> > [    0.323186] [<ffff0000084ad4ac>] acpi_ps_build_named_op+0xc4/0x198
>> >> > [    0.329428] [<ffff0000084ad6cc>] acpi_ps_create_op+0x14c/0x270
>> >> > [    0.335319] [<ffff0000084acfa8>] acpi_ps_parse_loop+0x188/0x5c8
>> >> > [    0.341298] [<ffff0000084ae048>] acpi_ps_parse_aml+0xb0/0x2b8
>> >> > [    0.347101] [<ffff0000084a8e10>] acpi_ns_one_complete_parse+0x144/0x184
>> >> > [    0.353783] [<ffff0000084a8e98>] acpi_ns_parse_table+0x48/0x68
>> >> > [    0.359675] [<ffff0000084a82cc>] acpi_ns_load_table+0x4c/0xdc
>> >> > [    0.365479] [<ffff0000084b32f8>] acpi_tb_load_namespace+0xe4/0x264
>> >> > [    0.371723] [<ffff000008baf9b4>] acpi_load_tables+0x48/0xc0
>> >> > [    0.377350] [<ffff000008badc20>] acpi_early_init+0x9c/0xd0
>> >> > [    0.382891] [<ffff000008b70d50>] start_kernel+0x3b4/0x43c
>> >> > [    0.388343] Code: b9008fb9 2a000318 36380054 32190318 (b94002c0)
>> >> > [    0.394500] ---[ end trace c46ed37f9651c58e ]---
>> >> > [    0.399160] Kernel panic - not syncing: Fatal exception
>> >> > [    0.404437] Rebooting in 10 seconds.
>> >> >
>> >> > So, I think the linear mapping done by the primary kernel does not
>> >> > make these accessible in the crash kernel directly.
>> >> >
>> >> > Any pointers?
>> >>
>> >> Can you get the code line number for acpi_ns_lookup+0x25c?
>> >
>> > So should we always avoid ioremap() in acpi_os_ioremap() entirely, or
>> > modify acpi_ns_lookup() (or any acpi functions') to prevent unaligned
>> > accesses?
>> > (I didn't find out how unaligned accesses could happen there.)
>> >
>>
>> Right. Like I captured somewhere in this thread (perhaps the first
>> email on this subject),
>> this is indeed an unaligned address access.
>>
>> Now, modifying acpi_os_ioremap() to not ioremap() and thus avoiding
>> assigning this memory range
>> as device memory doesn't seem a neat solution as it means we are not
>> marking some thing with the right memory attribute and we can fall in
>> similar/related issues later.
>>
>> Regarding the later suggestion, what I am seeing now is that the acpi
>> table access functions are perhaps reused from the earlier x86
>> implementation, but on the arm64 (or even arm) arch we should not be
>> allowing unaligned accesses which might cause UNDEFINED behaviour and
>> resultant crash.
>>
>> So I can try going this approach and see if it works for me.
>>
>> However, I am still not very sure as to why the crashkernel ranges
>> historically do not include the System RAM regions (which may include
>> the ACPI regions as well). These regions are available for the kernel
>> usage and perhaps should be exported to the crashkernel as well.
>>
>> I am not fully aware of the previous discussions on capp'ing the
>> crashkernel memory being passed to the kdump kernel, but did we run
>> into any issues while doing so?
>>
>> Also, even if I extend the kexec-tools to modify the
>> linux,usable-memory-range and add the ACPI regions to it, the
>> crashkernel fails to boot with the below message (I have added some
>> logic to print the DTB on the crash kernel boot start):
>>
>> [    0.000000]     chosen {
>> [    0.000000]         linux,usable-memory-range
>> [    0.000000]  = <
>> [    0.000000] 0x00000000
>> [    0.000000] 0x0e800000
>> [    0.000000] 0x00000000
>> [    0.000000] 0x20000000
>> [    0.000000] 0x00000000
>> [    0.000000] 0x396c0000
>> [    0.000000] 0x00000000
>> [    0.000000] 0x000a0000
>> [    0.000000] 0x00000000
>> [    0.000000] 0x39770000
>> [    0.000000] 0x00000000
>> [    0.000000] 0x00040000
>> [    0.000000] 0x00000000
>> [    0.000000] 0x398a0000
>> [    0.000000] 0x00000000
>> [    0.000000] 0x00020000
>> [    0.000000] >
>> [    0.000000] ;
>>
>> [snip..]
>>
>> [    0.000000] linux,usable-memory-range base e800000, size 20000000
>> [    0.000000]  - e800000 ,  20000000
>> [    0.000000] linux,usable-memory-range base 396c0000, size a0000
>> [    0.000000]  - 396c0000 ,  a0000
>> [    0.000000] linux,usable-memory-range base 39770000, size 40000
>> [    0.000000]  - 39770000 ,  40000
>> [    0.000000] linux,usable-memory-range base 398a0000, size 20000
>> [    0.000000]  - 398a0000 ,  20000
>> [    0.000000] initrd not fully accessible via the linear mapping --
>> please check your bootloader ...
>> [    0.000000] ------------[ cut here ]------------
>> [    0.000000] WARNING: CPU: 0 PID: 0 at arch/arm64/mm/init.c:597
>> arm64_memblock_init+0x210/0x484
>> [    0.000000] Modules linked in:
>> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0+ #7
>> [    0.000000] task: ffff000008d05580 task.stack: ffff000008cc0000
>> [    0.000000] PC is at arm64_memblock_init+0x210/0x484
>> [    0.000000] LR is at arm64_memblock_init+0x210/0x484
>> [    0.000000] pc : [<ffff000008b76984>] lr : [<ffff000008b76984>]
>> pstate: 600000c5
>> [    0.000000] sp : ffff000008ccfe80
>> [    0.000000] x29: ffff000008ccfe80 x28: 000000000f370018
>> [    0.000000] x27: 0000000011230000 x26: 00000000013b0000
>> [    0.000000] x25: 000000000fe80000 x24: ffff000008cf3000
>> [    0.000000] x23: ffff000008ec0000 x22: ffff000009680000
>> [    0.000000] x21: ffff000008afa000 x20: ffff000008080000
>> [    0.000000] x19: ffff000008afa000 x18: 000000000c283806
>> [    0.000000] x17: 0000000000000000 x16: ffff000008d05580
>> [    0.000000] x15: 000000002be00842 x14: 79206b6365686320
>> [    0.000000] x13: 657361656c70202d x12: 2d20676e69707061
>> [    0.000000] x11: 6d207261656e696c x10: 2065687420616976
>> [    0.000000] x9 : 00000000000000f4 x8 : ffff000008517414
>> [    0.000000] x7 : 746f6f622072756f x6 : 000000000000000d
>> [    0.000000] x5 : ffff000008c96360 x4 : 0000000000000001
>> [    0.000000] x3 : 0000000000000000 x2 : 0000000000000000
>> [    0.000000] x1 : 0000000000000000 x0 : 0000000000000056
>> [    0.000000] Call trace:
>> [    0.000000] Exception stack(0xffff000008ccfd40 to 0xffff000008ccfe80)
>> [    0.000000] fd40: 0000000000000056 0000000000000000
>> 0000000000000000 0000000000000000
>> [    0.000000] fd60: 0000000000000001 ffff000008c96360
>> 000000000000000d 746f6f622072756f
>> [    0.000000] fd80: ffff000008517414 00000000000000f4
>> 2065687420616976 6d207261656e696c
>> [    0.000000] fda0: 2d20676e69707061 657361656c70202d
>> 79206b6365686320 000000002be00842
>> [    0.000000] fdc0: ffff000008d05580 0000000000000000
>> 000000000c283806 ffff000008afa000
>> [    0.000000] fde0: ffff000008080000 ffff000008afa000
>> ffff000009680000 ffff000008ec0000
>> [    0.000000] fe00: ffff000008cf3000 000000000fe80000
>> 00000000013b0000 0000000011230000
>> [    0.000000] fe20: 000000000f370018 ffff000008ccfe80
>> ffff000008b76984 ffff000008ccfe80
>> [    0.000000] fe40: ffff000008b76984 00000000600000c5
>> ffff00000959b7a8 ffff000008ec0000
>> [    0.000000] fe60: ffffffffffffffff 0000000000000005
>> ffff000008ccfe80 ffff000008b76984
>> [    0.000000] [<ffff000008b76984>] arm64_memblock_init+0x210/0x484
>> [    0.000000] [<ffff000008b7398c>] setup_arch+0x1b8/0x5f4
>> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
>> [    0.000000] random: get_random_bytes called from
>> print_oops_end_marker+0x50/0x6c with crng_init=0
>> [    0.000000] ---[ end trace 0000000000000000 ]---
>> [    0.000000] Reserving 4KB of memory at 0x2e7f0000 for elfcorehdr
>> [    0.000000] cma: Failed to reserve 512 MiB
>> [    0.000000] Kernel panic - not syncing: ERROR: Failed to allocate
>> 0x0000000000010000 bytes below 0x0000000000000000.
>> [    0.000000]
>> [    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G        W
>> ------------   4.14.0+ #7
>> [    0.000000] Call trace:
>> [    0.000000] [<ffff000008088da8>] dump_backtrace+0x0/0x23c
>> [    0.000000] [<ffff000008089008>] show_stack+0x24/0x2c
>> [    0.000000] [<ffff0000087f647c>] dump_stack+0x84/0xa8
>> [    0.000000] [<ffff0000080cfd44>] panic+0x138/0x2a0
>> [    0.000000] [<ffff000008b95c88>] memblock_alloc_base+0x44/0x4c
>> [    0.000000] [<ffff000008b95cbc>] memblock_alloc+0x2c/0x38
>> [    0.000000] [<ffff000008b772dc>] early_pgtable_alloc+0x20/0x74
>> [    0.000000] [<ffff000008b7755c>] paging_init+0x28/0x544
>> [    0.000000] [<ffff000008b73990>] setup_arch+0x1bc/0x5f4
>> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
>> [    0.000000] ---[ end Kernel panic - not syncing: ERROR: Failed to
>> allocate 0x0000000000010000 bytes below 0x0000000000000000.
>> [    0.000000]
>>
>> I guess it is because of the 1G alignment requirement between the
>> kernel image and the initrd and how we populate the holes between the
>> kernel image, segments (including dtb) and the initrd from the
>> kexec-tools.
>>
>> Akashi, any pointers on this will be helpful as well.
>
> Please show me:
>  * "Virtual kernel memory layout" in dmesg
>  * /proc/iomem
>  * debug messages from kexec-tools (kexec -d)

So here are the changes which I have done so far in the kernel and
kexec-tools to allow mapping ACPI reclaim regions as identifiable
regions in '/proc/iomem' and to append them to the DTB property:
linux,usable-memory-range:

Linux patch: <https://github.com/bhupesh-sharma/linux/commit/88d2ff6a1c16f5aa107b567a9d9c60343e52f263>,
and

<https://github.com/bhupesh-sharma/linux/commit/23262febd29a6665d483a707a05f8869757b8848>

kexec-tools patch:
<https://github.com/bhupesh-sharma/kexec-tools/commit/3e3d7c50648b1195674d1b7667cbbfd8d899b650>

Note that I am not very clear about the hole margins that the
kexec-tools adds (so that the crashkernel's expectation that the
kernel image and initrd lie within a 1G boundary), so I have not added
my temporary changes to the github code - but any suggestions on how
to correctly put them in place would be appreciated.

And here are the rest of the inputs you asked for:

(1) # dmesg | grep -A 15 -B 4 -i "Virtual kernel memory layout"

[    0.000000] Kernel command line:
BOOT_IMAGE=/vmlinuz-4.15.0-rc2-next-20171207+
root=/dev/mapper/rhelaa_huawei--t2280--01-root ro
earlycon=pl011,mmio,0x602B0000 efi=debug memblock_debug=1
pcie_aspm=off crashkernel=512M rd.lvm.lv=rhelaa_huawei-t2280-01/root
rd.lvm.lv=rhelaa_huawei-t2280-01/swap acpi=force
console=ttyAMA0,115200
[    0.000000] PCIe ASPM is disabled
[    0.000000] software IO TLB [mem 0x35620000-0x39620000] (64MB)
mapped at [        (ptrval)-        (ptrval)]
[    0.000000] Memory: 267251520K/268169216K available (7868K kernel
code, 1764K rwdata, 3328K rodata, 1280K init, 7727K bss, 917696K
reserved, 0K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000
(   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7bdfffff0000
(126847 GB)
[    0.000000]       .text : 0x        (ptrval) - 0x        (ptrval)
(  7872 KB)
[    0.000000]     .rodata : 0x        (ptrval) - 0x        (ptrval)
(  3392 KB)
[    0.000000]       .init : 0x        (ptrval) - 0x        (ptrval)
(  1280 KB)
[    0.000000]       .data : 0x        (ptrval) - 0x        (ptrval)
(  1765 KB)
[    0.000000]        .bss : 0x        (ptrval) - 0x        (ptrval)
(  7728 KB)
[    0.000000]     fixed   : 0xffff7fdffe7b0000 - 0xffff7fdffec00000
(  4416 KB)
[    0.000000]     PCI I/O : 0xffff7fdffee00000 - 0xffff7fdfffe00000
(    16 MB)
[    0.000000]     vmemmap : 0xffff7fe000000000 - 0xffff800000000000
(   128 GB maximum)
[    0.000000]               0xffff7fe000000000 - 0xffff7fe02bff0000
(   703 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff80affc000000
(720832 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=4
[    0.000000] ftrace: allocating 29903 entries in 8 pages
[    0.000000] Hierarchical RCU implementation.

(2) # cat /proc/iomem
00000000-3961ffff : System RAM
  00080000-00b7ffff : Kernel code
  00cc0000-0166ffff : Kernel data
  0e800000-2e7fffff : Crash kernel
39620000-396bffff : reserved
396c0000-3975ffff : ACPI reclaim region
39760000-3976ffff : reserved
39770000-397affff : ACPI reclaim region
397b0000-3989ffff : reserved
398a0000-398bffff : ACPI reclaim region
398c0000-39d3ffff : reserved
39d40000-3ed2ffff : System RAM
3ed30000-3ed5ffff : reserved
3ed60000-3fbfffff : System RAM
40500000-40500fff : sbsa-gwdt.0
  40500000-40500fff : sbsa-gwdt.0
40600000-40600fff : sbsa-gwdt.0
  40600000-40600fff : sbsa-gwdt.0
60080000-6008ffff : HISI0152:00
602b0000-602b0fff : ARMH0011:00
  602b0000-602b0fff : ARMH0011:00
603c0000-603cffff : HISI0141:00
  603c0000-603cffff : HISI0141:00
a0080000-a008ffff : HISI0152:05
  a0080000-a008ffff : HISI0152:04
    a0080000-a008ffff : HISI0152:03
a00a0000-a00affff : pnp 00:01
a01b0000-a01b0fff : HISI0191:00
a2000000-a200ffff : HISI0162:01
  a2000000-a200ffff : HISI0162:01
a3000000-a300ffff : HISI0162:02
  a3000000-a300ffff : HISI0162:02
a7020000-a702ffff : PNP0D20:00
  a7020000-a702ffff : PNP0D20:00
b0000000-be7fffff : PCI Bus 0002:e8
  b0000000-b06fffff : PCI Bus 0002:e9
    b0000000-b00fffff : 0002:e9:00.0
      b0000000-b00fffff : igb
    b0100000-b01fffff : 0002:e9:00.0
    b0200000-b02fffff : 0002:e9:00.1
      b0200000-b02fffff : igb
    b0300000-b03fffff : 0002:e9:00.1
    b0400000-b04fffff : 0002:e9:00.2
      b0400000-b04fffff : igb
    b0500000-b05fffff : 0002:e9:00.3
      b0500000-b05fffff : igb
    b0600000-b0603fff : 0002:e9:00.0
      b0600000-b0603fff : igb
    b0604000-b0607fff : 0002:e9:00.1
      b0604000-b0607fff : igb
    b0608000-b060bfff : 0002:e9:00.2
      b0608000-b060bfff : igb
    b060c000-b060ffff : 0002:e9:00.3
      b060c000-b060ffff : igb
  b0700000-b0afffff : PCI Bus 0002:e9
    b0700000-b077ffff : 0002:e9:00.0
    b0780000-b07fffff : 0002:e9:00.0
    b0800000-b087ffff : 0002:e9:00.1
    b0880000-b08fffff : 0002:e9:00.1
    b0900000-b097ffff : 0002:e9:00.2
    b0980000-b09fffff : 0002:e9:00.2
    b0a00000-b0a7ffff : 0002:e9:00.3
    b0a80000-b0afffff : 0002:e9:00.3
  b0b00000-b0b0ffff : 0002:e8:00.0
be800000-beffffff : PCI ECAM
c0080000-c008ffff : HISI0152:02
  c0080000-c008ffff : HISI0152:01
c3000000-c300ffff : HISI0162:00
  c3000000-c300ffff : HISI0162:00
c5000000-c588ffff : HISI00B2:00
  c5000000-c588ffff : HISI00B2:00
c7000000-c705ffff : HISI00B2:00
  c7000000-c705ffff : HISI00B2:00
d0080000-d008ffff : HISI0152:07
  d0080000-d008ffff : HISI0152:06
d0100000-d010ffff : HISI02A1:00
  d0100000-d010ffff : HISI02A1:00
400000000-4007fffff : PCI ECAM
440000000-4ffffffff : PCI Bus 0005:00
  440000000-4407fffff : PCI Bus 0005:01
    440000000-4403fffff : 0005:01:00.0
    440400000-4407fffff : 0005:01:00.1
  440800000-4421fffff : PCI Bus 0005:01
    440800000-440bfffff : 0005:01:00.0
      440800000-440bfffff : ixgbe
    440c00000-440ffffff : 0005:01:00.1
      440c00000-440ffffff : ixgbe
    441000000-4413fffff : 0005:01:00.0
    441400000-4417fffff : 0005:01:00.0
    441800000-441bfffff : 0005:01:00.1
    441c00000-441ffffff : 0005:01:00.1
    442000000-442003fff : 0005:01:00.0
      442000000-442003fff : ixgbe
    442004000-442007fff : 0005:01:00.1
      442004000-442007fff : ixgbe
  442200000-442200fff : 0005:00:00.0
700090000-70009ffff : pnp 00:03
7000a0000-7000affff : pnp 00:05
7000b0000-7000bffff : pnp 00:06
700200000-70020ffff : pnp 00:04
740800000-740ffffff : PCI ECAM
741000000-77ffeffff : PCI Bus 0006:08
  741000000-74100ffff : 0006:08:00.0
784000000-7847fffff : PCI ECAM
784800000-7bffeffff : PCI Bus 0007:40
  784800000-7849fffff : PCI Bus 0007:41
    784800000-7849fffff : 0007:41:00.0
  786000000-787ffffff : PCI Bus 0007:41
    786000000-787ffffff : 0007:41:00.0
7c4800000-7c4ffffff : PCI ECAM
7c5000000-7fffeffff : PCI Bus 0004:48
  7c5000000-7c51fffff : PCI Bus 0004:49
    7c5000000-7c50fffff : 0004:49:00.0
    7c5100000-7c513ffff : 0004:49:00.0
      7c5100000-7c513ffff : mpt3sas
    7c5140000-7c514ffff : 0004:49:00.0
      7c5140000-7c514ffff : mpt3sas
  7c5200000-7c520ffff : 0004:48:00.0
1040000000-1ffbffffff : System RAM
2000000000-2ffbffffff : System RAM
9000000000-9ffbffffff : System RAM
a000000000-affbffffff : System RAM
400c0080000-400c008ffff : HISI0152:08
600a00a0000-600a00affff : pnp 00:08
64001000000-64001ffffff : PCI ECAM
65040000000-650ffffffff : PCI Bus 000a:10
  65040000000-6504000ffff : 000a:10:00.0
700a0090000-700a009ffff : pnp 00:0a
700a0200000-700a020ffff : pnp 00:0b
74002000000-74002ffffff : PCI ECAM
75040000000-750ffffffff : PCI Bus 000c:20
  75040000000-7504000ffff : 000c:20:00.0
78003000000-78003ffffff : PCI ECAM
79040000000-790ffffffff : PCI Bus 000d:30
  79040000000-79040000fff : 000d:30:00.0

(3)

# kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
-r`.img --reuse-cmdline -d
arch_process_options:149: command_line:
root=/dev/mapper/rhelaa_huawei--t2280--01-root ro
earlycon=pl011,mmio,0x602B0000 efi=debug memblock_debug=1
pcie_aspm=off rd.lvm.lv=rhelaa_huawei-t2280-01/root
rd.lvm.lv=rhelaa_huawei-t2280-01/swap acpi=force
console=ttyAMA0,115200
arch_process_options:151: initrd: /boot/initramfs-4.15.0-rc2-next-20171207+.img
arch_process_options:152: dtb: (null)
Try gzip decompression.
kernel: 0xffff968d0010 kernel_size: 0xdf9200
get_memory_ranges_iomem_cb: 0000000000000000 - 000000003961ffff : System RAM
get_memory_ranges_iomem_cb: 0000000039620000 - 00000000396bffff : reserved
get_memory_ranges_iomem_cb: 0000000039760000 - 000000003976ffff : reserved
get_memory_ranges_iomem_cb: 00000000397b0000 - 000000003989ffff : reserved
get_memory_ranges_iomem_cb: 00000000398c0000 - 0000000039d3ffff : reserved
get_memory_ranges_iomem_cb: 0000000039d40000 - 000000003ed2ffff : System RAM
get_memory_ranges_iomem_cb: 000000003ed30000 - 000000003ed5ffff : reserved
get_memory_ranges_iomem_cb: 000000003ed60000 - 000000003fbfffff : System RAM
get_memory_ranges_iomem_cb: 0000001040000000 - 0000001ffbffffff : System RAM
get_memory_ranges_iomem_cb: 0000002000000000 - 0000002ffbffffff : System RAM
get_memory_ranges_iomem_cb: 0000009000000000 - 0000009ffbffffff : System RAM
get_memory_ranges_iomem_cb: 000000a000000000 - 000000affbffffff : System RAM
elf_arm64_probe: Not an ELF executable.
image_arm64_load: kernel_segment: 000000000e800000
image_arm64_load: text_offset:    0000000000080000
image_arm64_load: image_size:     00000000015f0000
image_arm64_load: phys_offset:    0000000000000000
image_arm64_load: vp_offset:      ffffffffffffffff
image_arm64_load: PE format:      yes
Reserved memory range
000000000e800000-000000002e7fffff (0)
Coredump memory ranges
0000000000000000-000000000e7fffff (0)
000000002e800000-000000003961ffff (0)
0000000039d40000-000000003ed2ffff (0)
000000003ed60000-000000003fbfffff (0)
0000001040000000-0000001ffbffffff (0)
0000002000000000-0000002ffbffffff (0)
0000009000000000-0000009ffbffffff (0)
000000a000000000-000000affbffffff (0)
ACPI reclaim memory ranges
00000000396c0000-000000003975ffff (0)
0000000039770000-00000000397affff (0)
00000000398a0000-00000000398bffff (0)
crashkernel memory ranges
000000000e800000-000000002e7fffff (0)
00000000396c0000-000000003975ffff (0)
0000000039770000-00000000397affff (0)
00000000398a0000-00000000398bffff (0)
kernel symbol _text vaddr = ffff000008080000
load_crashdump_segments: page_offset:   ffff800000000000
get_crash_notes_per_cpu: crash_notes addr = 1ff7cf3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7cf3200 p_paddr = 0x1ff7cf3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7d23200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7d23200 p_paddr = 0x1ff7d23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7d53200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7d53200 p_paddr = 0x1ff7d53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7d83200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7d83200 p_paddr = 0x1ff7d83200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7db3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7db3200 p_paddr = 0x1ff7db3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7de3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7de3200 p_paddr = 0x1ff7de3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7e13200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7e13200 p_paddr = 0x1ff7e13200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7e43200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7e43200 p_paddr = 0x1ff7e43200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7e73200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7e73200 p_paddr = 0x1ff7e73200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7ea3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7ea3200 p_paddr = 0x1ff7ea3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7ed3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7ed3200 p_paddr = 0x1ff7ed3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f03200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f03200 p_paddr = 0x1ff7f03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f33200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f33200 p_paddr = 0x1ff7f33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f63200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f63200 p_paddr = 0x1ff7f63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f93200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f93200 p_paddr = 0x1ff7f93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7fc3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7fc3200 p_paddr = 0x1ff7fc3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7d13200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7d13200 p_paddr = 0x2ff7d13200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7d43200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7d43200 p_paddr = 0x2ff7d43200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7d73200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7d73200 p_paddr = 0x2ff7d73200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7da3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7da3200 p_paddr = 0x2ff7da3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7dd3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7dd3200 p_paddr = 0x2ff7dd3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e03200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e03200 p_paddr = 0x2ff7e03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e33200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e33200 p_paddr = 0x2ff7e33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e63200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e63200 p_paddr = 0x2ff7e63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e93200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e93200 p_paddr = 0x2ff7e93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7ec3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7ec3200 p_paddr = 0x2ff7ec3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7ef3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7ef3200 p_paddr = 0x2ff7ef3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7f23200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7f23200 p_paddr = 0x2ff7f23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7f53200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7f53200 p_paddr = 0x2ff7f53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7f83200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7f83200 p_paddr = 0x2ff7f83200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7fb3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7fb3200 p_paddr = 0x2ff7fb3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7fe3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7fe3200 p_paddr = 0x2ff7fe3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d03200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d03200 p_paddr = 0x9ff7d03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d33200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d33200 p_paddr = 0x9ff7d33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d63200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d63200 p_paddr = 0x9ff7d63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d93200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d93200 p_paddr = 0x9ff7d93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7dc3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7dc3200 p_paddr = 0x9ff7dc3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7df3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7df3200 p_paddr = 0x9ff7df3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7e23200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7e23200 p_paddr = 0x9ff7e23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7e53200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7e53200 p_paddr = 0x9ff7e53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7e83200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7e83200 p_paddr = 0x9ff7e83200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7eb3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7eb3200 p_paddr = 0x9ff7eb3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7ee3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7ee3200 p_paddr = 0x9ff7ee3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7f13200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7f13200 p_paddr = 0x9ff7f13200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7f43200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7f43200 p_paddr = 0x9ff7f43200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7f73200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7f73200 p_paddr = 0x9ff7f73200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7fa3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7fa3200 p_paddr = 0x9ff7fa3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7fd3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7fd3200 p_paddr = 0x9ff7fd3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7883200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7883200 p_paddr = 0xaff7883200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff78b3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff78b3200 p_paddr = 0xaff78b3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff78e3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff78e3200 p_paddr = 0xaff78e3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7913200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7913200 p_paddr = 0xaff7913200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7943200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7943200 p_paddr = 0xaff7943200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7973200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7973200 p_paddr = 0xaff7973200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff79a3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff79a3200 p_paddr = 0xaff79a3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff79d3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff79d3200 p_paddr = 0xaff79d3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a03200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a03200 p_paddr = 0xaff7a03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a33200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a33200 p_paddr = 0xaff7a33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a63200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a63200 p_paddr = 0xaff7a63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a93200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a93200 p_paddr = 0xaff7a93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7ac3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7ac3200 p_paddr = 0xaff7ac3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7af3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7af3200 p_paddr = 0xaff7af3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7b23200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7b23200 p_paddr = 0xaff7b23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7b53200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7b53200 p_paddr = 0xaff7b53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
vmcoreinfo header: p_type = 4, p_offset = 0x9fc4720000 p_paddr =
0x9fc4720000 p_vaddr = 0x0 p_filesz = 0x10024 p_memsz = 0x10024
Kernel text Elf header: p_type = 1, p_offset = 0x80000 p_paddr =
0x80000 p_vaddr = 0xffff000008080000 p_filesz = 0x15f0000 p_memsz =
0x15f0000
Elf header: p_type = 1, p_offset = 0x0 p_paddr = 0x0 p_vaddr =
0xffff800000000000 p_filesz = 0xe800000 p_memsz = 0xe800000
Elf header: p_type = 1, p_offset = 0x2e800000 p_paddr = 0x2e800000
p_vaddr = 0xffff80002e800000 p_filesz = 0xae20000 p_memsz = 0xae20000
Elf header: p_type = 1, p_offset = 0x39d40000 p_paddr = 0x39d40000
p_vaddr = 0xffff800039d40000 p_filesz = 0x4ff0000 p_memsz = 0x4ff0000
Elf header: p_type = 1, p_offset = 0x3ed60000 p_paddr = 0x3ed60000
p_vaddr = 0xffff80003ed60000 p_filesz = 0xea0000 p_memsz = 0xea0000
Elf header: p_type = 1, p_offset = 0x1040000000 p_paddr = 0x1040000000
p_vaddr = 0xffff801040000000 p_filesz = 0xfbc000000 p_memsz =
0xfbc000000
Elf header: p_type = 1, p_offset = 0x2000000000 p_paddr = 0x2000000000
p_vaddr = 0xffff802000000000 p_filesz = 0xffc000000 p_memsz =
0xffc000000
Elf header: p_type = 1, p_offset = 0x9000000000 p_paddr = 0x9000000000
p_vaddr = 0xffff809000000000 p_filesz = 0xffc000000 p_memsz =
0xffc000000
Elf header: p_type = 1, p_offset = 0xa000000000 p_paddr = 0xa000000000
p_vaddr = 0xffff80a000000000 p_filesz = 0xffc000000 p_memsz =
0xffc000000
get_crash_notes_per_cpu: crash_notes addr = 1ff7cf3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7cf3200 p_paddr = 0x1ff7cf3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7d23200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7d23200 p_paddr = 0x1ff7d23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7d53200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7d53200 p_paddr = 0x1ff7d53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7d83200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7d83200 p_paddr = 0x1ff7d83200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7db3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7db3200 p_paddr = 0x1ff7db3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7de3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7de3200 p_paddr = 0x1ff7de3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7e13200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7e13200 p_paddr = 0x1ff7e13200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7e43200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7e43200 p_paddr = 0x1ff7e43200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7e73200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7e73200 p_paddr = 0x1ff7e73200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7ea3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7ea3200 p_paddr = 0x1ff7ea3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7ed3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7ed3200 p_paddr = 0x1ff7ed3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f03200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f03200 p_paddr = 0x1ff7f03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f33200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f33200 p_paddr = 0x1ff7f33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f63200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f63200 p_paddr = 0x1ff7f63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7f93200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7f93200 p_paddr = 0x1ff7f93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 1ff7fc3200, size = 424
Elf header: p_type = 4, p_offset = 0x1ff7fc3200 p_paddr = 0x1ff7fc3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7d13200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7d13200 p_paddr = 0x2ff7d13200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7d43200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7d43200 p_paddr = 0x2ff7d43200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7d73200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7d73200 p_paddr = 0x2ff7d73200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7da3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7da3200 p_paddr = 0x2ff7da3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7dd3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7dd3200 p_paddr = 0x2ff7dd3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e03200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e03200 p_paddr = 0x2ff7e03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e33200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e33200 p_paddr = 0x2ff7e33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e63200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e63200 p_paddr = 0x2ff7e63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7e93200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7e93200 p_paddr = 0x2ff7e93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7ec3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7ec3200 p_paddr = 0x2ff7ec3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7ef3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7ef3200 p_paddr = 0x2ff7ef3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7f23200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7f23200 p_paddr = 0x2ff7f23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7f53200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7f53200 p_paddr = 0x2ff7f53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7f83200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7f83200 p_paddr = 0x2ff7f83200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7fb3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7fb3200 p_paddr = 0x2ff7fb3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 2ff7fe3200, size = 424
Elf header: p_type = 4, p_offset = 0x2ff7fe3200 p_paddr = 0x2ff7fe3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d03200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d03200 p_paddr = 0x9ff7d03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d33200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d33200 p_paddr = 0x9ff7d33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d63200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d63200 p_paddr = 0x9ff7d63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7d93200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7d93200 p_paddr = 0x9ff7d93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7dc3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7dc3200 p_paddr = 0x9ff7dc3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7df3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7df3200 p_paddr = 0x9ff7df3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7e23200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7e23200 p_paddr = 0x9ff7e23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7e53200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7e53200 p_paddr = 0x9ff7e53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7e83200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7e83200 p_paddr = 0x9ff7e83200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7eb3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7eb3200 p_paddr = 0x9ff7eb3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7ee3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7ee3200 p_paddr = 0x9ff7ee3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7f13200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7f13200 p_paddr = 0x9ff7f13200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7f43200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7f43200 p_paddr = 0x9ff7f43200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7f73200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7f73200 p_paddr = 0x9ff7f73200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7fa3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7fa3200 p_paddr = 0x9ff7fa3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = 9ff7fd3200, size = 424
Elf header: p_type = 4, p_offset = 0x9ff7fd3200 p_paddr = 0x9ff7fd3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7883200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7883200 p_paddr = 0xaff7883200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff78b3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff78b3200 p_paddr = 0xaff78b3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff78e3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff78e3200 p_paddr = 0xaff78e3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7913200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7913200 p_paddr = 0xaff7913200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7943200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7943200 p_paddr = 0xaff7943200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7973200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7973200 p_paddr = 0xaff7973200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff79a3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff79a3200 p_paddr = 0xaff79a3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff79d3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff79d3200 p_paddr = 0xaff79d3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a03200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a03200 p_paddr = 0xaff7a03200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a33200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a33200 p_paddr = 0xaff7a33200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a63200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a63200 p_paddr = 0xaff7a63200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7a93200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7a93200 p_paddr = 0xaff7a93200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7ac3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7ac3200 p_paddr = 0xaff7ac3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7af3200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7af3200 p_paddr = 0xaff7af3200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7b23200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7b23200 p_paddr = 0xaff7b23200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
get_crash_notes_per_cpu: crash_notes addr = aff7b53200, size = 424
Elf header: p_type = 4, p_offset = 0xaff7b53200 p_paddr = 0xaff7b53200
p_vaddr = 0x0 p_filesz = 0x1a8 p_memsz = 0x1a8
vmcoreinfo header: p_type = 4, p_offset = 0x9fc4720000 p_paddr =
0x9fc4720000 p_vaddr = 0x0 p_filesz = 0x10024 p_memsz = 0x10024
Kernel text Elf header: p_type = 1, p_offset = 0x80000 p_paddr =
0x80000 p_vaddr = 0xffff000008080000 p_filesz = 0x15f0000 p_memsz =
0x15f0000
Elf header: p_type = 1, p_offset = 0x396c0000 p_paddr = 0x396c0000
p_vaddr = 0xffff8000396c0000 p_filesz = 0xa0000 p_memsz = 0xa0000
Elf header: p_type = 1, p_offset = 0x39770000 p_paddr = 0x39770000
p_vaddr = 0xffff800039770000 p_filesz = 0x40000 p_memsz = 0x40000
Elf header: p_type = 1, p_offset = 0x398a0000 p_paddr = 0x398a0000
p_vaddr = 0xffff8000398a0000 p_filesz = 0x20000 p_memsz = 0x20000
load_crashdump_segments: elfcorehdr 0x2e7f0000-0x2e7f0fff
read_1st_dtb: found /sys/firmware/fdt
get_cells_size: #address-cells:2 #size-cells:2
cells_size_fitted: 2e7f0000-2e7f0fff
cells_size_fitted: e800000-2e7fffff
cells_size_fitted: 396c0000-3975ffff
cells_size_fitted: 39770000-397affff
cells_size_fitted: 398a0000-398bffff
 / {
    #size-cells = <0x00000002>;
    #address-cells = <0x00000002>;
    chosen {
        linux,usable-memory-range = <0x00000000 0x0e800000 0x00000000
0x20000000 0x00000000 0x396c0000 0x00000000 0x000a0000 0x00000000
0x39770000 0x00000000 0x00040000 0x00000000 0x398a0000 0x00000000
0x00020000>;
        linux,elfcorehdr = <0x00000000 0x2e7f0000 0x00000000 0x00001000>;
        linux,uefi-mmap-desc-ver = <0x00000001>;
        linux,uefi-mmap-desc-size = <0x00000030>;
        linux,uefi-mmap-size = <0x00000e40>;
        linux,uefi-mmap-start = <0x00000000 0x30288018>;
        linux,uefi-system-table = <0x00000000 0x3ed50018>;
        bootargs = "root=/dev/mapper/rhelaa_huawei--t2280--01-root ro
earlycon=pl011,mmio,0x602B0000 efi=debug memblock_debug=1
pcie_aspm=off rd.lvm.lv=rhelaa_huawei-t2280-01/root
rd.lvm.lv=rhelaa_huawei-t2280-01/swap acpi=force
console=ttyAMA0,115200";
        linux,initrd-end = <0x00000000 0x2fbff9e0>;
        linux,initrd-start = <0x00000000 0x2e84d000>;
    };
 };
initrd: base fe70000, size 13b29e0h (20654560), end 112229e0

[snip..]

sym: sha256_starts info: 12 other: 00 shndx: 1 value: eb0 size: 6c
sym: sha256_starts value: 11240eb0 addr: 11240018
machine_apply_elf_rel: CALL26 580006b394000000->580006b3940003a6
sym: sha256_update info: 12 other: 00 shndx: 1 value: 5158 size: c
sym: sha256_update value: 11245158 addr: 11240034
machine_apply_elf_rel: CALL26 9100427394000000->9100427394001449
sym: sha256_finish info: 12 other: 00 shndx: 1 value: 5164 size: 1cc
sym: sha256_finish value: 11245164 addr: 11240050
machine_apply_elf_rel: CALL26 aa1403e094000000->aa1403e094001445
sym:     memcmp info: 12 other: 00 shndx: 1 value: 634 size: 34
sym: memcmp value: 11240634 addr: 11240060
machine_apply_elf_rel: CALL26 340003c094000000->340003c094000175
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 11240070
machine_apply_elf_rel: CALL26 5800046094000000->5800046094000135
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 11240078
machine_apply_elf_rel: CALL26 5800047594000000->5800047594000133
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 11240088
machine_apply_elf_rel: CALL26 9100067394000000->910006739400012f
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 112400a8
machine_apply_elf_rel: CALL26 5800036094000000->5800036094000127
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 112400b0
machine_apply_elf_rel: CALL26 910402e194000000->910402e194000125
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 112400c0
machine_apply_elf_rel: CALL26 9100067394000000->9100067394000121
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 112400d4
machine_apply_elf_rel: CALL26 5280002094000000->528000209400011c
sym:      .data info: 03 other: 00 shndx: 4 value: 0 size: 0
sym: .data value: 112453a8 addr: 112400f0
machine_apply_elf_rel: ABS64 0000000000000000->00000000112453a8
sym: .rodata.str1.1 info: 03 other: 00 shndx: 3 value: 0 size: 0
sym: .rodata.str1.1 value: 11245338 addr: 112400f8
machine_apply_elf_rel: ABS64 0000000000000000->0000000011245338
sym: .rodata.str1.1 info: 03 other: 00 shndx: 3 value: 0 size: 0
sym: .rodata.str1.1 value: 11245358 addr: 11240100
machine_apply_elf_rel: ABS64 0000000000000000->0000000011245358
sym: .rodata.str1.1 info: 03 other: 00 shndx: 3 value: 0 size: 0
sym: .rodata.str1.1 value: 11245368 addr: 11240108
machine_apply_elf_rel: ABS64 0000000000000000->0000000011245368
sym: .rodata.str1.1 info: 03 other: 00 shndx: 3 value: 0 size: 0
sym: .rodata.str1.1 value: 1124536e addr: 11240110
machine_apply_elf_rel: ABS64 0000000000000000->000000001124536e
sym: .rodata.str1.1 info: 03 other: 00 shndx: 3 value: 0 size: 0
sym: .rodata.str1.1 value: 11245370 addr: 11240118
machine_apply_elf_rel: ABS64 0000000000000000->0000000011245370
sym:     printf info: 12 other: 00 shndx: 1 value: 544 size: 90
sym: printf value: 11240544 addr: 1124012c
machine_apply_elf_rel: CALL26 9400000094000000->9400000094000106
sym: setup_arch info: 12 other: 00 shndx: 1 value: ea8 size: 4
sym: setup_arch value: 11240ea8 addr: 11240130
machine_apply_elf_rel: CALL26 9400000094000000->940000009400035e
sym: verify_sha256_digest info: 12 other: 00 shndx: 1 value: 0 size: f0
sym: verify_sha256_digest value: 11240000 addr: 11240134
machine_apply_elf_rel: CALL26 3400004094000000->3400004097ffffb3
sym: post_verification_setup_arch info: 12 other: 00 shndx: 1 value: ea4 size: 4
sym: post_verification_setup_arch value: 11240ea4 addr: 11240144
machine_apply_elf_rel: JUMP26 0000000014000000->0000000014000358
sym: .rodata.str1.1 info: 03 other: 00 shndx: 3 value: 0 size: 0
sym: .rodata.str1.1 value: 11245380 addr: 11240148
machine_apply_elf_rel: ABS64 0000000000000000->0000000011245380
sym:    putchar info: 12 other: 00 shndx: 1 value: ea0 size: 4
sym: putchar value: 11240ea0 addr: 112401ac
machine_apply_elf_rel: CALL26 f94037a194000000->f94037a19400033d
sym:    putchar info: 12 other: 00 shndx: 1 value: ea0 size: 4
sym: putchar value: 11240ea0 addr: 11240220
machine_apply_elf_rel: CALL26 910006f794000000->910006f794000320
sym:    putchar info: 12 other: 00 shndx: 1 value: ea0 size: 4
sym: putchar value: 11240ea0 addr: 11240478
machine_apply_elf_rel: CALL26 9100073994000000->910007399400028a
sym: .rodata.str1.1 info: 03 other: 00 shndx: 3 value: 0 size: 0
sym: .rodata.str1.1 value: 11245392 addr: 112404b8
machine_apply_elf_rel: ABS64 0000000000000000->0000000011245392
sym:   vsprintf info: 12 other: 00 shndx: 1 value: 150 size: 364
sym: vsprintf value: 11240150 addr: 11240538
machine_apply_elf_rel: CALL26 a8d07bfd94000000->a8d07bfd97ffff06
sym:   vsprintf info: 12 other: 00 shndx: 1 value: 150 size: 364
sym: vsprintf value: 11240150 addr: 112405c8
machine_apply_elf_rel: CALL26 a8d17bfd94000000->a8d17bfd97fffee2
sym:  purgatory info: 12 other: 00 shndx: 1 value: 120 size: 28
sym: purgatory value: 11240120 addr: 11240678
machine_apply_elf_rel: CALL26 5800001194000000->5800001197fffeaa
sym: arm64_kernel_entry info: 10 other: 00 shndx: 4 value: 120 size: 8
sym: arm64_kernel_entry value: 112454c8 addr: 1124067c
machine_apply_elf_rel: LD_PREL_LO19 5800000058000011->5800000058027271
sym: arm64_dtb_addr info: 10 other: 00 shndx: 4 value: 128 size: 8
sym: arm64_dtb_addr value: 112454d0 addr: 11240680
machine_apply_elf_rel: LD_PREL_LO19 aa1f03e158000000->aa1f03e158027280
sym: sha256_process info: 12 other: 00 shndx: 1 value: f1c size: 4134
sym: sha256_process value: 11240f1c addr: 112450bc
machine_apply_elf_rel: CALL26 d101029494000000->d101029497ffef98
sym:     memcpy info: 12 other: 00 shndx: 1 value: 614 size: 20
sym: memcpy value: 11240614 addr: 11245118
machine_apply_elf_rel: JUMP26 b4fffc5814000000->b4fffc5817ffed3f
sym:     memcpy info: 12 other: 00 shndx: 1 value: 614 size: 20
sym: memcpy value: 11240614 addr: 11245130
machine_apply_elf_rel: CALL26 aa1503e094000000->aa1503e097ffed39
sym: sha256_process info: 12 other: 00 shndx: 1 value: f1c size: 4134
sym: sha256_process value: 11240f1c addr: 1124513c
machine_apply_elf_rel: CALL26 cb1302d694000000->cb1302d697ffef78
sym:      .data info: 03 other: 00 shndx: 4 value: 0 size: 0
sym: .data value: 112454d8 addr: 11245330
machine_apply_elf_rel: ABS64 0000000000000000->00000000112454d8
kexec_load: entry = 0x11240670 flags = 0xb70001
nr_segments = 5
segment[0].buf   = 0xffff968d0010
segment[0].bufsz = 0xdf9200
segment[0].mem   = 0xe880000
segment[0].memsz = 0x15f0000
segment[1].buf   = 0xffff950e0010
segment[1].bufsz = 0x13b29e0
segment[1].mem   = 0xfe70000
segment[1].memsz = 0x13c0000
segment[2].buf   = 0x1115b440
segment[2].bufsz = 0x33d
segment[2].mem   = 0x11230000
segment[2].memsz = 0x10000
segment[3].buf   = 0x1115bb70
segment[3].bufsz = 0x5518
segment[3].mem   = 0x11240000
segment[3].memsz = 0x10000
segment[4].buf   = 0x11159ca0
segment[4].bufsz = 0x1000
segment[4].mem   = 0x2e7f0000
segment[4].memsz = 0x10000

Regards,
Bhupesh

>
>
>> Regards,
>> Bhupesh
>>
>>
>> >> >
>> >> > Regards,
>> >> > Bhupesh
>> >> >
>> >> > >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
>> >> > >> via a kernel command line parameter, "memmap=".
>> >> > >>
>> >> > _______________________________________________
>> >> > kexec mailing list -- kexec at lists.fedoraproject.org
>> >> > To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From bhsharma at redhat.com  Wed Dec 20 11:52:45 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Thu, 21 Dec 2017 01:22:45 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171219050113.GF28046@linaro.org>
References: <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <CAKv+Gu-W5VpVrgA=FVZCCevksaRGOVvPdE+B8WkpZc6AE1jOPw@mail.gmail.com>
 <CACi5LpPDL0j4o3MuOZ3jRv3UOqM48yTgKtdm-hY+OWL8h-ETzQ@mail.gmail.com>
 <20171218051657.GA5998@dhcp-128-65.nay.redhat.com>
 <20171218055356.mq4coiibh3frii54@fireball>
 <CAFTCetQ55zUKe25jSku0DHp8uVZA4hB32d5W6MSCNsTVpxu7Gw@mail.gmail.com>
 <20171219050113.GF28046@linaro.org>
Message-ID: <CACi5LpMmuzT-f-zX+mzrJ9J3io9JbYjQ_qQqyeVGBe1ftfN8Ww@mail.gmail.com>

On Tue, Dec 19, 2017 at 10:31 AM, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
> On Mon, Dec 18, 2017 at 02:29:05PM +0530, Bhupesh SHARMA wrote:
>>
>> [snip..]
>>
>> [    0.000000] linux,usable-memory-range base e800000, size 20000000
>> [    0.000000]  - e800000 ,  20000000
>> [    0.000000] linux,usable-memory-range base 396c0000, size a0000
>> [    0.000000]  - 396c0000 ,  a0000
>> [    0.000000] linux,usable-memory-range base 39770000, size 40000
>> [    0.000000]  - 39770000 ,  40000
>> [    0.000000] linux,usable-memory-range base 398a0000, size 20000
>> [    0.000000]  - 398a0000 ,  20000
>> [    0.000000] initrd not fully accessible via the linear mapping --
>> please check your bootloader ...
>
> This is an odd message coming from:
> |void __init arm64_memblock_init(void)
> |...
> |
> |                if (WARN(base < memblock_start_of_DRAM() ||
> |                         base + size > memblock_start_of_DRAM() +
> |                                       linear_region_size,
> |                        "initrd not fully accessible via the linear mapping -- please check your bootloader ...\n")) {
>
> Can you confirm how the condition breaks here?
> I suppose
>     base: 0xfe70000
>     size: 0x13c0000
>     memblock_start_of_DRAM(): 0xe800000
> according to the information you gave me.

Indeed, the first check 'base < memblock_start_of_DRAM()' in the
following check fails:

        if (WARN(base < memblock_start_of_DRAM() ||
             base + size > memblock_start_of_DRAM() +
                       linear_region_size,

Here are the values I am seeing on this board using the kernel and
kexec-tools which have been modified to append the
'linux,usable-memory-range' with the acpi reclaim regions:

base=fe70000,
size=13c0000,
memblock_start_of_DRAM=39620000
linear_region_size=800000000000

I suspect that the holes introduced by kexec-tools inside
'arm64_load_other_segments()' in 'kexec/arch/arm64/kexec-arm64.c' (see
the code leg below):

    /* Put the other segments after the image. */

    hole_min = image_base + arm64_mem.image_size;
    if (info->kexec_flags & KEXEC_ON_CRASH)
        hole_max = crash_reserved_mem.end;
    else
        hole_max = ULONG_MAX;


should be updated to introduce appropriate handling of the acpi reclaim regions.
I am not aware of the background of this handling in the kexec-tools.
Do you think this can be at fault, Akashi?

Regards,
Bhupesh



>
>> [    0.000000] ------------[ cut here ]------------
>> [    0.000000] WARNING: CPU: 0 PID: 0 at arch/arm64/mm/init.c:597
>> arm64_memblock_init+0x210/0x484
>> [    0.000000] Modules linked in:
>> [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0+ #7
>> [    0.000000] task: ffff000008d05580 task.stack: ffff000008cc0000
>> [    0.000000] PC is at arm64_memblock_init+0x210/0x484
>> [    0.000000] LR is at arm64_memblock_init+0x210/0x484
>> [    0.000000] pc : [<ffff000008b76984>] lr : [<ffff000008b76984>]
>> pstate: 600000c5
>> [    0.000000] sp : ffff000008ccfe80
>> [    0.000000] x29: ffff000008ccfe80 x28: 000000000f370018
>> [    0.000000] x27: 0000000011230000 x26: 00000000013b0000
>> [    0.000000] x25: 000000000fe80000 x24: ffff000008cf3000
>> [    0.000000] x23: ffff000008ec0000 x22: ffff000009680000
>> [    0.000000] x21: ffff000008afa000 x20: ffff000008080000
>> [    0.000000] x19: ffff000008afa000 x18: 000000000c283806
>> [    0.000000] x17: 0000000000000000 x16: ffff000008d05580
>> [    0.000000] x15: 000000002be00842 x14: 79206b6365686320
>> [    0.000000] x13: 657361656c70202d x12: 2d20676e69707061
>> [    0.000000] x11: 6d207261656e696c x10: 2065687420616976
>> [    0.000000] x9 : 00000000000000f4 x8 : ffff000008517414
>> [    0.000000] x7 : 746f6f622072756f x6 : 000000000000000d
>> [    0.000000] x5 : ffff000008c96360 x4 : 0000000000000001
>> [    0.000000] x3 : 0000000000000000 x2 : 0000000000000000
>> [    0.000000] x1 : 0000000000000000 x0 : 0000000000000056
>> [    0.000000] Call trace:
>> [    0.000000] Exception stack(0xffff000008ccfd40 to 0xffff000008ccfe80)
>> [    0.000000] fd40: 0000000000000056 0000000000000000
>> 0000000000000000 0000000000000000
>> [    0.000000] fd60: 0000000000000001 ffff000008c96360
>> 000000000000000d 746f6f622072756f
>> [    0.000000] fd80: ffff000008517414 00000000000000f4
>> 2065687420616976 6d207261656e696c
>> [    0.000000] fda0: 2d20676e69707061 657361656c70202d
>> 79206b6365686320 000000002be00842
>> [    0.000000] fdc0: ffff000008d05580 0000000000000000
>> 000000000c283806 ffff000008afa000
>> [    0.000000] fde0: ffff000008080000 ffff000008afa000
>> ffff000009680000 ffff000008ec0000
>> [    0.000000] fe00: ffff000008cf3000 000000000fe80000
>> 00000000013b0000 0000000011230000
>> [    0.000000] fe20: 000000000f370018 ffff000008ccfe80
>> ffff000008b76984 ffff000008ccfe80
>> [    0.000000] fe40: ffff000008b76984 00000000600000c5
>> ffff00000959b7a8 ffff000008ec0000
>> [    0.000000] fe60: ffffffffffffffff 0000000000000005
>> ffff000008ccfe80 ffff000008b76984
>> [    0.000000] [<ffff000008b76984>] arm64_memblock_init+0x210/0x484
>> [    0.000000] [<ffff000008b7398c>] setup_arch+0x1b8/0x5f4
>> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
>> [    0.000000] random: get_random_bytes called from
>> print_oops_end_marker+0x50/0x6c with crng_init=0
>> [    0.000000] ---[ end trace 0000000000000000 ]---
>> [    0.000000] Reserving 4KB of memory at 0x2e7f0000 for elfcorehdr
>> [    0.000000] cma: Failed to reserve 512 MiB
>> [    0.000000] Kernel panic - not syncing: ERROR: Failed to allocate
>> 0x0000000000010000 bytes below 0x0000000000000000.
>> [    0.000000]
>> [    0.000000] CPU: 0 PID: 0 Comm: swapper Tainted: G        W
>> ------------   4.14.0+ #7
>> [    0.000000] Call trace:
>> [    0.000000] [<ffff000008088da8>] dump_backtrace+0x0/0x23c
>> [    0.000000] [<ffff000008089008>] show_stack+0x24/0x2c
>> [    0.000000] [<ffff0000087f647c>] dump_stack+0x84/0xa8
>> [    0.000000] [<ffff0000080cfd44>] panic+0x138/0x2a0
>> [    0.000000] [<ffff000008b95c88>] memblock_alloc_base+0x44/0x4c
>> [    0.000000] [<ffff000008b95cbc>] memblock_alloc+0x2c/0x38
>> [    0.000000] [<ffff000008b772dc>] early_pgtable_alloc+0x20/0x74
>> [    0.000000] [<ffff000008b7755c>] paging_init+0x28/0x544
>> [    0.000000] [<ffff000008b73990>] setup_arch+0x1bc/0x5f4
>> [    0.000000] [<ffff000008b70a10>] start_kernel+0x74/0x43c
>> [    0.000000] ---[ end Kernel panic - not syncing: ERROR: Failed to
>> allocate 0x0000000000010000 bytes below 0x0000000000000000.
>> [    0.000000]
>>
>> I guess it is because of the 1G alignment requirement between the
>> kernel image and the initrd and how we populate the holes between the
>> kernel image, segments (including dtb) and the initrd from the
>> kexec-tools.
>>
>> Akashi, any pointers on this will be helpful as well.
>>
>> Regards,
>> Bhupesh
>>
>>
>> >> >
>> >> > Regards,
>> >> > Bhupesh
>> >> >
>> >> > >> Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
>> >> > >> via a kernel command line parameter, "memmap=".
>> >> > >>
>> >> > _______________________________________________
>> >> > kexec mailing list -- kexec at lists.fedoraproject.org
>> >> > To unsubscribe send an email to kexec-leave at lists.fedoraproject.org


From takahiro.akashi at linaro.org  Thu Dec 21 02:34:42 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Thu, 21 Dec 2017 19:34:42 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
References: <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
 <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
Message-ID: <20171221103440.GJ28046@linaro.org>

Bhupesh,

Can you test the patch attached below, please?

It is intended to retain already-reserved regions (ACPI reclaim memory
in this case) in system ram (i.e. memblock.memory) without explicitly
exporting them via usable-memory-range.
(I still have to figure out what the side-effect of this patch is.)

Thanks,
-Takahiro AKASHI

On Thu, Dec 21, 2017 at 01:30:43AM +0530, Bhupesh Sharma wrote:
> On Tue, Dec 19, 2017 at 6:39 PM, Ard Biesheuvel
> <ard.biesheuvel at linaro.org> wrote:
> > On 19 December 2017 at 07:09, AKASHI Takahiro
> > <takahiro.akashi at linaro.org> wrote:
> >> On Mon, Dec 18, 2017 at 01:40:09PM +0800, Dave Young wrote:
> >>> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
> >>> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> >>> > > On 13 December 2017 at 12:16, AKASHI Takahiro
> >>> > > <takahiro.akashi at linaro.org> wrote:
> >>> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> >>> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
> >>> > > >> <takahiro.akashi at linaro.org> wrote:
> >>> > > >> > Bhupesh, Ard,
> >>> > > >> >
> >>> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> >>> > > >> >> Hi Ard, Akashi
> >>> > > >> >>
> >>> > > >> > (snip)
> >>> > > >> >
> >>> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> >>> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> >>> > > >> >> identify its own usable memory and exclude, at its boot time, any
> >>> > > >> >> other memory areas that are part of the panicked kernel's memory.
> >>> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> >>> > > >> >> , for details)
> >>> > > >> >
> >>> > > >> > Right.
> >>> > > >> >
> >>> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> >>> > > >> >> with the crashkernel memory range:
> >>> > > >> >>
> >>> > > >> >>                 /* add linux,usable-memory-range */
> >>> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> >>> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> >>> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> >>> > > >> >>                                 address_cells, size_cells);
> >>> > > >> >>
> >>> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> >>> > > >> >> , for details)
> >>> > > >> >>
> >>> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> >>> > > >> >> they are marked as System RAM or as RESERVED. As,
> >>> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
> >>> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> >>> > > >> >>
> >>> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
> >>> > > >> >> ACPI memory and crashes while trying to access the same:
> >>> > > >> >>
> >>> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> >>> > > >> >> -r`.img --reuse-cmdline -d
> >>> > > >> >>
> >>> > > >> >> [snip..]
> >>> > > >> >>
> >>> > > >> >> Reserved memory range
> >>> > > >> >> 000000000e800000-000000002e7fffff (0)
> >>> > > >> >>
> >>> > > >> >> Coredump memory ranges
> >>> > > >> >> 0000000000000000-000000000e7fffff (0)
> >>> > > >> >> 000000002e800000-000000003961ffff (0)
> >>> > > >> >> 0000000039d40000-000000003ed2ffff (0)
> >>> > > >> >> 000000003ed60000-000000003fbfffff (0)
> >>> > > >> >> 0000001040000000-0000001ffbffffff (0)
> >>> > > >> >> 0000002000000000-0000002ffbffffff (0)
> >>> > > >> >> 0000009000000000-0000009ffbffffff (0)
> >>> > > >> >> 000000a000000000-000000affbffffff (0)
> >>> > > >> >>
> >>> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> >>> > > >> >> memory cap'ing passed to the crash kernel inside
> >>> > > >> >> 'arch/arm64/mm/init.c' (see below):
> >>> > > >> >>
> >>> > > >> >> static void __init fdt_enforce_memory_region(void)
> >>> > > >> >> {
> >>> > > >> >>         struct memblock_region reg = {
> >>> > > >> >>                 .size = 0,
> >>> > > >> >>         };
> >>> > > >> >>
> >>> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >>> > > >> >>
> >>> > > >> >>         if (reg.size)
> >>> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> >>> > > >> >> comment this out */
> >>> > > >> >> }
> >>> > > >> >
> >>> > > >> > Please just don't do that. It can cause a fatal damage on
> >>> > > >> > memory contents of the *crashed* kernel.
> >>> > > >> >
> >>> > > >> >> 5). Both the above temporary solutions fix the problem.
> >>> > > >> >>
> >>> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
> >>> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> >>> > > >> >> fail.
> >>> > > >> >>
> >>> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> >>> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> >>> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> >>> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> >>> > > >> >> dt node 'linux,usable-memory-range'
> >>> > > >> >
> >>> > > >> > I still don't understand why we need to carry over the information
> >>> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> >>> > > >> > such regions are free to be reused by the kernel after some point of
> >>> > > >> > initialization. Why does crash dump kernel need to know about them?
> >>> > > >> >
> >>> > > >>
> >>> > > >> Not really. According to the UEFI spec, they can be reclaimed after
> >>> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> >>> > > >> no longer needs them. Of course, in order to be able to boot a kexec
> >>> > > >> kernel, those regions needs to be preserved, which is why they are
> >>> > > >> memblock_reserve()'d now.
> >>> > > >
> >>> > > > For my better understandings, who is actually accessing such regions
> >>> > > > during boot time, uefi itself or efistub?
> >>> > > >
> >>> > >
> >>> > > No, only the kernel. This is where the ACPI tables are stored. For
> >>> > > instance, on QEMU we have
> >>> > >
> >>> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> >>> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> >>> > >   01000013)
> >>> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> >>> > > BXPC 00000001)
> >>> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> >>> > > BXPC 00000001)
> >>> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> >>> > > BXPC 00000001)
> >>> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> >>> > > BXPC 00000001)
> >>> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> >>> > > BXPC 00000001)
> >>> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> >>> > > BXPC 00000001)
> >>> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> >>> > > BXPC 00000001)
> >>> > >
> >>> > > covered by
> >>> > >
> >>> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> >>> > >  ...
> >>> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> >>> >
> >>> > OK. I mistakenly understood those regions could be freed after exiting
> >>> > UEFI boot services.
> >>> >
> >>> > >
> >>> > > >> So it seems that kexec does not honour the memblock_reserve() table
> >>> > > >> when booting the next kernel.
> >>> > > >
> >>> > > > not really.
> >>> > > >
> >>> > > >> > (In other words, can or should we skip some part of ACPI-related init code
> >>> > > >> > on crash dump kernel?)
> >>> > > >> >
> >>> > > >>
> >>> > > >> I don't think so. And the change to the handling of ACPI reclaim
> >>> > > >> regions only revealed the bug, not created it (given that other
> >>> > > >> memblock_reserve regions may be affected as well)
> >>> > > >
> >>> > > > As whether we should honor such reserved regions over kexec'ing
> >>> > > > depends on each one's specific nature, we will have to take care one-by-one.
> >>> > > > As a matter of fact, no information about "reserved" memblocks is
> >>> > > > exposed to user space (via proc/iomem).
> >>> > > >
> >>> > >
> >>> > > That is why I suggested (somewhere in this thread?) to not expose them
> >>> > > as 'System RAM'. Do you think that could solve this?
> >>> >
> >>> > Memblock-reserv'ing them is necessary to prevent their corruption and
> >>> > marking them under another name in /proc/iomem would also be good in order
> >>> > not to allocate them as part of crash kernel's memory.
> >>> >
> >>> > But I'm not still convinced that we should export them in useable-
> >>> > memory-range to crash dump kernel. They will be accessed through
> >>> > acpi_os_map_memory() and so won't be required to be part of system ram
> >>> > (or memblocks), I guess.
> >>> >     -> Bhupesh?
> >>>
> >>> I forgot how arm64 kernel retrieve the memory ranges and initialize
> >>> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
> >>> the memory according to the efi memmap?  For kdump kernel anything other
> >>> than usable memory (which is from the dt node instead) should be
> >>> reinitialized according to efi passed info, no?
> >>
> >> All the regions exported in efi memmap will be added to memblock.memory
> >> in (u)efi_init() and then trimmed down to the exact range specified as
> >> usable-memory-range by fdt_enforce_memory_region().
> >>
> >> Now I noticed that the current fdt_enforce_memory_region() may not work well
> >> with multiple entries in usable-memory-range.
> >>
> >
> > In any case, the root of the problem is that memory regions lose their
> > 'memory' annotation due to the way the memory map is mangled before
> > being supplied to the kexec kernel.
> >
> > Would it be possible to classify all memory that we want to hide from
> > the kexec kernel as NOMAP instead? That way, it will not be mapped
> > implicitly, but will still be mapped cacheable by acpi_os_ioremap(),
> > so this seems to be the most appropriate way to deal with the host
> > kernel's memory contents.
> 
> Hmm. wouldn't appending the acpi reclaim regions to
> 'linux,usable-memory-range' in the dtb being passed to the crashkernel
> be better? Because its indirectly achieving a similar objective
> (although may be a subset of all System RAM regions on the primary
> kernel's memory).
> 
> I am not aware of the background about the current kexec-tools
> implementation where we add only the crashkernel range to the dtb
> being passed to the crashkernel.
> 
> Probably Akashi can answer better, as to how we arrived at this design
> approach and why we didn't want to expose all System RAM regions (i.e.
> ! NOMPAP regions) to the crashkernel.
> 
> I am suspecting that some issues were seen/meet when the System RAM (!
> NOMAP regions) were exposed to the crashkernel, and that's why we
> finalized on this design approach, but this is something which is just
> my guess.
> 
> Regards,
> Bhupesh
> 
> >>> >
> >>> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >>> > via a kernel command line parameter, "memmap=".
> >>>
> >>> memmap= is only used in old kexec-tools, now we are passing them via
> >>> e820 table.
> >>
> >> Thanks. I remember that you have explained it before.
> >>
> >> -Takahiro AKASHI
> >>
> >>> [snip]
> >>>
> >>> Thanks
> >>> Dave

===8<==
>From 74e2451fea83d546feae76160ba7de426913fe03 Mon Sep 17 00:00:00 2001
From: AKASHI Takahiro <takahiro.akashi at linaro.org>
Date: Thu, 21 Dec 2017 19:14:23 +0900
Subject: [PATCH] arm64: kdump: mark unusable memory as NOMAP

---
 arch/arm64/mm/init.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 00e7b900ca41..8175db94257b 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -352,11 +352,17 @@ static void __init fdt_enforce_memory_region(void)
 	struct memblock_region reg = {
 		.size = 0,
 	};
+	u64 idx;
+	phys_addr_t start, end;
 
 	of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
 
-	if (reg.size)
-		memblock_cap_memory_range(reg.base, reg.size);
+	if (reg.size) {
+		for_each_free_mem_range(idx, NUMA_NO_NODE, MEMBLOCK_NONE,
+					&start, &end, NULL)
+			memblock_mark_nomap(start, end - start);
+		memblock_clear_nomap(reg.base, reg.size);
+	}
 }
 
 void __init arm64_memblock_init(void)
-- 
2.15.1



From bhsharma at redhat.com  Thu Dec 21 04:06:30 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Thu, 21 Dec 2017 17:36:30 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171221103440.GJ28046@linaro.org>
References: <CACi5LpOZ=WOx14gTwH5jfLozepT2Jw8JSY5x+bfEZ_YaiQvFpw@mail.gmail.com>
 <20171213102624.GC28046@linaro.org>
 <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
 <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
Message-ID: <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>

Hello Akashi,

On Thu, Dec 21, 2017 at 4:04 PM, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
> Bhupesh,
>
> Can you test the patch attached below, please?
>
> It is intended to retain already-reserved regions (ACPI reclaim memory
> in this case) in system ram (i.e. memblock.memory) without explicitly
> exporting them via usable-memory-range.
> (I still have to figure out what the side-effect of this patch is.)
>
> Thanks,
> -Takahiro AKASHI
>
> On Thu, Dec 21, 2017 at 01:30:43AM +0530, Bhupesh Sharma wrote:
>> On Tue, Dec 19, 2017 at 6:39 PM, Ard Biesheuvel
>> <ard.biesheuvel at linaro.org> wrote:
>> > On 19 December 2017 at 07:09, AKASHI Takahiro
>> > <takahiro.akashi at linaro.org> wrote:
>> >> On Mon, Dec 18, 2017 at 01:40:09PM +0800, Dave Young wrote:
>> >>> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
>> >>> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
>> >>> > > On 13 December 2017 at 12:16, AKASHI Takahiro
>> >>> > > <takahiro.akashi at linaro.org> wrote:
>> >>> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
>> >>> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
>> >>> > > >> <takahiro.akashi at linaro.org> wrote:
>> >>> > > >> > Bhupesh, Ard,
>> >>> > > >> >
>> >>> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
>> >>> > > >> >> Hi Ard, Akashi
>> >>> > > >> >>
>> >>> > > >> > (snip)
>> >>> > > >> >
>> >>> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
>> >>> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
>> >>> > > >> >> identify its own usable memory and exclude, at its boot time, any
>> >>> > > >> >> other memory areas that are part of the panicked kernel's memory.
>> >>> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
>> >>> > > >> >> , for details)
>> >>> > > >> >
>> >>> > > >> > Right.
>> >>> > > >> >
>> >>> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
>> >>> > > >> >> with the crashkernel memory range:
>> >>> > > >> >>
>> >>> > > >> >>                 /* add linux,usable-memory-range */
>> >>> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
>> >>> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
>> >>> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
>> >>> > > >> >>                                 address_cells, size_cells);
>> >>> > > >> >>
>> >>> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
>> >>> > > >> >> , for details)
>> >>> > > >> >>
>> >>> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
>> >>> > > >> >> they are marked as System RAM or as RESERVED. As,
>> >>> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
>> >>> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
>> >>> > > >> >>
>> >>> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
>> >>> > > >> >> ACPI memory and crashes while trying to access the same:
>> >>> > > >> >>
>> >>> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
>> >>> > > >> >> -r`.img --reuse-cmdline -d
>> >>> > > >> >>
>> >>> > > >> >> [snip..]
>> >>> > > >> >>
>> >>> > > >> >> Reserved memory range
>> >>> > > >> >> 000000000e800000-000000002e7fffff (0)
>> >>> > > >> >>
>> >>> > > >> >> Coredump memory ranges
>> >>> > > >> >> 0000000000000000-000000000e7fffff (0)
>> >>> > > >> >> 000000002e800000-000000003961ffff (0)
>> >>> > > >> >> 0000000039d40000-000000003ed2ffff (0)
>> >>> > > >> >> 000000003ed60000-000000003fbfffff (0)
>> >>> > > >> >> 0000001040000000-0000001ffbffffff (0)
>> >>> > > >> >> 0000002000000000-0000002ffbffffff (0)
>> >>> > > >> >> 0000009000000000-0000009ffbffffff (0)
>> >>> > > >> >> 000000a000000000-000000affbffffff (0)
>> >>> > > >> >>
>> >>> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
>> >>> > > >> >> memory cap'ing passed to the crash kernel inside
>> >>> > > >> >> 'arch/arm64/mm/init.c' (see below):
>> >>> > > >> >>
>> >>> > > >> >> static void __init fdt_enforce_memory_region(void)
>> >>> > > >> >> {
>> >>> > > >> >>         struct memblock_region reg = {
>> >>> > > >> >>                 .size = 0,
>> >>> > > >> >>         };
>> >>> > > >> >>
>> >>> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> >>> > > >> >>
>> >>> > > >> >>         if (reg.size)
>> >>> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
>> >>> > > >> >> comment this out */
>> >>> > > >> >> }
>> >>> > > >> >
>> >>> > > >> > Please just don't do that. It can cause a fatal damage on
>> >>> > > >> > memory contents of the *crashed* kernel.
>> >>> > > >> >
>> >>> > > >> >> 5). Both the above temporary solutions fix the problem.
>> >>> > > >> >>
>> >>> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
>> >>> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
>> >>> > > >> >> fail.
>> >>> > > >> >>
>> >>> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
>> >>> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
>> >>> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
>> >>> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
>> >>> > > >> >> dt node 'linux,usable-memory-range'
>> >>> > > >> >
>> >>> > > >> > I still don't understand why we need to carry over the information
>> >>> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
>> >>> > > >> > such regions are free to be reused by the kernel after some point of
>> >>> > > >> > initialization. Why does crash dump kernel need to know about them?
>> >>> > > >> >
>> >>> > > >>
>> >>> > > >> Not really. According to the UEFI spec, they can be reclaimed after
>> >>> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
>> >>> > > >> no longer needs them. Of course, in order to be able to boot a kexec
>> >>> > > >> kernel, those regions needs to be preserved, which is why they are
>> >>> > > >> memblock_reserve()'d now.
>> >>> > > >
>> >>> > > > For my better understandings, who is actually accessing such regions
>> >>> > > > during boot time, uefi itself or efistub?
>> >>> > > >
>> >>> > >
>> >>> > > No, only the kernel. This is where the ACPI tables are stored. For
>> >>> > > instance, on QEMU we have
>> >>> > >
>> >>> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
>> >>> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
>> >>> > >   01000013)
>> >>> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
>> >>> > > BXPC 00000001)
>> >>> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
>> >>> > > BXPC 00000001)
>> >>> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
>> >>> > > BXPC 00000001)
>> >>> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
>> >>> > > BXPC 00000001)
>> >>> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
>> >>> > > BXPC 00000001)
>> >>> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
>> >>> > > BXPC 00000001)
>> >>> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
>> >>> > > BXPC 00000001)
>> >>> > >
>> >>> > > covered by
>> >>> > >
>> >>> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
>> >>> > >  ...
>> >>> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
>> >>> >
>> >>> > OK. I mistakenly understood those regions could be freed after exiting
>> >>> > UEFI boot services.
>> >>> >
>> >>> > >
>> >>> > > >> So it seems that kexec does not honour the memblock_reserve() table
>> >>> > > >> when booting the next kernel.
>> >>> > > >
>> >>> > > > not really.
>> >>> > > >
>> >>> > > >> > (In other words, can or should we skip some part of ACPI-related init code
>> >>> > > >> > on crash dump kernel?)
>> >>> > > >> >
>> >>> > > >>
>> >>> > > >> I don't think so. And the change to the handling of ACPI reclaim
>> >>> > > >> regions only revealed the bug, not created it (given that other
>> >>> > > >> memblock_reserve regions may be affected as well)
>> >>> > > >
>> >>> > > > As whether we should honor such reserved regions over kexec'ing
>> >>> > > > depends on each one's specific nature, we will have to take care one-by-one.
>> >>> > > > As a matter of fact, no information about "reserved" memblocks is
>> >>> > > > exposed to user space (via proc/iomem).
>> >>> > > >
>> >>> > >
>> >>> > > That is why I suggested (somewhere in this thread?) to not expose them
>> >>> > > as 'System RAM'. Do you think that could solve this?
>> >>> >
>> >>> > Memblock-reserv'ing them is necessary to prevent their corruption and
>> >>> > marking them under another name in /proc/iomem would also be good in order
>> >>> > not to allocate them as part of crash kernel's memory.
>> >>> >
>> >>> > But I'm not still convinced that we should export them in useable-
>> >>> > memory-range to crash dump kernel. They will be accessed through
>> >>> > acpi_os_map_memory() and so won't be required to be part of system ram
>> >>> > (or memblocks), I guess.
>> >>> >     -> Bhupesh?
>> >>>
>> >>> I forgot how arm64 kernel retrieve the memory ranges and initialize
>> >>> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
>> >>> the memory according to the efi memmap?  For kdump kernel anything other
>> >>> than usable memory (which is from the dt node instead) should be
>> >>> reinitialized according to efi passed info, no?
>> >>
>> >> All the regions exported in efi memmap will be added to memblock.memory
>> >> in (u)efi_init() and then trimmed down to the exact range specified as
>> >> usable-memory-range by fdt_enforce_memory_region().
>> >>
>> >> Now I noticed that the current fdt_enforce_memory_region() may not work well
>> >> with multiple entries in usable-memory-range.
>> >>
>> >
>> > In any case, the root of the problem is that memory regions lose their
>> > 'memory' annotation due to the way the memory map is mangled before
>> > being supplied to the kexec kernel.
>> >
>> > Would it be possible to classify all memory that we want to hide from
>> > the kexec kernel as NOMAP instead? That way, it will not be mapped
>> > implicitly, but will still be mapped cacheable by acpi_os_ioremap(),
>> > so this seems to be the most appropriate way to deal with the host
>> > kernel's memory contents.
>>
>> Hmm. wouldn't appending the acpi reclaim regions to
>> 'linux,usable-memory-range' in the dtb being passed to the crashkernel
>> be better? Because its indirectly achieving a similar objective
>> (although may be a subset of all System RAM regions on the primary
>> kernel's memory).
>>
>> I am not aware of the background about the current kexec-tools
>> implementation where we add only the crashkernel range to the dtb
>> being passed to the crashkernel.
>>
>> Probably Akashi can answer better, as to how we arrived at this design
>> approach and why we didn't want to expose all System RAM regions (i.e.
>> ! NOMPAP regions) to the crashkernel.
>>
>> I am suspecting that some issues were seen/meet when the System RAM (!
>> NOMAP regions) were exposed to the crashkernel, and that's why we
>> finalized on this design approach, but this is something which is just
>> my guess.
>>
>> Regards,
>> Bhupesh
>>
>> >>> >
>> >>> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
>> >>> > via a kernel command line parameter, "memmap=".
>> >>>
>> >>> memmap= is only used in old kexec-tools, now we are passing them via
>> >>> e820 table.
>> >>
>> >> Thanks. I remember that you have explained it before.
>> >>
>> >> -Takahiro AKASHI
>> >>
>> >>> [snip]
>> >>>
>> >>> Thanks
>> >>> Dave
>
> ===8<==
> From 74e2451fea83d546feae76160ba7de426913fe03 Mon Sep 17 00:00:00 2001
> From: AKASHI Takahiro <takahiro.akashi at linaro.org>
> Date: Thu, 21 Dec 2017 19:14:23 +0900
> Subject: [PATCH] arm64: kdump: mark unusable memory as NOMAP
>
> ---
>  arch/arm64/mm/init.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index 00e7b900ca41..8175db94257b 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -352,11 +352,17 @@ static void __init fdt_enforce_memory_region(void)
>         struct memblock_region reg = {
>                 .size = 0,
>         };
> +       u64 idx;
> +       phys_addr_t start, end;
>
>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>
> -       if (reg.size)
> -               memblock_cap_memory_range(reg.base, reg.size);
> +       if (reg.size) {
> +               for_each_free_mem_range(idx, NUMA_NO_NODE, MEMBLOCK_NONE,
> +                                       &start, &end, NULL)
> +                       memblock_mark_nomap(start, end - start);
> +               memblock_clear_nomap(reg.base, reg.size);
> +       }
>  }
>
>  void __init arm64_memblock_init(void)
> --
> 2.15.1
>

Thanks for the patch. After applying this on top of
4.15.0-rc4-next-20171220, there seems to be a improvement and the
crashkernel boot no longer hangs while trying to access the acpi
tables.

However I notice a minor issue. Please see the log below for
reference, the following message keeps spamming the console but I see
the crashkernel boot proceed further.:

[    0.000000] ACPI: NUMA: SRAT: PXM 3 -> MPIDR 0x70303 -> Node 3
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x2000000000-0x2fffffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0x1fffffffff]
[    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xa000000000-0xafffffffff]
[    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x9000000000-0x9fffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe200-0x1ffbffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x1ffbffc400-0x1ffbffe1ff]
[    0.000000] NUMA: NODE_DATA(1) on node 0
[    0.000000] NUMA: NODE_DATA [mem 0x1ffbffa600-0x1ffbffc3ff]
[    0.000000] NUMA: NODE_DATA(2) on node 0
[    0.000000] NUMA: NODE_DATA [mem 0x1ffbff8800-0x1ffbffa5ff]
[    0.000000] NUMA: NODE_DATA(3) on node 0
[    0.000000] [ffff7fe008000000-ffff7fe00800ffff] potential offnode
page_structs
[    0.000000] [ffff7fe008010000-ffff7fe00801ffff] potential offnode
page_structs
[    0.000000] [ffff7fe008020000-ffff7fe00802ffff] potential offnode
page_structs
[    0.000000] [ffff7fe008030000-ffff7fe00803ffff] potential offnode
page_structs
[    0.000000] [ffff7fe008040000-ffff7fe00804ffff] potential offnode
page_structs
[    0.000000] [ffff7fe008050000-ffff7fe00805ffff] potential offnode
page_structs

[snip..]
[    0.000000] [ffff7fe0081f0000-ffff7fe0081fffff] potential offnode
page_structs

This WARNING message seems to come from vmemmap_verify() inside
'mm/sparse-vmemmap.c'

Regards,
Bhupesh


From ats-kumagai at wm.jp.nec.com  Thu Dec 21 00:48:28 2017
From: ats-kumagai at wm.jp.nec.com (Atsushi Kumagai)
Date: Thu, 21 Dec 2017 08:48:28 +0000
Subject: [PATCH] handle renamed init_level4_pgt -> init_top_pgt
In-Reply-To: <20171220072443.GA7413@dhcp-128-65.nay.redhat.com>
References: <0910DD04CBD6DE4193FCF86B9C00BE9701F22E63@BPXM01GP.gisp.nec.co.jp>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2A617@BPXM01GP.gisp.nec.co.jp>
 <20171220072443.GA7413@dhcp-128-65.nay.redhat.com>
Message-ID: <0910DD04CBD6DE4193FCF86B9C00BE9701F2D609@BPXM01GP.gisp.nec.co.jp>

Hello Dave,

>[dyoung at dhcp-*-* makedumpfile]$ sudo ./makedumpfile -l -d 31 /mnt/vmcore/vmcore /tmp/vmcore.1
>The kernel version is not supported.
>The makedumpfile operation may be incomplete.
>Checking for memory holes                         : [100.0 %] |                  __vtop4_x86_64: Can't get a valid pte.
>readmem: Can't convert a virtual address(ffff88007ebb1000) to physical address.
>readmem: type_addr: 0, addr:ffff88007ebb1000, size:32768
>__exclude_unnecessary_pages: Can't read the buffer of struct page.
>create_2nd_bitmap: Can't exclude unnecessary pages.
>
>makedumpfile Failed.
>
>If you need the vmcore for debugging please let me know, my test is just
>a normal test in kvm guest.

Thanks for your report.
I can't reproduce that in my environment, could you give me the vmcore ?
It's helpful if you append the vmlinux and the .config.

Thanks,
Atsushi Kumagai

>> >>Signed-off-by: Jeff Mahoney <jeffm at suse.com>
>> >>---
>> >> arch/x86_64.c  | 24 +++++++++++++++++++++---
>> >> makedumpfile.c |  6 ++++++
>> >> makedumpfile.h |  2 ++
>> >> 3 files changed, 29 insertions(+), 3 deletions(-)
>> >>
>> >>diff --git a/arch/x86_64.c b/arch/x86_64.c
>> >>index 08dd6b2..9b09035 100644
>> >>--- a/arch/x86_64.c
>> >>+++ b/arch/x86_64.c
>> >>@@ -259,16 +259,26 @@ vtop4_x86_64(unsigned long vaddr)
>> >> {
>> >> 	unsigned long page_dir, pml4, pgd_paddr, pgd_pte, pmd_paddr, pmd_pte;
>> >> 	unsigned long pte_paddr, pte;
>> >>+	unsigned long init_level4_pgt;
>> >>
>> >>-	if (SYMBOL(init_level4_pgt) == NOT_FOUND_SYMBOL) {
>> >>+	if (SYMBOL(init_level4_pgt) != NOT_FOUND_SYMBOL)
>> >>+		init_level4_pgt = SYMBOL(init_level4_pgt);
>> >>+	else if (SYMBOL(init_top_pgt) != NOT_FOUND_SYMBOL)
>> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
>> >>+	else {
>> >> 		ERRMSG("Can't get the symbol of init_level4_pgt.\n");
>> >> 		return NOT_PADDR;
>> >> 	}
>> >>
>> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
>> >>+		ERRMSG("Kernel is built with 5-level page tables\n");
>> >>+		return NOT_PADDR;
>> >>+	}
>> >>+
>> >> 	/*
>> >> 	 * Get PGD.
>> >> 	 */
>> >>-	page_dir = SYMBOL(init_level4_pgt) - __START_KERNEL_map + info->phys_base;
>> >>+	page_dir = init_level4_pgt - __START_KERNEL_map + info->phys_base;
>> >> 	if (is_xen_memory()) {
>> >> 		page_dir = ptom_xen(page_dir);
>> >> 		if (page_dir == NOT_PADDR)
>> >>@@ -549,8 +559,16 @@ find_vmemmap_x86_64()
>> >> 	struct vmap_pfns *vmapp, *vmaphead = NULL, *cur, *tail;
>> >>
>> >> 	init_level4_pgt = SYMBOL(init_level4_pgt);
>> >>+	if (init_level4_pgt == NOT_FOUND_SYMBOL)
>> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
>> >>+
>> >> 	if (init_level4_pgt == NOT_FOUND_SYMBOL) {
>> >>-		ERRMSG("init_level4_pgt not found\n");
>> >>+		ERRMSG("init_level4_pgt/init_top_pgt not found\n");
>> >>+		return FAILED;
>> >>+	}
>> >>+
>> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
>> >>+		ERRMSG("kernel is configured for 5-level page tables\n");
>> >> 		return FAILED;
>> >> 	}
>> >> 	pagestructsize = size_table.page;
>> >>diff --git a/makedumpfile.c b/makedumpfile.c
>> >>index f85003a..6e5ec34 100644
>> >>--- a/makedumpfile.c
>> >>+++ b/makedumpfile.c
>> >>@@ -1486,6 +1486,8 @@ get_symbol_info(void)
>> >> 	SYMBOL_INIT(_stext, "_stext");
>> >> 	SYMBOL_INIT(swapper_pg_dir, "swapper_pg_dir");
>> >> 	SYMBOL_INIT(init_level4_pgt, "init_level4_pgt");
>> >>+	SYMBOL_INIT(level4_kernel_pgt, "level4_kernel_pgt");
>> >>+	SYMBOL_INIT(init_top_pgt, "init_top_pgt");
>> >> 	SYMBOL_INIT(vmlist, "vmlist");
>> >> 	SYMBOL_INIT(vmap_area_list, "vmap_area_list");
>> >> 	SYMBOL_INIT(node_online_map, "node_online_map");
>> >>@@ -2105,6 +2107,8 @@ write_vmcoreinfo_data(void)
>> >> 	WRITE_SYMBOL("_stext", _stext);
>> >> 	WRITE_SYMBOL("swapper_pg_dir", swapper_pg_dir);
>> >> 	WRITE_SYMBOL("init_level4_pgt", init_level4_pgt);
>> >>+	WRITE_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
>> >>+	WRITE_SYMBOL("init_top_pgt", init_top_pgt);
>> >> 	WRITE_SYMBOL("vmlist", vmlist);
>> >> 	WRITE_SYMBOL("vmap_area_list", vmap_area_list);
>> >> 	WRITE_SYMBOL("node_online_map", node_online_map);
>> >>@@ -2500,6 +2504,8 @@ read_vmcoreinfo(void)
>> >> 	READ_SYMBOL("_stext", _stext);
>> >> 	READ_SYMBOL("swapper_pg_dir", swapper_pg_dir);
>> >> 	READ_SYMBOL("init_level4_pgt", init_level4_pgt);
>> >>+	READ_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
>> >>+	READ_SYMBOL("init_top_pgt", init_top_pgt);
>> >> 	READ_SYMBOL("vmlist", vmlist);
>> >> 	READ_SYMBOL("vmap_area_list", vmap_area_list);
>> >> 	READ_SYMBOL("node_online_map", node_online_map);
>> >>diff --git a/makedumpfile.h b/makedumpfile.h
>> >>index 8a05794..9357e47 100644
>> >>--- a/makedumpfile.h
>> >>+++ b/makedumpfile.h
>> >>@@ -1517,6 +1517,8 @@ struct symbol_table {
>> >> 	unsigned long long	_stext;
>> >> 	unsigned long long	swapper_pg_dir;
>> >> 	unsigned long long	init_level4_pgt;
>> >>+	unsigned long long	level4_kernel_pgt;
>> >>+	unsigned long long	init_top_pgt;
>> >> 	unsigned long long	vmlist;
>> >> 	unsigned long long	vmap_area_list;
>> >> 	unsigned long long	phys_base;
>> >>
>> >>
>> >>--
>> >>Jeff Mahoney
>> >>SUSE Labs
>> >
>> >
>> >
>> >_______________________________________________
>> >kexec mailing list
>> >kexec at lists.infradead.org
>> >http://lists.infradead.org/mailman/listinfo/kexec
>>
>>
>> _______________________________________________
>> kexec mailing list
>> kexec at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/kexec
>
>Thanks
>Dave




From dyoung at redhat.com  Thu Dec 21 20:54:44 2017
From: dyoung at redhat.com (Dave Young)
Date: Fri, 22 Dec 2017 12:54:44 +0800
Subject: [PATCH] handle renamed init_level4_pgt -> init_top_pgt
In-Reply-To: <0910DD04CBD6DE4193FCF86B9C00BE9701F2D609@BPXM01GP.gisp.nec.co.jp>
References: <0910DD04CBD6DE4193FCF86B9C00BE9701F22E63@BPXM01GP.gisp.nec.co.jp>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2A617@BPXM01GP.gisp.nec.co.jp>
 <20171220072443.GA7413@dhcp-128-65.nay.redhat.com>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2D609@BPXM01GP.gisp.nec.co.jp>
Message-ID: <20171222045444.GA978@dhcp-128-65.nay.redhat.com>

Hi Atsushi,
On 12/21/17 at 08:48am, Atsushi Kumagai wrote:
> Hello Dave,
> 
> >[dyoung at dhcp-*-* makedumpfile]$ sudo ./makedumpfile -l -d 31 /mnt/vmcore/vmcore /tmp/vmcore.1
> >The kernel version is not supported.
> >The makedumpfile operation may be incomplete.
> >Checking for memory holes                         : [100.0 %] |                  __vtop4_x86_64: Can't get a valid pte.
> >readmem: Can't convert a virtual address(ffff88007ebb1000) to physical address.
> >readmem: type_addr: 0, addr:ffff88007ebb1000, size:32768
> >__exclude_unnecessary_pages: Can't read the buffer of struct page.
> >create_2nd_bitmap: Can't exclude unnecessary pages.
> >
> >makedumpfile Failed.
> >
> >If you need the vmcore for debugging please let me know, my test is just
> >a normal test in kvm guest.
> 
> Thanks for your report.
> I can't reproduce that in my environment, could you give me the vmcore ?
> It's helpful if you append the vmlinux and the .config.

I'm bisecting it, nearly finished, will post the results and share
vmcore if needed soon.

> 
> Thanks,
> Atsushi Kumagai
> 
> >> >>Signed-off-by: Jeff Mahoney <jeffm at suse.com>
> >> >>---
> >> >> arch/x86_64.c  | 24 +++++++++++++++++++++---
> >> >> makedumpfile.c |  6 ++++++
> >> >> makedumpfile.h |  2 ++
> >> >> 3 files changed, 29 insertions(+), 3 deletions(-)
> >> >>
> >> >>diff --git a/arch/x86_64.c b/arch/x86_64.c
> >> >>index 08dd6b2..9b09035 100644
> >> >>--- a/arch/x86_64.c
> >> >>+++ b/arch/x86_64.c
> >> >>@@ -259,16 +259,26 @@ vtop4_x86_64(unsigned long vaddr)
> >> >> {
> >> >> 	unsigned long page_dir, pml4, pgd_paddr, pgd_pte, pmd_paddr, pmd_pte;
> >> >> 	unsigned long pte_paddr, pte;
> >> >>+	unsigned long init_level4_pgt;
> >> >>
> >> >>-	if (SYMBOL(init_level4_pgt) == NOT_FOUND_SYMBOL) {
> >> >>+	if (SYMBOL(init_level4_pgt) != NOT_FOUND_SYMBOL)
> >> >>+		init_level4_pgt = SYMBOL(init_level4_pgt);
> >> >>+	else if (SYMBOL(init_top_pgt) != NOT_FOUND_SYMBOL)
> >> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
> >> >>+	else {
> >> >> 		ERRMSG("Can't get the symbol of init_level4_pgt.\n");
> >> >> 		return NOT_PADDR;
> >> >> 	}
> >> >>
> >> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
> >> >>+		ERRMSG("Kernel is built with 5-level page tables\n");
> >> >>+		return NOT_PADDR;
> >> >>+	}
> >> >>+
> >> >> 	/*
> >> >> 	 * Get PGD.
> >> >> 	 */
> >> >>-	page_dir = SYMBOL(init_level4_pgt) - __START_KERNEL_map + info->phys_base;
> >> >>+	page_dir = init_level4_pgt - __START_KERNEL_map + info->phys_base;
> >> >> 	if (is_xen_memory()) {
> >> >> 		page_dir = ptom_xen(page_dir);
> >> >> 		if (page_dir == NOT_PADDR)
> >> >>@@ -549,8 +559,16 @@ find_vmemmap_x86_64()
> >> >> 	struct vmap_pfns *vmapp, *vmaphead = NULL, *cur, *tail;
> >> >>
> >> >> 	init_level4_pgt = SYMBOL(init_level4_pgt);
> >> >>+	if (init_level4_pgt == NOT_FOUND_SYMBOL)
> >> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
> >> >>+
> >> >> 	if (init_level4_pgt == NOT_FOUND_SYMBOL) {
> >> >>-		ERRMSG("init_level4_pgt not found\n");
> >> >>+		ERRMSG("init_level4_pgt/init_top_pgt not found\n");
> >> >>+		return FAILED;
> >> >>+	}
> >> >>+
> >> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
> >> >>+		ERRMSG("kernel is configured for 5-level page tables\n");
> >> >> 		return FAILED;
> >> >> 	}
> >> >> 	pagestructsize = size_table.page;
> >> >>diff --git a/makedumpfile.c b/makedumpfile.c
> >> >>index f85003a..6e5ec34 100644
> >> >>--- a/makedumpfile.c
> >> >>+++ b/makedumpfile.c
> >> >>@@ -1486,6 +1486,8 @@ get_symbol_info(void)
> >> >> 	SYMBOL_INIT(_stext, "_stext");
> >> >> 	SYMBOL_INIT(swapper_pg_dir, "swapper_pg_dir");
> >> >> 	SYMBOL_INIT(init_level4_pgt, "init_level4_pgt");
> >> >>+	SYMBOL_INIT(level4_kernel_pgt, "level4_kernel_pgt");
> >> >>+	SYMBOL_INIT(init_top_pgt, "init_top_pgt");
> >> >> 	SYMBOL_INIT(vmlist, "vmlist");
> >> >> 	SYMBOL_INIT(vmap_area_list, "vmap_area_list");
> >> >> 	SYMBOL_INIT(node_online_map, "node_online_map");
> >> >>@@ -2105,6 +2107,8 @@ write_vmcoreinfo_data(void)
> >> >> 	WRITE_SYMBOL("_stext", _stext);
> >> >> 	WRITE_SYMBOL("swapper_pg_dir", swapper_pg_dir);
> >> >> 	WRITE_SYMBOL("init_level4_pgt", init_level4_pgt);
> >> >>+	WRITE_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
> >> >>+	WRITE_SYMBOL("init_top_pgt", init_top_pgt);
> >> >> 	WRITE_SYMBOL("vmlist", vmlist);
> >> >> 	WRITE_SYMBOL("vmap_area_list", vmap_area_list);
> >> >> 	WRITE_SYMBOL("node_online_map", node_online_map);
> >> >>@@ -2500,6 +2504,8 @@ read_vmcoreinfo(void)
> >> >> 	READ_SYMBOL("_stext", _stext);
> >> >> 	READ_SYMBOL("swapper_pg_dir", swapper_pg_dir);
> >> >> 	READ_SYMBOL("init_level4_pgt", init_level4_pgt);
> >> >>+	READ_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
> >> >>+	READ_SYMBOL("init_top_pgt", init_top_pgt);
> >> >> 	READ_SYMBOL("vmlist", vmlist);
> >> >> 	READ_SYMBOL("vmap_area_list", vmap_area_list);
> >> >> 	READ_SYMBOL("node_online_map", node_online_map);
> >> >>diff --git a/makedumpfile.h b/makedumpfile.h
> >> >>index 8a05794..9357e47 100644
> >> >>--- a/makedumpfile.h
> >> >>+++ b/makedumpfile.h
> >> >>@@ -1517,6 +1517,8 @@ struct symbol_table {
> >> >> 	unsigned long long	_stext;
> >> >> 	unsigned long long	swapper_pg_dir;
> >> >> 	unsigned long long	init_level4_pgt;
> >> >>+	unsigned long long	level4_kernel_pgt;
> >> >>+	unsigned long long	init_top_pgt;
> >> >> 	unsigned long long	vmlist;
> >> >> 	unsigned long long	vmap_area_list;
> >> >> 	unsigned long long	phys_base;
> >> >>
> >> >>
> >> >>--
> >> >>Jeff Mahoney
> >> >>SUSE Labs
> >> >
> >> >
> >> >
> >> >_______________________________________________
> >> >kexec mailing list
> >> >kexec at lists.infradead.org
> >> >http://lists.infradead.org/mailman/listinfo/kexec
> >>
> >>
> >> _______________________________________________
> >> kexec mailing list
> >> kexec at lists.infradead.org
> >> http://lists.infradead.org/mailman/listinfo/kexec
> >
> >Thanks
> >Dave
> 
> 

Thanks
Dave


From dyoung at redhat.com  Thu Dec 21 21:48:29 2017
From: dyoung at redhat.com (Dave Young)
Date: Fri, 22 Dec 2017 13:48:29 +0800
Subject: [PATCH] handle renamed init_level4_pgt -> init_top_pgt
In-Reply-To: <20171222045444.GA978@dhcp-128-65.nay.redhat.com>
References: <0910DD04CBD6DE4193FCF86B9C00BE9701F22E63@BPXM01GP.gisp.nec.co.jp>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2A617@BPXM01GP.gisp.nec.co.jp>
 <20171220072443.GA7413@dhcp-128-65.nay.redhat.com>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2D609@BPXM01GP.gisp.nec.co.jp>
 <20171222045444.GA978@dhcp-128-65.nay.redhat.com>
Message-ID: <20171222054829.GA21523@dhcp-128-65.nay.redhat.com>

On 12/22/17 at 12:54pm, Dave Young wrote:
> Hi Atsushi,
> On 12/21/17 at 08:48am, Atsushi Kumagai wrote:
> > Hello Dave,
> > 
> > >[dyoung at dhcp-*-* makedumpfile]$ sudo ./makedumpfile -l -d 31 /mnt/vmcore/vmcore /tmp/vmcore.1
> > >The kernel version is not supported.
> > >The makedumpfile operation may be incomplete.
> > >Checking for memory holes                         : [100.0 %] |                  __vtop4_x86_64: Can't get a valid pte.
> > >readmem: Can't convert a virtual address(ffff88007ebb1000) to physical address.
> > >readmem: type_addr: 0, addr:ffff88007ebb1000, size:32768
> > >__exclude_unnecessary_pages: Can't read the buffer of struct page.
> > >create_2nd_bitmap: Can't exclude unnecessary pages.
> > >
> > >makedumpfile Failed.
> > >
> > >If you need the vmcore for debugging please let me know, my test is just
> > >a normal test in kvm guest.
> > 
> > Thanks for your report.
> > I can't reproduce that in my environment, could you give me the vmcore ?
> > It's helpful if you append the vmlinux and the .config.
> 
> I'm bisecting it, nearly finished, will post the results and share
> vmcore if needed soon.

vmcore/vmlinux (gzipped) and .config:
http://people.redhat.com/ruyang/rhbz1528542/

Bisect result is below:

83e3c48729d9ebb7af5a31a504f3fd6aff0348c4 is the first bad commit
commit 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4
Author: Kirill A. Shutemov <kirill.shutemov at linux.intel.com>
Date:   Fri Sep 29 17:08:16 2017 +0300

    mm/sparsemem: Allocate mem_section at runtime for CONFIG_SPARSEMEM_EXTREME=y
    
    Size of the mem_section[] array depends on the size of the physical address space.
    
    In preparation for boot-time switching between paging modes on x86-64
    we need to make the allocation of mem_section[] dynamic, because otherwise
    we waste a lot of RAM: with CONFIG_NODE_SHIFT=10, mem_section[] size is 32kB
    for 4-level paging and 2MB for 5-level paging mode.
    
    The patch allocates the array on the first call to sparse_memory_present_with_active_regions().
    
    Signed-off-by: Kirill A. Shutemov <kirill.shutemov at linux.intel.com>
    Cc: Andrew Morton <akpm at linux-foundation.org>
    Cc: Andy Lutomirski <luto at amacapital.net>
    Cc: Borislav Petkov <bp at suse.de>
    Cc: Cyrill Gorcunov <gorcunov at openvz.org>
    Cc: Linus Torvalds <torvalds at linux-foundation.org>
    Cc: Peter Zijlstra <peterz at infradead.org>
    Cc: Thomas Gleixner <tglx at linutronix.de>
    Cc: linux-mm at kvack.org
    Link: http://lkml.kernel.org/r/20170929140821.37654-2-kirill.shutemov at linux.intel.com
    Signed-off-by: Ingo Molnar <mingo at kernel.org>

:040000 040000 68b7ff3eedc2c9ff56e31108f7e982eacbb233fc 0014ee63bebe14efb0e36e0028e2cbe718fd6c30 M	include
:040000 040000 78adbc296527c802400b1f68e0fbd716920726fa a4eea8117cb318527c0d5a6281d68f312f644831 M	mm

> 
> > 
> > Thanks,
> > Atsushi Kumagai
> > 
> > >> >>Signed-off-by: Jeff Mahoney <jeffm at suse.com>
> > >> >>---
> > >> >> arch/x86_64.c  | 24 +++++++++++++++++++++---
> > >> >> makedumpfile.c |  6 ++++++
> > >> >> makedumpfile.h |  2 ++
> > >> >> 3 files changed, 29 insertions(+), 3 deletions(-)
> > >> >>
> > >> >>diff --git a/arch/x86_64.c b/arch/x86_64.c
> > >> >>index 08dd6b2..9b09035 100644
> > >> >>--- a/arch/x86_64.c
> > >> >>+++ b/arch/x86_64.c
> > >> >>@@ -259,16 +259,26 @@ vtop4_x86_64(unsigned long vaddr)
> > >> >> {
> > >> >> 	unsigned long page_dir, pml4, pgd_paddr, pgd_pte, pmd_paddr, pmd_pte;
> > >> >> 	unsigned long pte_paddr, pte;
> > >> >>+	unsigned long init_level4_pgt;
> > >> >>
> > >> >>-	if (SYMBOL(init_level4_pgt) == NOT_FOUND_SYMBOL) {
> > >> >>+	if (SYMBOL(init_level4_pgt) != NOT_FOUND_SYMBOL)
> > >> >>+		init_level4_pgt = SYMBOL(init_level4_pgt);
> > >> >>+	else if (SYMBOL(init_top_pgt) != NOT_FOUND_SYMBOL)
> > >> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
> > >> >>+	else {
> > >> >> 		ERRMSG("Can't get the symbol of init_level4_pgt.\n");
> > >> >> 		return NOT_PADDR;
> > >> >> 	}
> > >> >>
> > >> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
> > >> >>+		ERRMSG("Kernel is built with 5-level page tables\n");
> > >> >>+		return NOT_PADDR;
> > >> >>+	}
> > >> >>+
> > >> >> 	/*
> > >> >> 	 * Get PGD.
> > >> >> 	 */
> > >> >>-	page_dir = SYMBOL(init_level4_pgt) - __START_KERNEL_map + info->phys_base;
> > >> >>+	page_dir = init_level4_pgt - __START_KERNEL_map + info->phys_base;
> > >> >> 	if (is_xen_memory()) {
> > >> >> 		page_dir = ptom_xen(page_dir);
> > >> >> 		if (page_dir == NOT_PADDR)
> > >> >>@@ -549,8 +559,16 @@ find_vmemmap_x86_64()
> > >> >> 	struct vmap_pfns *vmapp, *vmaphead = NULL, *cur, *tail;
> > >> >>
> > >> >> 	init_level4_pgt = SYMBOL(init_level4_pgt);
> > >> >>+	if (init_level4_pgt == NOT_FOUND_SYMBOL)
> > >> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
> > >> >>+
> > >> >> 	if (init_level4_pgt == NOT_FOUND_SYMBOL) {
> > >> >>-		ERRMSG("init_level4_pgt not found\n");
> > >> >>+		ERRMSG("init_level4_pgt/init_top_pgt not found\n");
> > >> >>+		return FAILED;
> > >> >>+	}
> > >> >>+
> > >> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
> > >> >>+		ERRMSG("kernel is configured for 5-level page tables\n");
> > >> >> 		return FAILED;
> > >> >> 	}
> > >> >> 	pagestructsize = size_table.page;
> > >> >>diff --git a/makedumpfile.c b/makedumpfile.c
> > >> >>index f85003a..6e5ec34 100644
> > >> >>--- a/makedumpfile.c
> > >> >>+++ b/makedumpfile.c
> > >> >>@@ -1486,6 +1486,8 @@ get_symbol_info(void)
> > >> >> 	SYMBOL_INIT(_stext, "_stext");
> > >> >> 	SYMBOL_INIT(swapper_pg_dir, "swapper_pg_dir");
> > >> >> 	SYMBOL_INIT(init_level4_pgt, "init_level4_pgt");
> > >> >>+	SYMBOL_INIT(level4_kernel_pgt, "level4_kernel_pgt");
> > >> >>+	SYMBOL_INIT(init_top_pgt, "init_top_pgt");
> > >> >> 	SYMBOL_INIT(vmlist, "vmlist");
> > >> >> 	SYMBOL_INIT(vmap_area_list, "vmap_area_list");
> > >> >> 	SYMBOL_INIT(node_online_map, "node_online_map");
> > >> >>@@ -2105,6 +2107,8 @@ write_vmcoreinfo_data(void)
> > >> >> 	WRITE_SYMBOL("_stext", _stext);
> > >> >> 	WRITE_SYMBOL("swapper_pg_dir", swapper_pg_dir);
> > >> >> 	WRITE_SYMBOL("init_level4_pgt", init_level4_pgt);
> > >> >>+	WRITE_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
> > >> >>+	WRITE_SYMBOL("init_top_pgt", init_top_pgt);
> > >> >> 	WRITE_SYMBOL("vmlist", vmlist);
> > >> >> 	WRITE_SYMBOL("vmap_area_list", vmap_area_list);
> > >> >> 	WRITE_SYMBOL("node_online_map", node_online_map);
> > >> >>@@ -2500,6 +2504,8 @@ read_vmcoreinfo(void)
> > >> >> 	READ_SYMBOL("_stext", _stext);
> > >> >> 	READ_SYMBOL("swapper_pg_dir", swapper_pg_dir);
> > >> >> 	READ_SYMBOL("init_level4_pgt", init_level4_pgt);
> > >> >>+	READ_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
> > >> >>+	READ_SYMBOL("init_top_pgt", init_top_pgt);
> > >> >> 	READ_SYMBOL("vmlist", vmlist);
> > >> >> 	READ_SYMBOL("vmap_area_list", vmap_area_list);
> > >> >> 	READ_SYMBOL("node_online_map", node_online_map);
> > >> >>diff --git a/makedumpfile.h b/makedumpfile.h
> > >> >>index 8a05794..9357e47 100644
> > >> >>--- a/makedumpfile.h
> > >> >>+++ b/makedumpfile.h
> > >> >>@@ -1517,6 +1517,8 @@ struct symbol_table {
> > >> >> 	unsigned long long	_stext;
> > >> >> 	unsigned long long	swapper_pg_dir;
> > >> >> 	unsigned long long	init_level4_pgt;
> > >> >>+	unsigned long long	level4_kernel_pgt;
> > >> >>+	unsigned long long	init_top_pgt;
> > >> >> 	unsigned long long	vmlist;
> > >> >> 	unsigned long long	vmap_area_list;
> > >> >> 	unsigned long long	phys_base;
> > >> >>
> > >> >>
> > >> >>--
> > >> >>Jeff Mahoney
> > >> >>SUSE Labs
> > >> >
> > >> >
> > >> >
> > >> >_______________________________________________
> > >> >kexec mailing list
> > >> >kexec at lists.infradead.org
> > >> >http://lists.infradead.org/mailman/listinfo/kexec
> > >>
> > >>
> > >> _______________________________________________
> > >> kexec mailing list
> > >> kexec at lists.infradead.org
> > >> http://lists.infradead.org/mailman/listinfo/kexec
> > >
> > >Thanks
> > >Dave
> > 
> > 
> 
> Thanks
> Dave


From takahiro.akashi at linaro.org  Fri Dec 22 00:33:10 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Fri, 22 Dec 2017 17:33:10 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
References: <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
 <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
Message-ID: <20171222083308.cy3vwlyhig22q5ee@fireball>

On Thu, Dec 21, 2017 at 05:36:30PM +0530, Bhupesh Sharma wrote:
> Hello Akashi,
> 
> On Thu, Dec 21, 2017 at 4:04 PM, AKASHI Takahiro
> <takahiro.akashi at linaro.org> wrote:
> > Bhupesh,
> >
> > Can you test the patch attached below, please?
> >
> > It is intended to retain already-reserved regions (ACPI reclaim memory
> > in this case) in system ram (i.e. memblock.memory) without explicitly
> > exporting them via usable-memory-range.
> > (I still have to figure out what the side-effect of this patch is.)
> >
> > Thanks,
> > -Takahiro AKASHI
> >
> > On Thu, Dec 21, 2017 at 01:30:43AM +0530, Bhupesh Sharma wrote:
> >> On Tue, Dec 19, 2017 at 6:39 PM, Ard Biesheuvel
> >> <ard.biesheuvel at linaro.org> wrote:
> >> > On 19 December 2017 at 07:09, AKASHI Takahiro
> >> > <takahiro.akashi at linaro.org> wrote:
> >> >> On Mon, Dec 18, 2017 at 01:40:09PM +0800, Dave Young wrote:
> >> >>> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
> >> >>> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> >> >>> > > On 13 December 2017 at 12:16, AKASHI Takahiro
> >> >>> > > <takahiro.akashi at linaro.org> wrote:
> >> >>> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> >> >>> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
> >> >>> > > >> <takahiro.akashi at linaro.org> wrote:
> >> >>> > > >> > Bhupesh, Ard,
> >> >>> > > >> >
> >> >>> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> >> >>> > > >> >> Hi Ard, Akashi
> >> >>> > > >> >>
> >> >>> > > >> > (snip)
> >> >>> > > >> >
> >> >>> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> >> >>> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> >> >>> > > >> >> identify its own usable memory and exclude, at its boot time, any
> >> >>> > > >> >> other memory areas that are part of the panicked kernel's memory.
> >> >>> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> >> >>> > > >> >> , for details)
> >> >>> > > >> >
> >> >>> > > >> > Right.
> >> >>> > > >> >
> >> >>> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> >> >>> > > >> >> with the crashkernel memory range:
> >> >>> > > >> >>
> >> >>> > > >> >>                 /* add linux,usable-memory-range */
> >> >>> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> >> >>> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> >> >>> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> >> >>> > > >> >>                                 address_cells, size_cells);
> >> >>> > > >> >>
> >> >>> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> >> >>> > > >> >> , for details)
> >> >>> > > >> >>
> >> >>> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> >> >>> > > >> >> they are marked as System RAM or as RESERVED. As,
> >> >>> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
> >> >>> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> >> >>> > > >> >>
> >> >>> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
> >> >>> > > >> >> ACPI memory and crashes while trying to access the same:
> >> >>> > > >> >>
> >> >>> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> >> >>> > > >> >> -r`.img --reuse-cmdline -d
> >> >>> > > >> >>
> >> >>> > > >> >> [snip..]
> >> >>> > > >> >>
> >> >>> > > >> >> Reserved memory range
> >> >>> > > >> >> 000000000e800000-000000002e7fffff (0)
> >> >>> > > >> >>
> >> >>> > > >> >> Coredump memory ranges
> >> >>> > > >> >> 0000000000000000-000000000e7fffff (0)
> >> >>> > > >> >> 000000002e800000-000000003961ffff (0)
> >> >>> > > >> >> 0000000039d40000-000000003ed2ffff (0)
> >> >>> > > >> >> 000000003ed60000-000000003fbfffff (0)
> >> >>> > > >> >> 0000001040000000-0000001ffbffffff (0)
> >> >>> > > >> >> 0000002000000000-0000002ffbffffff (0)
> >> >>> > > >> >> 0000009000000000-0000009ffbffffff (0)
> >> >>> > > >> >> 000000a000000000-000000affbffffff (0)
> >> >>> > > >> >>
> >> >>> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> >> >>> > > >> >> memory cap'ing passed to the crash kernel inside
> >> >>> > > >> >> 'arch/arm64/mm/init.c' (see below):
> >> >>> > > >> >>
> >> >>> > > >> >> static void __init fdt_enforce_memory_region(void)
> >> >>> > > >> >> {
> >> >>> > > >> >>         struct memblock_region reg = {
> >> >>> > > >> >>                 .size = 0,
> >> >>> > > >> >>         };
> >> >>> > > >> >>
> >> >>> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >> >>> > > >> >>
> >> >>> > > >> >>         if (reg.size)
> >> >>> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> >> >>> > > >> >> comment this out */
> >> >>> > > >> >> }
> >> >>> > > >> >
> >> >>> > > >> > Please just don't do that. It can cause a fatal damage on
> >> >>> > > >> > memory contents of the *crashed* kernel.
> >> >>> > > >> >
> >> >>> > > >> >> 5). Both the above temporary solutions fix the problem.
> >> >>> > > >> >>
> >> >>> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
> >> >>> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> >> >>> > > >> >> fail.
> >> >>> > > >> >>
> >> >>> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> >> >>> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> >> >>> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> >> >>> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> >> >>> > > >> >> dt node 'linux,usable-memory-range'
> >> >>> > > >> >
> >> >>> > > >> > I still don't understand why we need to carry over the information
> >> >>> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> >> >>> > > >> > such regions are free to be reused by the kernel after some point of
> >> >>> > > >> > initialization. Why does crash dump kernel need to know about them?
> >> >>> > > >> >
> >> >>> > > >>
> >> >>> > > >> Not really. According to the UEFI spec, they can be reclaimed after
> >> >>> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> >> >>> > > >> no longer needs them. Of course, in order to be able to boot a kexec
> >> >>> > > >> kernel, those regions needs to be preserved, which is why they are
> >> >>> > > >> memblock_reserve()'d now.
> >> >>> > > >
> >> >>> > > > For my better understandings, who is actually accessing such regions
> >> >>> > > > during boot time, uefi itself or efistub?
> >> >>> > > >
> >> >>> > >
> >> >>> > > No, only the kernel. This is where the ACPI tables are stored. For
> >> >>> > > instance, on QEMU we have
> >> >>> > >
> >> >>> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> >> >>> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> >> >>> > >   01000013)
> >> >>> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> >> >>> > > BXPC 00000001)
> >> >>> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> >> >>> > > BXPC 00000001)
> >> >>> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> >> >>> > > BXPC 00000001)
> >> >>> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> >> >>> > > BXPC 00000001)
> >> >>> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> >> >>> > > BXPC 00000001)
> >> >>> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> >> >>> > > BXPC 00000001)
> >> >>> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> >> >>> > > BXPC 00000001)
> >> >>> > >
> >> >>> > > covered by
> >> >>> > >
> >> >>> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> >> >>> > >  ...
> >> >>> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> >> >>> >
> >> >>> > OK. I mistakenly understood those regions could be freed after exiting
> >> >>> > UEFI boot services.
> >> >>> >
> >> >>> > >
> >> >>> > > >> So it seems that kexec does not honour the memblock_reserve() table
> >> >>> > > >> when booting the next kernel.
> >> >>> > > >
> >> >>> > > > not really.
> >> >>> > > >
> >> >>> > > >> > (In other words, can or should we skip some part of ACPI-related init code
> >> >>> > > >> > on crash dump kernel?)
> >> >>> > > >> >
> >> >>> > > >>
> >> >>> > > >> I don't think so. And the change to the handling of ACPI reclaim
> >> >>> > > >> regions only revealed the bug, not created it (given that other
> >> >>> > > >> memblock_reserve regions may be affected as well)
> >> >>> > > >
> >> >>> > > > As whether we should honor such reserved regions over kexec'ing
> >> >>> > > > depends on each one's specific nature, we will have to take care one-by-one.
> >> >>> > > > As a matter of fact, no information about "reserved" memblocks is
> >> >>> > > > exposed to user space (via proc/iomem).
> >> >>> > > >
> >> >>> > >
> >> >>> > > That is why I suggested (somewhere in this thread?) to not expose them
> >> >>> > > as 'System RAM'. Do you think that could solve this?
> >> >>> >
> >> >>> > Memblock-reserv'ing them is necessary to prevent their corruption and
> >> >>> > marking them under another name in /proc/iomem would also be good in order
> >> >>> > not to allocate them as part of crash kernel's memory.
> >> >>> >
> >> >>> > But I'm not still convinced that we should export them in useable-
> >> >>> > memory-range to crash dump kernel. They will be accessed through
> >> >>> > acpi_os_map_memory() and so won't be required to be part of system ram
> >> >>> > (or memblocks), I guess.
> >> >>> >     -> Bhupesh?
> >> >>>
> >> >>> I forgot how arm64 kernel retrieve the memory ranges and initialize
> >> >>> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
> >> >>> the memory according to the efi memmap?  For kdump kernel anything other
> >> >>> than usable memory (which is from the dt node instead) should be
> >> >>> reinitialized according to efi passed info, no?
> >> >>
> >> >> All the regions exported in efi memmap will be added to memblock.memory
> >> >> in (u)efi_init() and then trimmed down to the exact range specified as
> >> >> usable-memory-range by fdt_enforce_memory_region().
> >> >>
> >> >> Now I noticed that the current fdt_enforce_memory_region() may not work well
> >> >> with multiple entries in usable-memory-range.
> >> >>
> >> >
> >> > In any case, the root of the problem is that memory regions lose their
> >> > 'memory' annotation due to the way the memory map is mangled before
> >> > being supplied to the kexec kernel.
> >> >
> >> > Would it be possible to classify all memory that we want to hide from
> >> > the kexec kernel as NOMAP instead? That way, it will not be mapped
> >> > implicitly, but will still be mapped cacheable by acpi_os_ioremap(),
> >> > so this seems to be the most appropriate way to deal with the host
> >> > kernel's memory contents.
> >>
> >> Hmm. wouldn't appending the acpi reclaim regions to
> >> 'linux,usable-memory-range' in the dtb being passed to the crashkernel
> >> be better? Because its indirectly achieving a similar objective
> >> (although may be a subset of all System RAM regions on the primary
> >> kernel's memory).
> >>
> >> I am not aware of the background about the current kexec-tools
> >> implementation where we add only the crashkernel range to the dtb
> >> being passed to the crashkernel.
> >>
> >> Probably Akashi can answer better, as to how we arrived at this design
> >> approach and why we didn't want to expose all System RAM regions (i.e.
> >> ! NOMPAP regions) to the crashkernel.
> >>
> >> I am suspecting that some issues were seen/meet when the System RAM (!
> >> NOMAP regions) were exposed to the crashkernel, and that's why we
> >> finalized on this design approach, but this is something which is just
> >> my guess.
> >>
> >> Regards,
> >> Bhupesh
> >>
> >> >>> >
> >> >>> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >> >>> > via a kernel command line parameter, "memmap=".
> >> >>>
> >> >>> memmap= is only used in old kexec-tools, now we are passing them via
> >> >>> e820 table.
> >> >>
> >> >> Thanks. I remember that you have explained it before.
> >> >>
> >> >> -Takahiro AKASHI
> >> >>
> >> >>> [snip]
> >> >>>
> >> >>> Thanks
> >> >>> Dave
> >
> > ===8<==
> > From 74e2451fea83d546feae76160ba7de426913fe03 Mon Sep 17 00:00:00 2001
> > From: AKASHI Takahiro <takahiro.akashi at linaro.org>
> > Date: Thu, 21 Dec 2017 19:14:23 +0900
> > Subject: [PATCH] arm64: kdump: mark unusable memory as NOMAP
> >
> > ---
> >  arch/arm64/mm/init.c | 10 ++++++++--
> >  1 file changed, 8 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > index 00e7b900ca41..8175db94257b 100644
> > --- a/arch/arm64/mm/init.c
> > +++ b/arch/arm64/mm/init.c
> > @@ -352,11 +352,17 @@ static void __init fdt_enforce_memory_region(void)
> >         struct memblock_region reg = {
> >                 .size = 0,
> >         };
> > +       u64 idx;
> > +       phys_addr_t start, end;
> >
> >         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >
> > -       if (reg.size)
> > -               memblock_cap_memory_range(reg.base, reg.size);
> > +       if (reg.size) {
> > +               for_each_free_mem_range(idx, NUMA_NO_NODE, MEMBLOCK_NONE,
> > +                                       &start, &end, NULL)
> > +                       memblock_mark_nomap(start, end - start);
> > +               memblock_clear_nomap(reg.base, reg.size);
> > +       }
> >  }
> >
> >  void __init arm64_memblock_init(void)
> > --
> > 2.15.1
> >
> 
> Thanks for the patch. After applying this on top of
> 4.15.0-rc4-next-20171220, there seems to be a improvement and the
> crashkernel boot no longer hangs while trying to access the acpi
> tables.
> 
> However I notice a minor issue. Please see the log below for
> reference, the following message keeps spamming the console but I see
> the crashkernel boot proceed further.:
> 
> [    0.000000] ACPI: NUMA: SRAT: PXM 3 -> MPIDR 0x70303 -> Node 3
> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff]
> [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x2000000000-0x2fffffffff]
> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0x1fffffffff]
> [    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xa000000000-0xafffffffff]
> [    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x9000000000-0x9fffffffff]
> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe200-0x1ffbffffff]
> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffc400-0x1ffbffe1ff]
> [    0.000000] NUMA: NODE_DATA(1) on node 0
> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffa600-0x1ffbffc3ff]
> [    0.000000] NUMA: NODE_DATA(2) on node 0
> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbff8800-0x1ffbffa5ff]
> [    0.000000] NUMA: NODE_DATA(3) on node 0
> [    0.000000] [ffff7fe008000000-ffff7fe00800ffff] potential offnode
> page_structs
> [    0.000000] [ffff7fe008010000-ffff7fe00801ffff] potential offnode
> page_structs
> [    0.000000] [ffff7fe008020000-ffff7fe00802ffff] potential offnode
> page_structs
> [    0.000000] [ffff7fe008030000-ffff7fe00803ffff] potential offnode
> page_structs
> [    0.000000] [ffff7fe008040000-ffff7fe00804ffff] potential offnode
> page_structs
> [    0.000000] [ffff7fe008050000-ffff7fe00805ffff] potential offnode
> page_structs
> 
> [snip..]
> [    0.000000] [ffff7fe0081f0000-ffff7fe0081fffff] potential offnode
> page_structs

These messages shows that some "struct page" data are allocated on remote
(numa) nodes.
Since on your crash dump kernel, all the usable system memory (starting
0x0e800000) belongs to Node#0, we can't avoid such non-local allocations.

In my best guess, you can ingore them except for some performance penality.
This may be one side-effect.

So does your crash dump kernel now boot successfully?

Thanks,
-Takahiro AKASHI

> This WARNING message seems to come from vmemmap_verify() inside
> 'mm/sparse-vmemmap.c'
> 
> Regards,
> Bhupesh


From bhsharma at redhat.com  Sat Dec 23 11:51:02 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Sun, 24 Dec 2017 01:21:02 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171222083308.cy3vwlyhig22q5ee@fireball>
References: <CAKv+Gu_BmFN9Zg861SCS+R=V4khFykjuOzkmfEknsL=NvWW3Eg@mail.gmail.com>
 <20171213121605.GE28046@linaro.org>
 <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
 <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
Message-ID: <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>

On Fri, Dec 22, 2017 at 2:03 PM, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
> On Thu, Dec 21, 2017 at 05:36:30PM +0530, Bhupesh Sharma wrote:
>> Hello Akashi,
>>
>> On Thu, Dec 21, 2017 at 4:04 PM, AKASHI Takahiro
>> <takahiro.akashi at linaro.org> wrote:
>> > Bhupesh,
>> >
>> > Can you test the patch attached below, please?
>> >
>> > It is intended to retain already-reserved regions (ACPI reclaim memory
>> > in this case) in system ram (i.e. memblock.memory) without explicitly
>> > exporting them via usable-memory-range.
>> > (I still have to figure out what the side-effect of this patch is.)
>> >
>> > Thanks,
>> > -Takahiro AKASHI
>> >
>> > On Thu, Dec 21, 2017 at 01:30:43AM +0530, Bhupesh Sharma wrote:
>> >> On Tue, Dec 19, 2017 at 6:39 PM, Ard Biesheuvel
>> >> <ard.biesheuvel at linaro.org> wrote:
>> >> > On 19 December 2017 at 07:09, AKASHI Takahiro
>> >> > <takahiro.akashi at linaro.org> wrote:
>> >> >> On Mon, Dec 18, 2017 at 01:40:09PM +0800, Dave Young wrote:
>> >> >>> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
>> >> >>> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
>> >> >>> > > On 13 December 2017 at 12:16, AKASHI Takahiro
>> >> >>> > > <takahiro.akashi at linaro.org> wrote:
>> >> >>> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
>> >> >>> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
>> >> >>> > > >> <takahiro.akashi at linaro.org> wrote:
>> >> >>> > > >> > Bhupesh, Ard,
>> >> >>> > > >> >
>> >> >>> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
>> >> >>> > > >> >> Hi Ard, Akashi
>> >> >>> > > >> >>
>> >> >>> > > >> > (snip)
>> >> >>> > > >> >
>> >> >>> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
>> >> >>> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
>> >> >>> > > >> >> identify its own usable memory and exclude, at its boot time, any
>> >> >>> > > >> >> other memory areas that are part of the panicked kernel's memory.
>> >> >>> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
>> >> >>> > > >> >> , for details)
>> >> >>> > > >> >
>> >> >>> > > >> > Right.
>> >> >>> > > >> >
>> >> >>> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
>> >> >>> > > >> >> with the crashkernel memory range:
>> >> >>> > > >> >>
>> >> >>> > > >> >>                 /* add linux,usable-memory-range */
>> >> >>> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
>> >> >>> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
>> >> >>> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
>> >> >>> > > >> >>                                 address_cells, size_cells);
>> >> >>> > > >> >>
>> >> >>> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
>> >> >>> > > >> >> , for details)
>> >> >>> > > >> >>
>> >> >>> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
>> >> >>> > > >> >> they are marked as System RAM or as RESERVED. As,
>> >> >>> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
>> >> >>> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
>> >> >>> > > >> >>
>> >> >>> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
>> >> >>> > > >> >> ACPI memory and crashes while trying to access the same:
>> >> >>> > > >> >>
>> >> >>> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
>> >> >>> > > >> >> -r`.img --reuse-cmdline -d
>> >> >>> > > >> >>
>> >> >>> > > >> >> [snip..]
>> >> >>> > > >> >>
>> >> >>> > > >> >> Reserved memory range
>> >> >>> > > >> >> 000000000e800000-000000002e7fffff (0)
>> >> >>> > > >> >>
>> >> >>> > > >> >> Coredump memory ranges
>> >> >>> > > >> >> 0000000000000000-000000000e7fffff (0)
>> >> >>> > > >> >> 000000002e800000-000000003961ffff (0)
>> >> >>> > > >> >> 0000000039d40000-000000003ed2ffff (0)
>> >> >>> > > >> >> 000000003ed60000-000000003fbfffff (0)
>> >> >>> > > >> >> 0000001040000000-0000001ffbffffff (0)
>> >> >>> > > >> >> 0000002000000000-0000002ffbffffff (0)
>> >> >>> > > >> >> 0000009000000000-0000009ffbffffff (0)
>> >> >>> > > >> >> 000000a000000000-000000affbffffff (0)
>> >> >>> > > >> >>
>> >> >>> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
>> >> >>> > > >> >> memory cap'ing passed to the crash kernel inside
>> >> >>> > > >> >> 'arch/arm64/mm/init.c' (see below):
>> >> >>> > > >> >>
>> >> >>> > > >> >> static void __init fdt_enforce_memory_region(void)
>> >> >>> > > >> >> {
>> >> >>> > > >> >>         struct memblock_region reg = {
>> >> >>> > > >> >>                 .size = 0,
>> >> >>> > > >> >>         };
>> >> >>> > > >> >>
>> >> >>> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> >> >>> > > >> >>
>> >> >>> > > >> >>         if (reg.size)
>> >> >>> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
>> >> >>> > > >> >> comment this out */
>> >> >>> > > >> >> }
>> >> >>> > > >> >
>> >> >>> > > >> > Please just don't do that. It can cause a fatal damage on
>> >> >>> > > >> > memory contents of the *crashed* kernel.
>> >> >>> > > >> >
>> >> >>> > > >> >> 5). Both the above temporary solutions fix the problem.
>> >> >>> > > >> >>
>> >> >>> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
>> >> >>> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
>> >> >>> > > >> >> fail.
>> >> >>> > > >> >>
>> >> >>> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
>> >> >>> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
>> >> >>> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
>> >> >>> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
>> >> >>> > > >> >> dt node 'linux,usable-memory-range'
>> >> >>> > > >> >
>> >> >>> > > >> > I still don't understand why we need to carry over the information
>> >> >>> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
>> >> >>> > > >> > such regions are free to be reused by the kernel after some point of
>> >> >>> > > >> > initialization. Why does crash dump kernel need to know about them?
>> >> >>> > > >> >
>> >> >>> > > >>
>> >> >>> > > >> Not really. According to the UEFI spec, they can be reclaimed after
>> >> >>> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
>> >> >>> > > >> no longer needs them. Of course, in order to be able to boot a kexec
>> >> >>> > > >> kernel, those regions needs to be preserved, which is why they are
>> >> >>> > > >> memblock_reserve()'d now.
>> >> >>> > > >
>> >> >>> > > > For my better understandings, who is actually accessing such regions
>> >> >>> > > > during boot time, uefi itself or efistub?
>> >> >>> > > >
>> >> >>> > >
>> >> >>> > > No, only the kernel. This is where the ACPI tables are stored. For
>> >> >>> > > instance, on QEMU we have
>> >> >>> > >
>> >> >>> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
>> >> >>> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
>> >> >>> > >   01000013)
>> >> >>> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
>> >> >>> > > BXPC 00000001)
>> >> >>> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
>> >> >>> > > BXPC 00000001)
>> >> >>> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
>> >> >>> > > BXPC 00000001)
>> >> >>> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
>> >> >>> > > BXPC 00000001)
>> >> >>> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
>> >> >>> > > BXPC 00000001)
>> >> >>> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
>> >> >>> > > BXPC 00000001)
>> >> >>> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
>> >> >>> > > BXPC 00000001)
>> >> >>> > >
>> >> >>> > > covered by
>> >> >>> > >
>> >> >>> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
>> >> >>> > >  ...
>> >> >>> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
>> >> >>> >
>> >> >>> > OK. I mistakenly understood those regions could be freed after exiting
>> >> >>> > UEFI boot services.
>> >> >>> >
>> >> >>> > >
>> >> >>> > > >> So it seems that kexec does not honour the memblock_reserve() table
>> >> >>> > > >> when booting the next kernel.
>> >> >>> > > >
>> >> >>> > > > not really.
>> >> >>> > > >
>> >> >>> > > >> > (In other words, can or should we skip some part of ACPI-related init code
>> >> >>> > > >> > on crash dump kernel?)
>> >> >>> > > >> >
>> >> >>> > > >>
>> >> >>> > > >> I don't think so. And the change to the handling of ACPI reclaim
>> >> >>> > > >> regions only revealed the bug, not created it (given that other
>> >> >>> > > >> memblock_reserve regions may be affected as well)
>> >> >>> > > >
>> >> >>> > > > As whether we should honor such reserved regions over kexec'ing
>> >> >>> > > > depends on each one's specific nature, we will have to take care one-by-one.
>> >> >>> > > > As a matter of fact, no information about "reserved" memblocks is
>> >> >>> > > > exposed to user space (via proc/iomem).
>> >> >>> > > >
>> >> >>> > >
>> >> >>> > > That is why I suggested (somewhere in this thread?) to not expose them
>> >> >>> > > as 'System RAM'. Do you think that could solve this?
>> >> >>> >
>> >> >>> > Memblock-reserv'ing them is necessary to prevent their corruption and
>> >> >>> > marking them under another name in /proc/iomem would also be good in order
>> >> >>> > not to allocate them as part of crash kernel's memory.
>> >> >>> >
>> >> >>> > But I'm not still convinced that we should export them in useable-
>> >> >>> > memory-range to crash dump kernel. They will be accessed through
>> >> >>> > acpi_os_map_memory() and so won't be required to be part of system ram
>> >> >>> > (or memblocks), I guess.
>> >> >>> >     -> Bhupesh?
>> >> >>>
>> >> >>> I forgot how arm64 kernel retrieve the memory ranges and initialize
>> >> >>> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
>> >> >>> the memory according to the efi memmap?  For kdump kernel anything other
>> >> >>> than usable memory (which is from the dt node instead) should be
>> >> >>> reinitialized according to efi passed info, no?
>> >> >>
>> >> >> All the regions exported in efi memmap will be added to memblock.memory
>> >> >> in (u)efi_init() and then trimmed down to the exact range specified as
>> >> >> usable-memory-range by fdt_enforce_memory_region().
>> >> >>
>> >> >> Now I noticed that the current fdt_enforce_memory_region() may not work well
>> >> >> with multiple entries in usable-memory-range.
>> >> >>
>> >> >
>> >> > In any case, the root of the problem is that memory regions lose their
>> >> > 'memory' annotation due to the way the memory map is mangled before
>> >> > being supplied to the kexec kernel.
>> >> >
>> >> > Would it be possible to classify all memory that we want to hide from
>> >> > the kexec kernel as NOMAP instead? That way, it will not be mapped
>> >> > implicitly, but will still be mapped cacheable by acpi_os_ioremap(),
>> >> > so this seems to be the most appropriate way to deal with the host
>> >> > kernel's memory contents.
>> >>
>> >> Hmm. wouldn't appending the acpi reclaim regions to
>> >> 'linux,usable-memory-range' in the dtb being passed to the crashkernel
>> >> be better? Because its indirectly achieving a similar objective
>> >> (although may be a subset of all System RAM regions on the primary
>> >> kernel's memory).
>> >>
>> >> I am not aware of the background about the current kexec-tools
>> >> implementation where we add only the crashkernel range to the dtb
>> >> being passed to the crashkernel.
>> >>
>> >> Probably Akashi can answer better, as to how we arrived at this design
>> >> approach and why we didn't want to expose all System RAM regions (i.e.
>> >> ! NOMPAP regions) to the crashkernel.
>> >>
>> >> I am suspecting that some issues were seen/meet when the System RAM (!
>> >> NOMAP regions) were exposed to the crashkernel, and that's why we
>> >> finalized on this design approach, but this is something which is just
>> >> my guess.
>> >>
>> >> Regards,
>> >> Bhupesh
>> >>
>> >> >>> >
>> >> >>> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
>> >> >>> > via a kernel command line parameter, "memmap=".
>> >> >>>
>> >> >>> memmap= is only used in old kexec-tools, now we are passing them via
>> >> >>> e820 table.
>> >> >>
>> >> >> Thanks. I remember that you have explained it before.
>> >> >>
>> >> >> -Takahiro AKASHI
>> >> >>
>> >> >>> [snip]
>> >> >>>
>> >> >>> Thanks
>> >> >>> Dave
>> >
>> > ===8<==
>> > From 74e2451fea83d546feae76160ba7de426913fe03 Mon Sep 17 00:00:00 2001
>> > From: AKASHI Takahiro <takahiro.akashi at linaro.org>
>> > Date: Thu, 21 Dec 2017 19:14:23 +0900
>> > Subject: [PATCH] arm64: kdump: mark unusable memory as NOMAP
>> >
>> > ---
>> >  arch/arm64/mm/init.c | 10 ++++++++--
>> >  1 file changed, 8 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
>> > index 00e7b900ca41..8175db94257b 100644
>> > --- a/arch/arm64/mm/init.c
>> > +++ b/arch/arm64/mm/init.c
>> > @@ -352,11 +352,17 @@ static void __init fdt_enforce_memory_region(void)
>> >         struct memblock_region reg = {
>> >                 .size = 0,
>> >         };
>> > +       u64 idx;
>> > +       phys_addr_t start, end;
>> >
>> >         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> >
>> > -       if (reg.size)
>> > -               memblock_cap_memory_range(reg.base, reg.size);
>> > +       if (reg.size) {
>> > +               for_each_free_mem_range(idx, NUMA_NO_NODE, MEMBLOCK_NONE,
>> > +                                       &start, &end, NULL)
>> > +                       memblock_mark_nomap(start, end - start);
>> > +               memblock_clear_nomap(reg.base, reg.size);
>> > +       }
>> >  }
>> >
>> >  void __init arm64_memblock_init(void)
>> > --
>> > 2.15.1
>> >
>>
>> Thanks for the patch. After applying this on top of
>> 4.15.0-rc4-next-20171220, there seems to be a improvement and the
>> crashkernel boot no longer hangs while trying to access the acpi
>> tables.
>>
>> However I notice a minor issue. Please see the log below for
>> reference, the following message keeps spamming the console but I see
>> the crashkernel boot proceed further.:
>>
>> [    0.000000] ACPI: NUMA: SRAT: PXM 3 -> MPIDR 0x70303 -> Node 3
>> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff]
>> [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x2000000000-0x2fffffffff]
>> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0x1fffffffff]
>> [    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xa000000000-0xafffffffff]
>> [    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x9000000000-0x9fffffffff]
>> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe200-0x1ffbffffff]
>> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffc400-0x1ffbffe1ff]
>> [    0.000000] NUMA: NODE_DATA(1) on node 0
>> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffa600-0x1ffbffc3ff]
>> [    0.000000] NUMA: NODE_DATA(2) on node 0
>> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbff8800-0x1ffbffa5ff]
>> [    0.000000] NUMA: NODE_DATA(3) on node 0
>> [    0.000000] [ffff7fe008000000-ffff7fe00800ffff] potential offnode
>> page_structs
>> [    0.000000] [ffff7fe008010000-ffff7fe00801ffff] potential offnode
>> page_structs
>> [    0.000000] [ffff7fe008020000-ffff7fe00802ffff] potential offnode
>> page_structs
>> [    0.000000] [ffff7fe008030000-ffff7fe00803ffff] potential offnode
>> page_structs
>> [    0.000000] [ffff7fe008040000-ffff7fe00804ffff] potential offnode
>> page_structs
>> [    0.000000] [ffff7fe008050000-ffff7fe00805ffff] potential offnode
>> page_structs
>>
>> [snip..]
>> [    0.000000] [ffff7fe0081f0000-ffff7fe0081fffff] potential offnode
>> page_structs
>
> These messages shows that some "struct page" data are allocated on remote
> (numa) nodes.
> Since on your crash dump kernel, all the usable system memory (starting
> 0x0e800000) belongs to Node#0, we can't avoid such non-local allocations.
>
> In my best guess, you can ingore them except for some performance penality.
> This may be one side-effect.
>
> So does your crash dump kernel now boot successfully?
>

Indeed. The crash dump kernel now boots successfully and the crash
dump core can be saved properly as well (I tried saving it to local
disk).

However, the 'potential offnode page_structs' WARN messages hog the
console and delay crashkernel boot for a significant duration, which
can be irritating.

Can we also consider ratelimiting this WARNING message [which seems to
come from vmemmap_verify()] if invoked in the context of crash kernel,
in addition to making the above change suggested by  you.

Thanks for the help.

Regards,
Bhupesh


From takahiro.akashi at linaro.org  Sun Dec 24 19:25:02 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Mon, 25 Dec 2017 12:25:02 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
References: <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
 <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
Message-ID: <20171225032500.GA8877@linaro.org>

On Sun, Dec 24, 2017 at 01:21:02AM +0530, Bhupesh Sharma wrote:
> On Fri, Dec 22, 2017 at 2:03 PM, AKASHI Takahiro
> <takahiro.akashi at linaro.org> wrote:
> > On Thu, Dec 21, 2017 at 05:36:30PM +0530, Bhupesh Sharma wrote:
> >> Hello Akashi,
> >>
> >> On Thu, Dec 21, 2017 at 4:04 PM, AKASHI Takahiro
> >> <takahiro.akashi at linaro.org> wrote:
> >> > Bhupesh,
> >> >
> >> > Can you test the patch attached below, please?
> >> >
> >> > It is intended to retain already-reserved regions (ACPI reclaim memory
> >> > in this case) in system ram (i.e. memblock.memory) without explicitly
> >> > exporting them via usable-memory-range.
> >> > (I still have to figure out what the side-effect of this patch is.)
> >> >
> >> > Thanks,
> >> > -Takahiro AKASHI
> >> >
> >> > On Thu, Dec 21, 2017 at 01:30:43AM +0530, Bhupesh Sharma wrote:
> >> >> On Tue, Dec 19, 2017 at 6:39 PM, Ard Biesheuvel
> >> >> <ard.biesheuvel at linaro.org> wrote:
> >> >> > On 19 December 2017 at 07:09, AKASHI Takahiro
> >> >> > <takahiro.akashi at linaro.org> wrote:
> >> >> >> On Mon, Dec 18, 2017 at 01:40:09PM +0800, Dave Young wrote:
> >> >> >>> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
> >> >> >>> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> >> >> >>> > > On 13 December 2017 at 12:16, AKASHI Takahiro
> >> >> >>> > > <takahiro.akashi at linaro.org> wrote:
> >> >> >>> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> >> >> >>> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
> >> >> >>> > > >> <takahiro.akashi at linaro.org> wrote:
> >> >> >>> > > >> > Bhupesh, Ard,
> >> >> >>> > > >> >
> >> >> >>> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> >> >> >>> > > >> >> Hi Ard, Akashi
> >> >> >>> > > >> >>
> >> >> >>> > > >> > (snip)
> >> >> >>> > > >> >
> >> >> >>> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> >> >> >>> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> >> >> >>> > > >> >> identify its own usable memory and exclude, at its boot time, any
> >> >> >>> > > >> >> other memory areas that are part of the panicked kernel's memory.
> >> >> >>> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> >> >> >>> > > >> >> , for details)
> >> >> >>> > > >> >
> >> >> >>> > > >> > Right.
> >> >> >>> > > >> >
> >> >> >>> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> >> >> >>> > > >> >> with the crashkernel memory range:
> >> >> >>> > > >> >>
> >> >> >>> > > >> >>                 /* add linux,usable-memory-range */
> >> >> >>> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> >> >> >>> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> >> >> >>> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> >> >> >>> > > >> >>                                 address_cells, size_cells);
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> >> >> >>> > > >> >> , for details)
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> >> >> >>> > > >> >> they are marked as System RAM or as RESERVED. As,
> >> >> >>> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
> >> >> >>> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
> >> >> >>> > > >> >> ACPI memory and crashes while trying to access the same:
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> >> >> >>> > > >> >> -r`.img --reuse-cmdline -d
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> [snip..]
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> Reserved memory range
> >> >> >>> > > >> >> 000000000e800000-000000002e7fffff (0)
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> Coredump memory ranges
> >> >> >>> > > >> >> 0000000000000000-000000000e7fffff (0)
> >> >> >>> > > >> >> 000000002e800000-000000003961ffff (0)
> >> >> >>> > > >> >> 0000000039d40000-000000003ed2ffff (0)
> >> >> >>> > > >> >> 000000003ed60000-000000003fbfffff (0)
> >> >> >>> > > >> >> 0000001040000000-0000001ffbffffff (0)
> >> >> >>> > > >> >> 0000002000000000-0000002ffbffffff (0)
> >> >> >>> > > >> >> 0000009000000000-0000009ffbffffff (0)
> >> >> >>> > > >> >> 000000a000000000-000000affbffffff (0)
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> >> >> >>> > > >> >> memory cap'ing passed to the crash kernel inside
> >> >> >>> > > >> >> 'arch/arm64/mm/init.c' (see below):
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> static void __init fdt_enforce_memory_region(void)
> >> >> >>> > > >> >> {
> >> >> >>> > > >> >>         struct memblock_region reg = {
> >> >> >>> > > >> >>                 .size = 0,
> >> >> >>> > > >> >>         };
> >> >> >>> > > >> >>
> >> >> >>> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >> >> >>> > > >> >>
> >> >> >>> > > >> >>         if (reg.size)
> >> >> >>> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> >> >> >>> > > >> >> comment this out */
> >> >> >>> > > >> >> }
> >> >> >>> > > >> >
> >> >> >>> > > >> > Please just don't do that. It can cause a fatal damage on
> >> >> >>> > > >> > memory contents of the *crashed* kernel.
> >> >> >>> > > >> >
> >> >> >>> > > >> >> 5). Both the above temporary solutions fix the problem.
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
> >> >> >>> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> >> >> >>> > > >> >> fail.
> >> >> >>> > > >> >>
> >> >> >>> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> >> >> >>> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> >> >> >>> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> >> >> >>> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> >> >> >>> > > >> >> dt node 'linux,usable-memory-range'
> >> >> >>> > > >> >
> >> >> >>> > > >> > I still don't understand why we need to carry over the information
> >> >> >>> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> >> >> >>> > > >> > such regions are free to be reused by the kernel after some point of
> >> >> >>> > > >> > initialization. Why does crash dump kernel need to know about them?
> >> >> >>> > > >> >
> >> >> >>> > > >>
> >> >> >>> > > >> Not really. According to the UEFI spec, they can be reclaimed after
> >> >> >>> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> >> >> >>> > > >> no longer needs them. Of course, in order to be able to boot a kexec
> >> >> >>> > > >> kernel, those regions needs to be preserved, which is why they are
> >> >> >>> > > >> memblock_reserve()'d now.
> >> >> >>> > > >
> >> >> >>> > > > For my better understandings, who is actually accessing such regions
> >> >> >>> > > > during boot time, uefi itself or efistub?
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> > > No, only the kernel. This is where the ACPI tables are stored. For
> >> >> >>> > > instance, on QEMU we have
> >> >> >>> > >
> >> >> >>> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> >> >> >>> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> >> >> >>> > >   01000013)
> >> >> >>> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> >> >> >>> > > BXPC 00000001)
> >> >> >>> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> >> >> >>> > > BXPC 00000001)
> >> >> >>> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> >> >> >>> > > BXPC 00000001)
> >> >> >>> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> >> >> >>> > > BXPC 00000001)
> >> >> >>> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> >> >> >>> > > BXPC 00000001)
> >> >> >>> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> >> >> >>> > > BXPC 00000001)
> >> >> >>> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> >> >> >>> > > BXPC 00000001)
> >> >> >>> > >
> >> >> >>> > > covered by
> >> >> >>> > >
> >> >> >>> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> >> >> >>> > >  ...
> >> >> >>> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> >> >> >>> >
> >> >> >>> > OK. I mistakenly understood those regions could be freed after exiting
> >> >> >>> > UEFI boot services.
> >> >> >>> >
> >> >> >>> > >
> >> >> >>> > > >> So it seems that kexec does not honour the memblock_reserve() table
> >> >> >>> > > >> when booting the next kernel.
> >> >> >>> > > >
> >> >> >>> > > > not really.
> >> >> >>> > > >
> >> >> >>> > > >> > (In other words, can or should we skip some part of ACPI-related init code
> >> >> >>> > > >> > on crash dump kernel?)
> >> >> >>> > > >> >
> >> >> >>> > > >>
> >> >> >>> > > >> I don't think so. And the change to the handling of ACPI reclaim
> >> >> >>> > > >> regions only revealed the bug, not created it (given that other
> >> >> >>> > > >> memblock_reserve regions may be affected as well)
> >> >> >>> > > >
> >> >> >>> > > > As whether we should honor such reserved regions over kexec'ing
> >> >> >>> > > > depends on each one's specific nature, we will have to take care one-by-one.
> >> >> >>> > > > As a matter of fact, no information about "reserved" memblocks is
> >> >> >>> > > > exposed to user space (via proc/iomem).
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> > > That is why I suggested (somewhere in this thread?) to not expose them
> >> >> >>> > > as 'System RAM'. Do you think that could solve this?
> >> >> >>> >
> >> >> >>> > Memblock-reserv'ing them is necessary to prevent their corruption and
> >> >> >>> > marking them under another name in /proc/iomem would also be good in order
> >> >> >>> > not to allocate them as part of crash kernel's memory.
> >> >> >>> >
> >> >> >>> > But I'm not still convinced that we should export them in useable-
> >> >> >>> > memory-range to crash dump kernel. They will be accessed through
> >> >> >>> > acpi_os_map_memory() and so won't be required to be part of system ram
> >> >> >>> > (or memblocks), I guess.
> >> >> >>> >     -> Bhupesh?
> >> >> >>>
> >> >> >>> I forgot how arm64 kernel retrieve the memory ranges and initialize
> >> >> >>> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
> >> >> >>> the memory according to the efi memmap?  For kdump kernel anything other
> >> >> >>> than usable memory (which is from the dt node instead) should be
> >> >> >>> reinitialized according to efi passed info, no?
> >> >> >>
> >> >> >> All the regions exported in efi memmap will be added to memblock.memory
> >> >> >> in (u)efi_init() and then trimmed down to the exact range specified as
> >> >> >> usable-memory-range by fdt_enforce_memory_region().
> >> >> >>
> >> >> >> Now I noticed that the current fdt_enforce_memory_region() may not work well
> >> >> >> with multiple entries in usable-memory-range.
> >> >> >>
> >> >> >
> >> >> > In any case, the root of the problem is that memory regions lose their
> >> >> > 'memory' annotation due to the way the memory map is mangled before
> >> >> > being supplied to the kexec kernel.
> >> >> >
> >> >> > Would it be possible to classify all memory that we want to hide from
> >> >> > the kexec kernel as NOMAP instead? That way, it will not be mapped
> >> >> > implicitly, but will still be mapped cacheable by acpi_os_ioremap(),
> >> >> > so this seems to be the most appropriate way to deal with the host
> >> >> > kernel's memory contents.
> >> >>
> >> >> Hmm. wouldn't appending the acpi reclaim regions to
> >> >> 'linux,usable-memory-range' in the dtb being passed to the crashkernel
> >> >> be better? Because its indirectly achieving a similar objective
> >> >> (although may be a subset of all System RAM regions on the primary
> >> >> kernel's memory).
> >> >>
> >> >> I am not aware of the background about the current kexec-tools
> >> >> implementation where we add only the crashkernel range to the dtb
> >> >> being passed to the crashkernel.
> >> >>
> >> >> Probably Akashi can answer better, as to how we arrived at this design
> >> >> approach and why we didn't want to expose all System RAM regions (i.e.
> >> >> ! NOMPAP regions) to the crashkernel.
> >> >>
> >> >> I am suspecting that some issues were seen/meet when the System RAM (!
> >> >> NOMAP regions) were exposed to the crashkernel, and that's why we
> >> >> finalized on this design approach, but this is something which is just
> >> >> my guess.
> >> >>
> >> >> Regards,
> >> >> Bhupesh
> >> >>
> >> >> >>> >
> >> >> >>> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >> >> >>> > via a kernel command line parameter, "memmap=".
> >> >> >>>
> >> >> >>> memmap= is only used in old kexec-tools, now we are passing them via
> >> >> >>> e820 table.
> >> >> >>
> >> >> >> Thanks. I remember that you have explained it before.
> >> >> >>
> >> >> >> -Takahiro AKASHI
> >> >> >>
> >> >> >>> [snip]
> >> >> >>>
> >> >> >>> Thanks
> >> >> >>> Dave
> >> >
> >> > ===8<==
> >> > From 74e2451fea83d546feae76160ba7de426913fe03 Mon Sep 17 00:00:00 2001
> >> > From: AKASHI Takahiro <takahiro.akashi at linaro.org>
> >> > Date: Thu, 21 Dec 2017 19:14:23 +0900
> >> > Subject: [PATCH] arm64: kdump: mark unusable memory as NOMAP
> >> >
> >> > ---
> >> >  arch/arm64/mm/init.c | 10 ++++++++--
> >> >  1 file changed, 8 insertions(+), 2 deletions(-)
> >> >
> >> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> >> > index 00e7b900ca41..8175db94257b 100644
> >> > --- a/arch/arm64/mm/init.c
> >> > +++ b/arch/arm64/mm/init.c
> >> > @@ -352,11 +352,17 @@ static void __init fdt_enforce_memory_region(void)
> >> >         struct memblock_region reg = {
> >> >                 .size = 0,
> >> >         };
> >> > +       u64 idx;
> >> > +       phys_addr_t start, end;
> >> >
> >> >         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >> >
> >> > -       if (reg.size)
> >> > -               memblock_cap_memory_range(reg.base, reg.size);
> >> > +       if (reg.size) {
> >> > +               for_each_free_mem_range(idx, NUMA_NO_NODE, MEMBLOCK_NONE,
> >> > +                                       &start, &end, NULL)
> >> > +                       memblock_mark_nomap(start, end - start);
> >> > +               memblock_clear_nomap(reg.base, reg.size);
> >> > +       }
> >> >  }
> >> >
> >> >  void __init arm64_memblock_init(void)
> >> > --
> >> > 2.15.1
> >> >
> >>
> >> Thanks for the patch. After applying this on top of
> >> 4.15.0-rc4-next-20171220, there seems to be a improvement and the
> >> crashkernel boot no longer hangs while trying to access the acpi
> >> tables.
> >>
> >> However I notice a minor issue. Please see the log below for
> >> reference, the following message keeps spamming the console but I see
> >> the crashkernel boot proceed further.:
> >>
> >> [    0.000000] ACPI: NUMA: SRAT: PXM 3 -> MPIDR 0x70303 -> Node 3
> >> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff]
> >> [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x2000000000-0x2fffffffff]
> >> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0x1fffffffff]
> >> [    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xa000000000-0xafffffffff]
> >> [    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x9000000000-0x9fffffffff]
> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe200-0x1ffbffffff]
> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffc400-0x1ffbffe1ff]
> >> [    0.000000] NUMA: NODE_DATA(1) on node 0
> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffa600-0x1ffbffc3ff]
> >> [    0.000000] NUMA: NODE_DATA(2) on node 0
> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbff8800-0x1ffbffa5ff]
> >> [    0.000000] NUMA: NODE_DATA(3) on node 0
> >> [    0.000000] [ffff7fe008000000-ffff7fe00800ffff] potential offnode
> >> page_structs
> >> [    0.000000] [ffff7fe008010000-ffff7fe00801ffff] potential offnode
> >> page_structs
> >> [    0.000000] [ffff7fe008020000-ffff7fe00802ffff] potential offnode
> >> page_structs
> >> [    0.000000] [ffff7fe008030000-ffff7fe00803ffff] potential offnode
> >> page_structs
> >> [    0.000000] [ffff7fe008040000-ffff7fe00804ffff] potential offnode
> >> page_structs
> >> [    0.000000] [ffff7fe008050000-ffff7fe00805ffff] potential offnode
> >> page_structs
> >>
> >> [snip..]
> >> [    0.000000] [ffff7fe0081f0000-ffff7fe0081fffff] potential offnode
> >> page_structs
> >
> > These messages shows that some "struct page" data are allocated on remote
> > (numa) nodes.
> > Since on your crash dump kernel, all the usable system memory (starting
> > 0x0e800000) belongs to Node#0, we can't avoid such non-local allocations.
> >
> > In my best guess, you can ingore them except for some performance penality.
> > This may be one side-effect.
> >
> > So does your crash dump kernel now boot successfully?
> >
> 
> Indeed. The crash dump kernel now boots successfully and the crash
> dump core can be saved properly as well (I tried saving it to local
> disk).

Thank you for the confirmation.
(I'd like to suggest you to examine the core dump with crash utility.)

> However, the 'potential offnode page_structs' WARN messages hog the
> console and delay crashkernel boot for a significant duration, which
> can be irritating.
> 
> Can we also consider ratelimiting this WARNING message [which seems to
> come from vmemmap_verify()] if invoked in the context of crash kernel,
> in addition to making the above change suggested by  you.

Well, we may be able to change pr_warn() to pr_warn_once() here, but
I hope that adding "numa=off" to kernel command line should also work.

Thanks,
-Takahiro AKASHI


> Thanks for the help.
> 
> Regards,
> Bhupesh


From bhsharma at redhat.com  Mon Dec 25 12:14:41 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Tue, 26 Dec 2017 01:44:41 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171225032500.GA8877@linaro.org>
References: <CAKv+Gu_G8kBEAdAznVauZVAdJOFkr1vmu0Gf6tOwJfH2CgdufA@mail.gmail.com>
 <20171215085924.sqlcwm4copzba5ag@fireball>
 <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
 <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
 <20171225032500.GA8877@linaro.org>
Message-ID: <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>

On Mon, Dec 25, 2017 at 8:55 AM, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
> On Sun, Dec 24, 2017 at 01:21:02AM +0530, Bhupesh Sharma wrote:
>> On Fri, Dec 22, 2017 at 2:03 PM, AKASHI Takahiro
>> <takahiro.akashi at linaro.org> wrote:
>> > On Thu, Dec 21, 2017 at 05:36:30PM +0530, Bhupesh Sharma wrote:
>> >> Hello Akashi,
>> >>
>> >> On Thu, Dec 21, 2017 at 4:04 PM, AKASHI Takahiro
>> >> <takahiro.akashi at linaro.org> wrote:
>> >> > Bhupesh,
>> >> >
>> >> > Can you test the patch attached below, please?
>> >> >
>> >> > It is intended to retain already-reserved regions (ACPI reclaim memory
>> >> > in this case) in system ram (i.e. memblock.memory) without explicitly
>> >> > exporting them via usable-memory-range.
>> >> > (I still have to figure out what the side-effect of this patch is.)
>> >> >
>> >> > Thanks,
>> >> > -Takahiro AKASHI
>> >> >
>> >> > On Thu, Dec 21, 2017 at 01:30:43AM +0530, Bhupesh Sharma wrote:
>> >> >> On Tue, Dec 19, 2017 at 6:39 PM, Ard Biesheuvel
>> >> >> <ard.biesheuvel at linaro.org> wrote:
>> >> >> > On 19 December 2017 at 07:09, AKASHI Takahiro
>> >> >> > <takahiro.akashi at linaro.org> wrote:
>> >> >> >> On Mon, Dec 18, 2017 at 01:40:09PM +0800, Dave Young wrote:
>> >> >> >>> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
>> >> >> >>> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
>> >> >> >>> > > On 13 December 2017 at 12:16, AKASHI Takahiro
>> >> >> >>> > > <takahiro.akashi at linaro.org> wrote:
>> >> >> >>> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
>> >> >> >>> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
>> >> >> >>> > > >> <takahiro.akashi at linaro.org> wrote:
>> >> >> >>> > > >> > Bhupesh, Ard,
>> >> >> >>> > > >> >
>> >> >> >>> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
>> >> >> >>> > > >> >> Hi Ard, Akashi
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> > (snip)
>> >> >> >>> > > >> >
>> >> >> >>> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
>> >> >> >>> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
>> >> >> >>> > > >> >> identify its own usable memory and exclude, at its boot time, any
>> >> >> >>> > > >> >> other memory areas that are part of the panicked kernel's memory.
>> >> >> >>> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
>> >> >> >>> > > >> >> , for details)
>> >> >> >>> > > >> >
>> >> >> >>> > > >> > Right.
>> >> >> >>> > > >> >
>> >> >> >>> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
>> >> >> >>> > > >> >> with the crashkernel memory range:
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >>                 /* add linux,usable-memory-range */
>> >> >> >>> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
>> >> >> >>> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
>> >> >> >>> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
>> >> >> >>> > > >> >>                                 address_cells, size_cells);
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
>> >> >> >>> > > >> >> , for details)
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
>> >> >> >>> > > >> >> they are marked as System RAM or as RESERVED. As,
>> >> >> >>> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
>> >> >> >>> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
>> >> >> >>> > > >> >> ACPI memory and crashes while trying to access the same:
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
>> >> >> >>> > > >> >> -r`.img --reuse-cmdline -d
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> [snip..]
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> Reserved memory range
>> >> >> >>> > > >> >> 000000000e800000-000000002e7fffff (0)
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> Coredump memory ranges
>> >> >> >>> > > >> >> 0000000000000000-000000000e7fffff (0)
>> >> >> >>> > > >> >> 000000002e800000-000000003961ffff (0)
>> >> >> >>> > > >> >> 0000000039d40000-000000003ed2ffff (0)
>> >> >> >>> > > >> >> 000000003ed60000-000000003fbfffff (0)
>> >> >> >>> > > >> >> 0000001040000000-0000001ffbffffff (0)
>> >> >> >>> > > >> >> 0000002000000000-0000002ffbffffff (0)
>> >> >> >>> > > >> >> 0000009000000000-0000009ffbffffff (0)
>> >> >> >>> > > >> >> 000000a000000000-000000affbffffff (0)
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
>> >> >> >>> > > >> >> memory cap'ing passed to the crash kernel inside
>> >> >> >>> > > >> >> 'arch/arm64/mm/init.c' (see below):
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> static void __init fdt_enforce_memory_region(void)
>> >> >> >>> > > >> >> {
>> >> >> >>> > > >> >>         struct memblock_region reg = {
>> >> >> >>> > > >> >>                 .size = 0,
>> >> >> >>> > > >> >>         };
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >>         if (reg.size)
>> >> >> >>> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
>> >> >> >>> > > >> >> comment this out */
>> >> >> >>> > > >> >> }
>> >> >> >>> > > >> >
>> >> >> >>> > > >> > Please just don't do that. It can cause a fatal damage on
>> >> >> >>> > > >> > memory contents of the *crashed* kernel.
>> >> >> >>> > > >> >
>> >> >> >>> > > >> >> 5). Both the above temporary solutions fix the problem.
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
>> >> >> >>> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
>> >> >> >>> > > >> >> fail.
>> >> >> >>> > > >> >>
>> >> >> >>> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
>> >> >> >>> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
>> >> >> >>> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
>> >> >> >>> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
>> >> >> >>> > > >> >> dt node 'linux,usable-memory-range'
>> >> >> >>> > > >> >
>> >> >> >>> > > >> > I still don't understand why we need to carry over the information
>> >> >> >>> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
>> >> >> >>> > > >> > such regions are free to be reused by the kernel after some point of
>> >> >> >>> > > >> > initialization. Why does crash dump kernel need to know about them?
>> >> >> >>> > > >> >
>> >> >> >>> > > >>
>> >> >> >>> > > >> Not really. According to the UEFI spec, they can be reclaimed after
>> >> >> >>> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
>> >> >> >>> > > >> no longer needs them. Of course, in order to be able to boot a kexec
>> >> >> >>> > > >> kernel, those regions needs to be preserved, which is why they are
>> >> >> >>> > > >> memblock_reserve()'d now.
>> >> >> >>> > > >
>> >> >> >>> > > > For my better understandings, who is actually accessing such regions
>> >> >> >>> > > > during boot time, uefi itself or efistub?
>> >> >> >>> > > >
>> >> >> >>> > >
>> >> >> >>> > > No, only the kernel. This is where the ACPI tables are stored. For
>> >> >> >>> > > instance, on QEMU we have
>> >> >> >>> > >
>> >> >> >>> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
>> >> >> >>> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
>> >> >> >>> > >   01000013)
>> >> >> >>> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
>> >> >> >>> > > BXPC 00000001)
>> >> >> >>> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
>> >> >> >>> > > BXPC 00000001)
>> >> >> >>> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
>> >> >> >>> > > BXPC 00000001)
>> >> >> >>> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
>> >> >> >>> > > BXPC 00000001)
>> >> >> >>> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
>> >> >> >>> > > BXPC 00000001)
>> >> >> >>> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
>> >> >> >>> > > BXPC 00000001)
>> >> >> >>> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
>> >> >> >>> > > BXPC 00000001)
>> >> >> >>> > >
>> >> >> >>> > > covered by
>> >> >> >>> > >
>> >> >> >>> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
>> >> >> >>> > >  ...
>> >> >> >>> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
>> >> >> >>> >
>> >> >> >>> > OK. I mistakenly understood those regions could be freed after exiting
>> >> >> >>> > UEFI boot services.
>> >> >> >>> >
>> >> >> >>> > >
>> >> >> >>> > > >> So it seems that kexec does not honour the memblock_reserve() table
>> >> >> >>> > > >> when booting the next kernel.
>> >> >> >>> > > >
>> >> >> >>> > > > not really.
>> >> >> >>> > > >
>> >> >> >>> > > >> > (In other words, can or should we skip some part of ACPI-related init code
>> >> >> >>> > > >> > on crash dump kernel?)
>> >> >> >>> > > >> >
>> >> >> >>> > > >>
>> >> >> >>> > > >> I don't think so. And the change to the handling of ACPI reclaim
>> >> >> >>> > > >> regions only revealed the bug, not created it (given that other
>> >> >> >>> > > >> memblock_reserve regions may be affected as well)
>> >> >> >>> > > >
>> >> >> >>> > > > As whether we should honor such reserved regions over kexec'ing
>> >> >> >>> > > > depends on each one's specific nature, we will have to take care one-by-one.
>> >> >> >>> > > > As a matter of fact, no information about "reserved" memblocks is
>> >> >> >>> > > > exposed to user space (via proc/iomem).
>> >> >> >>> > > >
>> >> >> >>> > >
>> >> >> >>> > > That is why I suggested (somewhere in this thread?) to not expose them
>> >> >> >>> > > as 'System RAM'. Do you think that could solve this?
>> >> >> >>> >
>> >> >> >>> > Memblock-reserv'ing them is necessary to prevent their corruption and
>> >> >> >>> > marking them under another name in /proc/iomem would also be good in order
>> >> >> >>> > not to allocate them as part of crash kernel's memory.
>> >> >> >>> >
>> >> >> >>> > But I'm not still convinced that we should export them in useable-
>> >> >> >>> > memory-range to crash dump kernel. They will be accessed through
>> >> >> >>> > acpi_os_map_memory() and so won't be required to be part of system ram
>> >> >> >>> > (or memblocks), I guess.
>> >> >> >>> >     -> Bhupesh?
>> >> >> >>>
>> >> >> >>> I forgot how arm64 kernel retrieve the memory ranges and initialize
>> >> >> >>> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
>> >> >> >>> the memory according to the efi memmap?  For kdump kernel anything other
>> >> >> >>> than usable memory (which is from the dt node instead) should be
>> >> >> >>> reinitialized according to efi passed info, no?
>> >> >> >>
>> >> >> >> All the regions exported in efi memmap will be added to memblock.memory
>> >> >> >> in (u)efi_init() and then trimmed down to the exact range specified as
>> >> >> >> usable-memory-range by fdt_enforce_memory_region().
>> >> >> >>
>> >> >> >> Now I noticed that the current fdt_enforce_memory_region() may not work well
>> >> >> >> with multiple entries in usable-memory-range.
>> >> >> >>
>> >> >> >
>> >> >> > In any case, the root of the problem is that memory regions lose their
>> >> >> > 'memory' annotation due to the way the memory map is mangled before
>> >> >> > being supplied to the kexec kernel.
>> >> >> >
>> >> >> > Would it be possible to classify all memory that we want to hide from
>> >> >> > the kexec kernel as NOMAP instead? That way, it will not be mapped
>> >> >> > implicitly, but will still be mapped cacheable by acpi_os_ioremap(),
>> >> >> > so this seems to be the most appropriate way to deal with the host
>> >> >> > kernel's memory contents.
>> >> >>
>> >> >> Hmm. wouldn't appending the acpi reclaim regions to
>> >> >> 'linux,usable-memory-range' in the dtb being passed to the crashkernel
>> >> >> be better? Because its indirectly achieving a similar objective
>> >> >> (although may be a subset of all System RAM regions on the primary
>> >> >> kernel's memory).
>> >> >>
>> >> >> I am not aware of the background about the current kexec-tools
>> >> >> implementation where we add only the crashkernel range to the dtb
>> >> >> being passed to the crashkernel.
>> >> >>
>> >> >> Probably Akashi can answer better, as to how we arrived at this design
>> >> >> approach and why we didn't want to expose all System RAM regions (i.e.
>> >> >> ! NOMPAP regions) to the crashkernel.
>> >> >>
>> >> >> I am suspecting that some issues were seen/meet when the System RAM (!
>> >> >> NOMAP regions) were exposed to the crashkernel, and that's why we
>> >> >> finalized on this design approach, but this is something which is just
>> >> >> my guess.
>> >> >>
>> >> >> Regards,
>> >> >> Bhupesh
>> >> >>
>> >> >> >>> >
>> >> >> >>> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
>> >> >> >>> > via a kernel command line parameter, "memmap=".
>> >> >> >>>
>> >> >> >>> memmap= is only used in old kexec-tools, now we are passing them via
>> >> >> >>> e820 table.
>> >> >> >>
>> >> >> >> Thanks. I remember that you have explained it before.
>> >> >> >>
>> >> >> >> -Takahiro AKASHI
>> >> >> >>
>> >> >> >>> [snip]
>> >> >> >>>
>> >> >> >>> Thanks
>> >> >> >>> Dave
>> >> >
>> >> > ===8<==
>> >> > From 74e2451fea83d546feae76160ba7de426913fe03 Mon Sep 17 00:00:00 2001
>> >> > From: AKASHI Takahiro <takahiro.akashi at linaro.org>
>> >> > Date: Thu, 21 Dec 2017 19:14:23 +0900
>> >> > Subject: [PATCH] arm64: kdump: mark unusable memory as NOMAP
>> >> >
>> >> > ---
>> >> >  arch/arm64/mm/init.c | 10 ++++++++--
>> >> >  1 file changed, 8 insertions(+), 2 deletions(-)
>> >> >
>> >> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
>> >> > index 00e7b900ca41..8175db94257b 100644
>> >> > --- a/arch/arm64/mm/init.c
>> >> > +++ b/arch/arm64/mm/init.c
>> >> > @@ -352,11 +352,17 @@ static void __init fdt_enforce_memory_region(void)
>> >> >         struct memblock_region reg = {
>> >> >                 .size = 0,
>> >> >         };
>> >> > +       u64 idx;
>> >> > +       phys_addr_t start, end;
>> >> >
>> >> >         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
>> >> >
>> >> > -       if (reg.size)
>> >> > -               memblock_cap_memory_range(reg.base, reg.size);
>> >> > +       if (reg.size) {
>> >> > +               for_each_free_mem_range(idx, NUMA_NO_NODE, MEMBLOCK_NONE,
>> >> > +                                       &start, &end, NULL)
>> >> > +                       memblock_mark_nomap(start, end - start);
>> >> > +               memblock_clear_nomap(reg.base, reg.size);
>> >> > +       }
>> >> >  }
>> >> >
>> >> >  void __init arm64_memblock_init(void)
>> >> > --
>> >> > 2.15.1
>> >> >
>> >>
>> >> Thanks for the patch. After applying this on top of
>> >> 4.15.0-rc4-next-20171220, there seems to be a improvement and the
>> >> crashkernel boot no longer hangs while trying to access the acpi
>> >> tables.
>> >>
>> >> However I notice a minor issue. Please see the log below for
>> >> reference, the following message keeps spamming the console but I see
>> >> the crashkernel boot proceed further.:
>> >>
>> >> [    0.000000] ACPI: NUMA: SRAT: PXM 3 -> MPIDR 0x70303 -> Node 3
>> >> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff]
>> >> [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x2000000000-0x2fffffffff]
>> >> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0x1fffffffff]
>> >> [    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xa000000000-0xafffffffff]
>> >> [    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x9000000000-0x9fffffffff]
>> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe200-0x1ffbffffff]
>> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffc400-0x1ffbffe1ff]
>> >> [    0.000000] NUMA: NODE_DATA(1) on node 0
>> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffa600-0x1ffbffc3ff]
>> >> [    0.000000] NUMA: NODE_DATA(2) on node 0
>> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbff8800-0x1ffbffa5ff]
>> >> [    0.000000] NUMA: NODE_DATA(3) on node 0
>> >> [    0.000000] [ffff7fe008000000-ffff7fe00800ffff] potential offnode
>> >> page_structs
>> >> [    0.000000] [ffff7fe008010000-ffff7fe00801ffff] potential offnode
>> >> page_structs
>> >> [    0.000000] [ffff7fe008020000-ffff7fe00802ffff] potential offnode
>> >> page_structs
>> >> [    0.000000] [ffff7fe008030000-ffff7fe00803ffff] potential offnode
>> >> page_structs
>> >> [    0.000000] [ffff7fe008040000-ffff7fe00804ffff] potential offnode
>> >> page_structs
>> >> [    0.000000] [ffff7fe008050000-ffff7fe00805ffff] potential offnode
>> >> page_structs
>> >>
>> >> [snip..]
>> >> [    0.000000] [ffff7fe0081f0000-ffff7fe0081fffff] potential offnode
>> >> page_structs
>> >
>> > These messages shows that some "struct page" data are allocated on remote
>> > (numa) nodes.
>> > Since on your crash dump kernel, all the usable system memory (starting
>> > 0x0e800000) belongs to Node#0, we can't avoid such non-local allocations.
>> >
>> > In my best guess, you can ingore them except for some performance penality.
>> > This may be one side-effect.
>> >
>> > So does your crash dump kernel now boot successfully?
>> >
>>
>> Indeed. The crash dump kernel now boots successfully and the crash
>> dump core can be saved properly as well (I tried saving it to local
>> disk).
>
> Thank you for the confirmation.
> (I'd like to suggest you to examine the core dump with crash utility.)
>
>> However, the 'potential offnode page_structs' WARN messages hog the
>> console and delay crashkernel boot for a significant duration, which
>> can be irritating.
>>
>> Can we also consider ratelimiting this WARNING message [which seems to
>> come from vmemmap_verify()] if invoked in the context of crash kernel,
>> in addition to making the above change suggested by  you.
>
> Well, we may be able to change pr_warn() to pr_warn_once() here, but
> I hope that adding "numa=off" to kernel command line should also work.

Hmm, adding "numa=off" to crashkernel bootargs works, and TBH it was
my initial thought process as well, but I am not sure if this will
cause any regressions on aarch64 systems which use crashdump feature.

I think the 2nd solution, i.e limiting the warn message print
frequency might be a better option. Can you please add the following
patch (may be as a separate one) and send it along the patch which
marks all areas other than the crashkernel region being passed to the
crashkernel as NOMAP, so that we can get this issue fixed in upstream
aarch64 kernel:

diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index 17acf01791fa..4c13fe3c644d 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -169,7 +169,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node,
        int actual_node = early_pfn_to_nid(pfn);

        if (node_distance(actual_node, node) > LOCAL_DISTANCE)
-               pr_warn("[%lx-%lx] potential offnode page_structs\n",
+               pr_warn_once("[%lx-%lx] potential offnode page_structs\n",
                        start, end - 1);
 }

I have tested this solution on huawei taishan board and can boot
crashkernel successfully and also save the crash core properly
(without the  console warn message flooding which used to hold up the
crashkernel boot).

Thanks,
Bhupesh


From dyoung at redhat.com  Mon Dec 25 17:32:17 2017
From: dyoung at redhat.com (Dave Young)
Date: Tue, 26 Dec 2017 09:32:17 +0800
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>
References: <20171218054009.GA6392@dhcp-128-65.nay.redhat.com>
 <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
 <20171225032500.GA8877@linaro.org>
 <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>
Message-ID: <20171226013217.GA2119@dhcp-128-65.nay.redhat.com>

On 12/26/17 at 01:44am, Bhupesh Sharma wrote:
> On Mon, Dec 25, 2017 at 8:55 AM, AKASHI Takahiro
> <takahiro.akashi at linaro.org> wrote:
> > On Sun, Dec 24, 2017 at 01:21:02AM +0530, Bhupesh Sharma wrote:
> >> On Fri, Dec 22, 2017 at 2:03 PM, AKASHI Takahiro
> >> <takahiro.akashi at linaro.org> wrote:
> >> > On Thu, Dec 21, 2017 at 05:36:30PM +0530, Bhupesh Sharma wrote:
> >> >> Hello Akashi,
> >> >>
> >> >> On Thu, Dec 21, 2017 at 4:04 PM, AKASHI Takahiro
> >> >> <takahiro.akashi at linaro.org> wrote:
> >> >> > Bhupesh,
> >> >> >
> >> >> > Can you test the patch attached below, please?
> >> >> >
> >> >> > It is intended to retain already-reserved regions (ACPI reclaim memory
> >> >> > in this case) in system ram (i.e. memblock.memory) without explicitly
> >> >> > exporting them via usable-memory-range.
> >> >> > (I still have to figure out what the side-effect of this patch is.)
> >> >> >
> >> >> > Thanks,
> >> >> > -Takahiro AKASHI
> >> >> >
> >> >> > On Thu, Dec 21, 2017 at 01:30:43AM +0530, Bhupesh Sharma wrote:
> >> >> >> On Tue, Dec 19, 2017 at 6:39 PM, Ard Biesheuvel
> >> >> >> <ard.biesheuvel at linaro.org> wrote:
> >> >> >> > On 19 December 2017 at 07:09, AKASHI Takahiro
> >> >> >> > <takahiro.akashi at linaro.org> wrote:
> >> >> >> >> On Mon, Dec 18, 2017 at 01:40:09PM +0800, Dave Young wrote:
> >> >> >> >>> On 12/15/17 at 05:59pm, AKASHI Takahiro wrote:
> >> >> >> >>> > On Wed, Dec 13, 2017 at 12:17:22PM +0000, Ard Biesheuvel wrote:
> >> >> >> >>> > > On 13 December 2017 at 12:16, AKASHI Takahiro
> >> >> >> >>> > > <takahiro.akashi at linaro.org> wrote:
> >> >> >> >>> > > > On Wed, Dec 13, 2017 at 10:49:27AM +0000, Ard Biesheuvel wrote:
> >> >> >> >>> > > >> On 13 December 2017 at 10:26, AKASHI Takahiro
> >> >> >> >>> > > >> <takahiro.akashi at linaro.org> wrote:
> >> >> >> >>> > > >> > Bhupesh, Ard,
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >> > On Wed, Dec 13, 2017 at 03:21:59AM +0530, Bhupesh Sharma wrote:
> >> >> >> >>> > > >> >> Hi Ard, Akashi
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> > (snip)
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >> >> Looking deeper into the issue, since the arm64 kexec-tools uses the
> >> >> >> >>> > > >> >> 'linux,usable-memory-range' dt property to allow crash dump kernel to
> >> >> >> >>> > > >> >> identify its own usable memory and exclude, at its boot time, any
> >> >> >> >>> > > >> >> other memory areas that are part of the panicked kernel's memory.
> >> >> >> >>> > > >> >> (see https://www.kernel.org/doc/Documentation/devicetree/bindings/chosen.txt
> >> >> >> >>> > > >> >> , for details)
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >> > Right.
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >> >> 1). Now when 'kexec -p' is executed, this node is patched up only
> >> >> >> >>> > > >> >> with the crashkernel memory range:
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >>                 /* add linux,usable-memory-range */
> >> >> >> >>> > > >> >>                 nodeoffset = fdt_path_offset(new_buf, "/chosen");
> >> >> >> >>> > > >> >>                 result = fdt_setprop_range(new_buf, nodeoffset,
> >> >> >> >>> > > >> >>                                 PROP_USABLE_MEM_RANGE, &crash_reserved_mem,
> >> >> >> >>> > > >> >>                                 address_cells, size_cells);
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> (see https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/tree/kexec/arch/arm64/kexec-arm64.c#n465
> >> >> >> >>> > > >> >> , for details)
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> 2). This excludes the ACPI reclaim regions irrespective of whether
> >> >> >> >>> > > >> >> they are marked as System RAM or as RESERVED. As,
> >> >> >> >>> > > >> >> 'linux,usable-memory-range' dt node is patched up only with
> >> >> >> >>> > > >> >> 'crash_reserved_mem' and not 'system_memory_ranges'
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> 3). As a result when the crashkernel boots up it doesn't find this
> >> >> >> >>> > > >> >> ACPI memory and crashes while trying to access the same:
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> # kexec -p /boot/vmlinuz-`uname -r` --initrd=/boot/initramfs-`uname
> >> >> >> >>> > > >> >> -r`.img --reuse-cmdline -d
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> [snip..]
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> Reserved memory range
> >> >> >> >>> > > >> >> 000000000e800000-000000002e7fffff (0)
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> Coredump memory ranges
> >> >> >> >>> > > >> >> 0000000000000000-000000000e7fffff (0)
> >> >> >> >>> > > >> >> 000000002e800000-000000003961ffff (0)
> >> >> >> >>> > > >> >> 0000000039d40000-000000003ed2ffff (0)
> >> >> >> >>> > > >> >> 000000003ed60000-000000003fbfffff (0)
> >> >> >> >>> > > >> >> 0000001040000000-0000001ffbffffff (0)
> >> >> >> >>> > > >> >> 0000002000000000-0000002ffbffffff (0)
> >> >> >> >>> > > >> >> 0000009000000000-0000009ffbffffff (0)
> >> >> >> >>> > > >> >> 000000a000000000-000000affbffffff (0)
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> 4). So if we revert Ard's patch or just comment the fixing up of the
> >> >> >> >>> > > >> >> memory cap'ing passed to the crash kernel inside
> >> >> >> >>> > > >> >> 'arch/arm64/mm/init.c' (see below):
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> static void __init fdt_enforce_memory_region(void)
> >> >> >> >>> > > >> >> {
> >> >> >> >>> > > >> >>         struct memblock_region reg = {
> >> >> >> >>> > > >> >>                 .size = 0,
> >> >> >> >>> > > >> >>         };
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >>         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >>         if (reg.size)
> >> >> >> >>> > > >> >>                 //memblock_cap_memory_range(reg.base, reg.size); /*
> >> >> >> >>> > > >> >> comment this out */
> >> >> >> >>> > > >> >> }
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >> > Please just don't do that. It can cause a fatal damage on
> >> >> >> >>> > > >> > memory contents of the *crashed* kernel.
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >> >> 5). Both the above temporary solutions fix the problem.
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> 6). However exposing all System RAM regions to the crashkernel is not
> >> >> >> >>> > > >> >> advisable and may cause the crashkernel or some crashkernel drivers to
> >> >> >> >>> > > >> >> fail.
> >> >> >> >>> > > >> >>
> >> >> >> >>> > > >> >> 6a). I am trying an approach now, where the ACPI reclaim regions are
> >> >> >> >>> > > >> >> added to '/proc/iomem' separately as ACPI reclaim regions by the
> >> >> >> >>> > > >> >> kernel code and on the other hand the user-space 'kexec-tools' will
> >> >> >> >>> > > >> >> pick up the ACPI reclaim regions from '/proc/iomem' and add it to the
> >> >> >> >>> > > >> >> dt node 'linux,usable-memory-range'
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >> > I still don't understand why we need to carry over the information
> >> >> >> >>> > > >> > about "ACPI Reclaim memory" to crash dump kernel. In my understandings,
> >> >> >> >>> > > >> > such regions are free to be reused by the kernel after some point of
> >> >> >> >>> > > >> > initialization. Why does crash dump kernel need to know about them?
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >>
> >> >> >> >>> > > >> Not really. According to the UEFI spec, they can be reclaimed after
> >> >> >> >>> > > >> the OS has initialized, i.e., when it has consumed the ACPI tables and
> >> >> >> >>> > > >> no longer needs them. Of course, in order to be able to boot a kexec
> >> >> >> >>> > > >> kernel, those regions needs to be preserved, which is why they are
> >> >> >> >>> > > >> memblock_reserve()'d now.
> >> >> >> >>> > > >
> >> >> >> >>> > > > For my better understandings, who is actually accessing such regions
> >> >> >> >>> > > > during boot time, uefi itself or efistub?
> >> >> >> >>> > > >
> >> >> >> >>> > >
> >> >> >> >>> > > No, only the kernel. This is where the ACPI tables are stored. For
> >> >> >> >>> > > instance, on QEMU we have
> >> >> >> >>> > >
> >> >> >> >>> > >  ACPI: RSDP 0x0000000078980000 000024 (v02 BOCHS )
> >> >> >> >>> > >  ACPI: XSDT 0x0000000078970000 000054 (v01 BOCHS  BXPCFACP 00000001
> >> >> >> >>> > >   01000013)
> >> >> >> >>> > >  ACPI: FACP 0x0000000078930000 00010C (v05 BOCHS  BXPCFACP 00000001
> >> >> >> >>> > > BXPC 00000001)
> >> >> >> >>> > >  ACPI: DSDT 0x0000000078940000 0011DA (v02 BOCHS  BXPCDSDT 00000001
> >> >> >> >>> > > BXPC 00000001)
> >> >> >> >>> > >  ACPI: APIC 0x0000000078920000 000140 (v03 BOCHS  BXPCAPIC 00000001
> >> >> >> >>> > > BXPC 00000001)
> >> >> >> >>> > >  ACPI: GTDT 0x0000000078910000 000060 (v02 BOCHS  BXPCGTDT 00000001
> >> >> >> >>> > > BXPC 00000001)
> >> >> >> >>> > >  ACPI: MCFG 0x0000000078900000 00003C (v01 BOCHS  BXPCMCFG 00000001
> >> >> >> >>> > > BXPC 00000001)
> >> >> >> >>> > >  ACPI: SPCR 0x00000000788F0000 000050 (v02 BOCHS  BXPCSPCR 00000001
> >> >> >> >>> > > BXPC 00000001)
> >> >> >> >>> > >  ACPI: IORT 0x00000000788E0000 00007C (v00 BOCHS  BXPCIORT 00000001
> >> >> >> >>> > > BXPC 00000001)
> >> >> >> >>> > >
> >> >> >> >>> > > covered by
> >> >> >> >>> > >
> >> >> >> >>> > >  efi:   0x0000788e0000-0x00007894ffff [ACPI Reclaim Memory ...]
> >> >> >> >>> > >  ...
> >> >> >> >>> > >  efi:   0x000078970000-0x00007898ffff [ACPI Reclaim Memory ...]
> >> >> >> >>> >
> >> >> >> >>> > OK. I mistakenly understood those regions could be freed after exiting
> >> >> >> >>> > UEFI boot services.
> >> >> >> >>> >
> >> >> >> >>> > >
> >> >> >> >>> > > >> So it seems that kexec does not honour the memblock_reserve() table
> >> >> >> >>> > > >> when booting the next kernel.
> >> >> >> >>> > > >
> >> >> >> >>> > > > not really.
> >> >> >> >>> > > >
> >> >> >> >>> > > >> > (In other words, can or should we skip some part of ACPI-related init code
> >> >> >> >>> > > >> > on crash dump kernel?)
> >> >> >> >>> > > >> >
> >> >> >> >>> > > >>
> >> >> >> >>> > > >> I don't think so. And the change to the handling of ACPI reclaim
> >> >> >> >>> > > >> regions only revealed the bug, not created it (given that other
> >> >> >> >>> > > >> memblock_reserve regions may be affected as well)
> >> >> >> >>> > > >
> >> >> >> >>> > > > As whether we should honor such reserved regions over kexec'ing
> >> >> >> >>> > > > depends on each one's specific nature, we will have to take care one-by-one.
> >> >> >> >>> > > > As a matter of fact, no information about "reserved" memblocks is
> >> >> >> >>> > > > exposed to user space (via proc/iomem).
> >> >> >> >>> > > >
> >> >> >> >>> > >
> >> >> >> >>> > > That is why I suggested (somewhere in this thread?) to not expose them
> >> >> >> >>> > > as 'System RAM'. Do you think that could solve this?
> >> >> >> >>> >
> >> >> >> >>> > Memblock-reserv'ing them is necessary to prevent their corruption and
> >> >> >> >>> > marking them under another name in /proc/iomem would also be good in order
> >> >> >> >>> > not to allocate them as part of crash kernel's memory.
> >> >> >> >>> >
> >> >> >> >>> > But I'm not still convinced that we should export them in useable-
> >> >> >> >>> > memory-range to crash dump kernel. They will be accessed through
> >> >> >> >>> > acpi_os_map_memory() and so won't be required to be part of system ram
> >> >> >> >>> > (or memblocks), I guess.
> >> >> >> >>> >     -> Bhupesh?
> >> >> >> >>>
> >> >> >> >>> I forgot how arm64 kernel retrieve the memory ranges and initialize
> >> >> >> >>> them.  If no "e820" like interfaces shouldn't kernel reinitialize all
> >> >> >> >>> the memory according to the efi memmap?  For kdump kernel anything other
> >> >> >> >>> than usable memory (which is from the dt node instead) should be
> >> >> >> >>> reinitialized according to efi passed info, no?
> >> >> >> >>
> >> >> >> >> All the regions exported in efi memmap will be added to memblock.memory
> >> >> >> >> in (u)efi_init() and then trimmed down to the exact range specified as
> >> >> >> >> usable-memory-range by fdt_enforce_memory_region().
> >> >> >> >>
> >> >> >> >> Now I noticed that the current fdt_enforce_memory_region() may not work well
> >> >> >> >> with multiple entries in usable-memory-range.
> >> >> >> >>
> >> >> >> >
> >> >> >> > In any case, the root of the problem is that memory regions lose their
> >> >> >> > 'memory' annotation due to the way the memory map is mangled before
> >> >> >> > being supplied to the kexec kernel.
> >> >> >> >
> >> >> >> > Would it be possible to classify all memory that we want to hide from
> >> >> >> > the kexec kernel as NOMAP instead? That way, it will not be mapped
> >> >> >> > implicitly, but will still be mapped cacheable by acpi_os_ioremap(),
> >> >> >> > so this seems to be the most appropriate way to deal with the host
> >> >> >> > kernel's memory contents.
> >> >> >>
> >> >> >> Hmm. wouldn't appending the acpi reclaim regions to
> >> >> >> 'linux,usable-memory-range' in the dtb being passed to the crashkernel
> >> >> >> be better? Because its indirectly achieving a similar objective
> >> >> >> (although may be a subset of all System RAM regions on the primary
> >> >> >> kernel's memory).
> >> >> >>
> >> >> >> I am not aware of the background about the current kexec-tools
> >> >> >> implementation where we add only the crashkernel range to the dtb
> >> >> >> being passed to the crashkernel.
> >> >> >>
> >> >> >> Probably Akashi can answer better, as to how we arrived at this design
> >> >> >> approach and why we didn't want to expose all System RAM regions (i.e.
> >> >> >> ! NOMPAP regions) to the crashkernel.
> >> >> >>
> >> >> >> I am suspecting that some issues were seen/meet when the System RAM (!
> >> >> >> NOMAP regions) were exposed to the crashkernel, and that's why we
> >> >> >> finalized on this design approach, but this is something which is just
> >> >> >> my guess.
> >> >> >>
> >> >> >> Regards,
> >> >> >> Bhupesh
> >> >> >>
> >> >> >> >>> >
> >> >> >> >>> > Just FYI, on x86, ACPI tables seems to be exposed to crash dump kernel
> >> >> >> >>> > via a kernel command line parameter, "memmap=".
> >> >> >> >>>
> >> >> >> >>> memmap= is only used in old kexec-tools, now we are passing them via
> >> >> >> >>> e820 table.
> >> >> >> >>
> >> >> >> >> Thanks. I remember that you have explained it before.
> >> >> >> >>
> >> >> >> >> -Takahiro AKASHI
> >> >> >> >>
> >> >> >> >>> [snip]
> >> >> >> >>>
> >> >> >> >>> Thanks
> >> >> >> >>> Dave
> >> >> >
> >> >> > ===8<==
> >> >> > From 74e2451fea83d546feae76160ba7de426913fe03 Mon Sep 17 00:00:00 2001
> >> >> > From: AKASHI Takahiro <takahiro.akashi at linaro.org>
> >> >> > Date: Thu, 21 Dec 2017 19:14:23 +0900
> >> >> > Subject: [PATCH] arm64: kdump: mark unusable memory as NOMAP
> >> >> >
> >> >> > ---
> >> >> >  arch/arm64/mm/init.c | 10 ++++++++--
> >> >> >  1 file changed, 8 insertions(+), 2 deletions(-)
> >> >> >
> >> >> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> >> >> > index 00e7b900ca41..8175db94257b 100644
> >> >> > --- a/arch/arm64/mm/init.c
> >> >> > +++ b/arch/arm64/mm/init.c
> >> >> > @@ -352,11 +352,17 @@ static void __init fdt_enforce_memory_region(void)
> >> >> >         struct memblock_region reg = {
> >> >> >                 .size = 0,
> >> >> >         };
> >> >> > +       u64 idx;
> >> >> > +       phys_addr_t start, end;
> >> >> >
> >> >> >         of_scan_flat_dt(early_init_dt_scan_usablemem, &reg);
> >> >> >
> >> >> > -       if (reg.size)
> >> >> > -               memblock_cap_memory_range(reg.base, reg.size);
> >> >> > +       if (reg.size) {
> >> >> > +               for_each_free_mem_range(idx, NUMA_NO_NODE, MEMBLOCK_NONE,
> >> >> > +                                       &start, &end, NULL)
> >> >> > +                       memblock_mark_nomap(start, end - start);
> >> >> > +               memblock_clear_nomap(reg.base, reg.size);
> >> >> > +       }
> >> >> >  }
> >> >> >
> >> >> >  void __init arm64_memblock_init(void)
> >> >> > --
> >> >> > 2.15.1
> >> >> >
> >> >>
> >> >> Thanks for the patch. After applying this on top of
> >> >> 4.15.0-rc4-next-20171220, there seems to be a improvement and the
> >> >> crashkernel boot no longer hangs while trying to access the acpi
> >> >> tables.
> >> >>
> >> >> However I notice a minor issue. Please see the log below for
> >> >> reference, the following message keeps spamming the console but I see
> >> >> the crashkernel boot proceed further.:
> >> >>
> >> >> [    0.000000] ACPI: NUMA: SRAT: PXM 3 -> MPIDR 0x70303 -> Node 3
> >> >> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x3fffffff]
> >> >> [    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x2000000000-0x2fffffffff]
> >> >> [    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x1000000000-0x1fffffffff]
> >> >> [    0.000000] ACPI: SRAT: Node 3 PXM 3 [mem 0xa000000000-0xafffffffff]
> >> >> [    0.000000] ACPI: SRAT: Node 2 PXM 2 [mem 0x9000000000-0x9fffffffff]
> >> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffe200-0x1ffbffffff]
> >> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffc400-0x1ffbffe1ff]
> >> >> [    0.000000] NUMA: NODE_DATA(1) on node 0
> >> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbffa600-0x1ffbffc3ff]
> >> >> [    0.000000] NUMA: NODE_DATA(2) on node 0
> >> >> [    0.000000] NUMA: NODE_DATA [mem 0x1ffbff8800-0x1ffbffa5ff]
> >> >> [    0.000000] NUMA: NODE_DATA(3) on node 0
> >> >> [    0.000000] [ffff7fe008000000-ffff7fe00800ffff] potential offnode
> >> >> page_structs
> >> >> [    0.000000] [ffff7fe008010000-ffff7fe00801ffff] potential offnode
> >> >> page_structs
> >> >> [    0.000000] [ffff7fe008020000-ffff7fe00802ffff] potential offnode
> >> >> page_structs
> >> >> [    0.000000] [ffff7fe008030000-ffff7fe00803ffff] potential offnode
> >> >> page_structs
> >> >> [    0.000000] [ffff7fe008040000-ffff7fe00804ffff] potential offnode
> >> >> page_structs
> >> >> [    0.000000] [ffff7fe008050000-ffff7fe00805ffff] potential offnode
> >> >> page_structs
> >> >>
> >> >> [snip..]
> >> >> [    0.000000] [ffff7fe0081f0000-ffff7fe0081fffff] potential offnode
> >> >> page_structs
> >> >
> >> > These messages shows that some "struct page" data are allocated on remote
> >> > (numa) nodes.
> >> > Since on your crash dump kernel, all the usable system memory (starting
> >> > 0x0e800000) belongs to Node#0, we can't avoid such non-local allocations.
> >> >
> >> > In my best guess, you can ingore them except for some performance penality.
> >> > This may be one side-effect.
> >> >
> >> > So does your crash dump kernel now boot successfully?
> >> >
> >>
> >> Indeed. The crash dump kernel now boots successfully and the crash
> >> dump core can be saved properly as well (I tried saving it to local
> >> disk).
> >
> > Thank you for the confirmation.
> > (I'd like to suggest you to examine the core dump with crash utility.)
> >
> >> However, the 'potential offnode page_structs' WARN messages hog the
> >> console and delay crashkernel boot for a significant duration, which
> >> can be irritating.
> >>
> >> Can we also consider ratelimiting this WARNING message [which seems to
> >> come from vmemmap_verify()] if invoked in the context of crash kernel,
> >> in addition to making the above change suggested by  you.
> >
> > Well, we may be able to change pr_warn() to pr_warn_once() here, but
> > I hope that adding "numa=off" to kernel command line should also work.
> 
> Hmm, adding "numa=off" to crashkernel bootargs works, and TBH it was
> my initial thought process as well, but I am not sure if this will
> cause any regressions on aarch64 systems which use crashdump feature.

It should be fine since we use numa=off by default for all other arches
ie. x86, ppc64 and s390. Actually disabling numa in kdump kernel can save
mm component memory usage. 

> 
> I think the 2nd solution, i.e limiting the warn message print
> frequency might be a better option. Can you please add the following
> patch (may be as a separate one) and send it along the patch which
> marks all areas other than the crashkernel region being passed to the
> crashkernel as NOMAP, so that we can get this issue fixed in upstream
> aarch64 kernel:
> 
> diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
> index 17acf01791fa..4c13fe3c644d 100644
> --- a/mm/sparse-vmemmap.c
> +++ b/mm/sparse-vmemmap.c
> @@ -169,7 +169,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node,
>         int actual_node = early_pfn_to_nid(pfn);
> 
>         if (node_distance(actual_node, node) > LOCAL_DISTANCE)
> -               pr_warn("[%lx-%lx] potential offnode page_structs\n",
> +               pr_warn_once("[%lx-%lx] potential offnode page_structs\n",
>                         start, end - 1);
>  }
> 
> I have tested this solution on huawei taishan board and can boot
> crashkernel successfully and also save the crash core properly
> (without the  console warn message flooding which used to hold up the
> crashkernel boot).
> 
> Thanks,
> Bhupesh

Thanks
Dave


From dyoung at redhat.com  Mon Dec 25 17:35:17 2017
From: dyoung at redhat.com (Dave Young)
Date: Tue, 26 Dec 2017 09:35:17 +0800
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <20171226013217.GA2119@dhcp-128-65.nay.redhat.com>
References: <20171219060927.GH28046@linaro.org>
 <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
 <20171225032500.GA8877@linaro.org>
 <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>
 <20171226013217.GA2119@dhcp-128-65.nay.redhat.com>
Message-ID: <20171226013517.GA2186@dhcp-128-65.nay.redhat.com>

[snip]
> > > Well, we may be able to change pr_warn() to pr_warn_once() here, but
> > > I hope that adding "numa=off" to kernel command line should also work.
> > 
> > Hmm, adding "numa=off" to crashkernel bootargs works, and TBH it was
> > my initial thought process as well, but I am not sure if this will
> > cause any regressions on aarch64 systems which use crashdump feature.
> 
> It should be fine since we use numa=off by default for all other arches
> ie. x86, ppc64 and s390. Actually disabling numa in kdump kernel can save
> mm component memory usage. 
> 

Forgot to say I means in RHEL and Fedora we use numa=off for kdump..


From takahiro.akashi at linaro.org  Mon Dec 25 18:28:09 2017
From: takahiro.akashi at linaro.org (AKASHI Takahiro)
Date: Tue, 26 Dec 2017 11:28:09 +0900
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <20171226013517.GA2186@dhcp-128-65.nay.redhat.com>
References: <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
 <20171225032500.GA8877@linaro.org>
 <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>
 <20171226013217.GA2119@dhcp-128-65.nay.redhat.com>
 <20171226013517.GA2186@dhcp-128-65.nay.redhat.com>
Message-ID: <20171226022807.GB8877@linaro.org>

On Tue, Dec 26, 2017 at 09:35:17AM +0800, Dave Young wrote:
> [snip]
> > > > Well, we may be able to change pr_warn() to pr_warn_once() here, but
> > > > I hope that adding "numa=off" to kernel command line should also work.
> > > 
> > > Hmm, adding "numa=off" to crashkernel bootargs works, and TBH it was
> > > my initial thought process as well, but I am not sure if this will
> > > cause any regressions on aarch64 systems which use crashdump feature.
> > 
> > It should be fine since we use numa=off by default for all other arches
> > ie. x86, ppc64 and s390. Actually disabling numa in kdump kernel can save
> > mm component memory usage. 
> > 
> 
> Forgot to say I means in RHEL and Fedora we use numa=off for kdump..

Thank you for the clarification.
(It might be better to make numa off automatically if maxcpus == 0 (and 1?).)

-Takahiro AKASHI


From bhsharma at redhat.com  Mon Dec 25 18:56:19 2017
From: bhsharma at redhat.com (Bhupesh Sharma)
Date: Tue, 26 Dec 2017 08:26:19 +0530
Subject: arm64 crashkernel fails to boot on acpi-only machines due to ACPI
 regions being no longer mapped as NOMAP
In-Reply-To: <20171226022807.GB8877@linaro.org>
References: <CAKv+Gu-gmbWdZ7rxp5qGrtSBQ7dM=3FqF-Pw=J0LaL=oKTMg4w@mail.gmail.com>
 <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
 <20171225032500.GA8877@linaro.org>
 <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>
 <20171226013217.GA2119@dhcp-128-65.nay.redhat.com>
 <20171226013517.GA2186@dhcp-128-65.nay.redhat.com>
 <20171226022807.GB8877@linaro.org>
Message-ID: <CACi5LpNRtXh-j9Y9HwRatDZwRMr++-ZeaSnk62vD3btpxsVv7w@mail.gmail.com>

On Tue, Dec 26, 2017 at 7:58 AM, AKASHI Takahiro
<takahiro.akashi at linaro.org> wrote:
> On Tue, Dec 26, 2017 at 09:35:17AM +0800, Dave Young wrote:
>> [snip]
>> > > > Well, we may be able to change pr_warn() to pr_warn_once() here, but
>> > > > I hope that adding "numa=off" to kernel command line should also work.
>> > >
>> > > Hmm, adding "numa=off" to crashkernel bootargs works, and TBH it was
>> > > my initial thought process as well, but I am not sure if this will
>> > > cause any regressions on aarch64 systems which use crashdump feature.
>> >
>> > It should be fine since we use numa=off by default for all other arches
>> > ie. x86, ppc64 and s390. Actually disabling numa in kdump kernel can save
>> > mm component memory usage.
>> >
>>
>> Forgot to say I means in RHEL and Fedora we use numa=off for kdump..
>
> Thank you for the clarification.
> (It might be better to make numa off automatically if maxcpus == 0 (and 1?).)
>

Not sure if we can leave this to the distribution-specific kdump
scripts (as the crashkernel boot can be held up for sufficient time
and may appear stuck). The distribution scripts may be different (for
e.g. ubuntu and RHEL/fedora) across distributions and may have
different bootarg options.

So how about considering a kernel fix only which doesn't require
relying on changing the distribution-specific kdump scripts, as we
should avoid introducing a regression while trying to fix a regression
:)

Just my 2 cents.

Thanks,
Bhupesh


From dyoung at redhat.com  Mon Dec 25 22:56:36 2017
From: dyoung at redhat.com (Dave Young)
Date: Tue, 26 Dec 2017 14:56:36 +0800
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <20171226022807.GB8877@linaro.org>
References: <CACi5LpOscbcBecWaC3Q9P22kheRYc+M2Ynfusszk14fPY-cJ5A@mail.gmail.com>
 <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
 <20171225032500.GA8877@linaro.org>
 <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>
 <20171226013217.GA2119@dhcp-128-65.nay.redhat.com>
 <20171226013517.GA2186@dhcp-128-65.nay.redhat.com>
 <20171226022807.GB8877@linaro.org>
Message-ID: <20171226065636.GA5354@dhcp-128-65.nay.redhat.com>

On 12/26/17 at 11:28am, AKASHI Takahiro wrote:
> On Tue, Dec 26, 2017 at 09:35:17AM +0800, Dave Young wrote:
> > [snip]
> > > > > Well, we may be able to change pr_warn() to pr_warn_once() here, but
> > > > > I hope that adding "numa=off" to kernel command line should also work.
> > > > 
> > > > Hmm, adding "numa=off" to crashkernel bootargs works, and TBH it was
> > > > my initial thought process as well, but I am not sure if this will
> > > > cause any regressions on aarch64 systems which use crashdump feature.
> > > 
> > > It should be fine since we use numa=off by default for all other arches
> > > ie. x86, ppc64 and s390. Actually disabling numa in kdump kernel can save
> > > mm component memory usage. 
> > > 
> > 
> > Forgot to say I means in RHEL and Fedora we use numa=off for kdump..
> 
> Thank you for the clarification.
> (It might be better to make numa off automatically if maxcpus == 0 (and 1?).)

Hmm, I did a quick test with qemu/kvm, kdump kernel boot without numa=off
I'm not sure why I do not see the warning messages on x86
machines, maybe something arm64 specific?

> 
> -Takahiro AKASHI


From dyoung at redhat.com  Mon Dec 25 22:58:45 2017
From: dyoung at redhat.com (Dave Young)
Date: Tue, 26 Dec 2017 14:58:45 +0800
Subject: arm64 crashkernel fails to boot on acpi-only machines due to
 ACPI regions being no longer mapped as NOMAP
In-Reply-To: <CACi5LpNRtXh-j9Y9HwRatDZwRMr++-ZeaSnk62vD3btpxsVv7w@mail.gmail.com>
References: <20171221103440.GJ28046@linaro.org>
 <CACi5LpMUnUKxiALAHW9_PE2RYC8GNWLPGpdJ5ca53g=v3rNkfg@mail.gmail.com>
 <20171222083308.cy3vwlyhig22q5ee@fireball>
 <CACi5LpNF5i3Eo7nMLr_z9r4VVbXhDwSJCQoiOh-A_jB6hV0_2Q@mail.gmail.com>
 <20171225032500.GA8877@linaro.org>
 <CACi5LpMzYidDaC0_yfwgVOisH-FqcNViYj+Z54uKfUtHkJKKXA@mail.gmail.com>
 <20171226013217.GA2119@dhcp-128-65.nay.redhat.com>
 <20171226013517.GA2186@dhcp-128-65.nay.redhat.com>
 <20171226022807.GB8877@linaro.org>
 <CACi5LpNRtXh-j9Y9HwRatDZwRMr++-ZeaSnk62vD3btpxsVv7w@mail.gmail.com>
Message-ID: <20171226065845.GB5354@dhcp-128-65.nay.redhat.com>

On 12/26/17 at 08:26am, Bhupesh Sharma wrote:
> On Tue, Dec 26, 2017 at 7:58 AM, AKASHI Takahiro
> <takahiro.akashi at linaro.org> wrote:
> > On Tue, Dec 26, 2017 at 09:35:17AM +0800, Dave Young wrote:
> >> [snip]
> >> > > > Well, we may be able to change pr_warn() to pr_warn_once() here, but
> >> > > > I hope that adding "numa=off" to kernel command line should also work.
> >> > >
> >> > > Hmm, adding "numa=off" to crashkernel bootargs works, and TBH it was
> >> > > my initial thought process as well, but I am not sure if this will
> >> > > cause any regressions on aarch64 systems which use crashdump feature.
> >> >
> >> > It should be fine since we use numa=off by default for all other arches
> >> > ie. x86, ppc64 and s390. Actually disabling numa in kdump kernel can save
> >> > mm component memory usage.
> >> >
> >>
> >> Forgot to say I means in RHEL and Fedora we use numa=off for kdump..
> >
> > Thank you for the clarification.
> > (It might be better to make numa off automatically if maxcpus == 0 (and 1?).)
> >
> 
> Not sure if we can leave this to the distribution-specific kdump
> scripts (as the crashkernel boot can be held up for sufficient time
> and may appear stuck). The distribution scripts may be different (for
> e.g. ubuntu and RHEL/fedora) across distributions and may have
> different bootarg options.

Personally I think distribution should take care of this param as for
kdump.  But as AKASHI said it could be a issue for 1st kernel with
nr_cpus=1 booting.  Problem is why we do not see this issue on other
machines.

> 
> So how about considering a kernel fix only which doesn't require
> relying on changing the distribution-specific kdump scripts, as we
> should avoid introducing a regression while trying to fix a regression
> :)
> 
> Just my 2 cents.
> 
> Thanks,
> Bhupesh

Thanks
Dave


From ats-kumagai at wm.jp.nec.com  Tue Dec 26 00:21:26 2017
From: ats-kumagai at wm.jp.nec.com (Atsushi Kumagai)
Date: Tue, 26 Dec 2017 08:21:26 +0000
Subject: [PATCH] handle renamed init_level4_pgt -> init_top_pgt
In-Reply-To: <20171222054829.GA21523@dhcp-128-65.nay.redhat.com>
References: <0910DD04CBD6DE4193FCF86B9C00BE9701F22E63@BPXM01GP.gisp.nec.co.jp>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2A617@BPXM01GP.gisp.nec.co.jp>
 <20171220072443.GA7413@dhcp-128-65.nay.redhat.com>
 <0910DD04CBD6DE4193FCF86B9C00BE9701F2D609@BPXM01GP.gisp.nec.co.jp>
 <20171222045444.GA978@dhcp-128-65.nay.redhat.com>
 <20171222054829.GA21523@dhcp-128-65.nay.redhat.com>
Message-ID: <0910DD04CBD6DE4193FCF86B9C00BE9701F31899@BPXM01GP.gisp.nec.co.jp>

>On 12/22/17 at 12:54pm, Dave Young wrote:
>> Hi Atsushi,
>> On 12/21/17 at 08:48am, Atsushi Kumagai wrote:
>> > Hello Dave,
>> >
>> > >[dyoung at dhcp-*-* makedumpfile]$ sudo ./makedumpfile -l -d 31 /mnt/vmcore/vmcore /tmp/vmcore.1
>> > >The kernel version is not supported.
>> > >The makedumpfile operation may be incomplete.
>> > >Checking for memory holes                         : [100.0 %] |                  __vtop4_x86_64: Can't get a valid
>pte.
>> > >readmem: Can't convert a virtual address(ffff88007ebb1000) to physical address.
>> > >readmem: type_addr: 0, addr:ffff88007ebb1000, size:32768
>> > >__exclude_unnecessary_pages: Can't read the buffer of struct page.
>> > >create_2nd_bitmap: Can't exclude unnecessary pages.
>> > >
>> > >makedumpfile Failed.
>> > >
>> > >If you need the vmcore for debugging please let me know, my test is just
>> > >a normal test in kvm guest.
>> >
>> > Thanks for your report.
>> > I can't reproduce that in my environment, could you give me the vmcore ?
>> > It's helpful if you append the vmlinux and the .config.
>>
>> I'm bisecting it, nearly finished, will post the results and share
>> vmcore if needed soon.
>
>vmcore/vmlinux (gzipped) and .config:
>http://people.redhat.com/ruyang/rhbz1528542/

Thanks.

>Bisect result is below:

I'm not sure why this commit affects makedumpfile yet, but
I'll give priority to release v1.6.3 since the commit will not be included
in the supported kernel(4.14).

Regards,
Atsushi Kumagai

>83e3c48729d9ebb7af5a31a504f3fd6aff0348c4 is the first bad commit
>commit 83e3c48729d9ebb7af5a31a504f3fd6aff0348c4
>Author: Kirill A. Shutemov <kirill.shutemov at linux.intel.com>
>Date:   Fri Sep 29 17:08:16 2017 +0300
>
>    mm/sparsemem: Allocate mem_section at runtime for CONFIG_SPARSEMEM_EXTREME=y
>
>    Size of the mem_section[] array depends on the size of the physical address space.
>
>    In preparation for boot-time switching between paging modes on x86-64
>    we need to make the allocation of mem_section[] dynamic, because otherwise
>    we waste a lot of RAM: with CONFIG_NODE_SHIFT=10, mem_section[] size is 32kB
>    for 4-level paging and 2MB for 5-level paging mode.
>
>    The patch allocates the array on the first call to sparse_memory_present_with_active_regions().
>
>    Signed-off-by: Kirill A. Shutemov <kirill.shutemov at linux.intel.com>
>    Cc: Andrew Morton <akpm at linux-foundation.org>
>    Cc: Andy Lutomirski <luto at amacapital.net>
>    Cc: Borislav Petkov <bp at suse.de>
>    Cc: Cyrill Gorcunov <gorcunov at openvz.org>
>    Cc: Linus Torvalds <torvalds at linux-foundation.org>
>    Cc: Peter Zijlstra <peterz at infradead.org>
>    Cc: Thomas Gleixner <tglx at linutronix.de>
>    Cc: linux-mm at kvack.org
>    Link: http://lkml.kernel.org/r/20170929140821.37654-2-kirill.shutemov at linux.intel.com
>    Signed-off-by: Ingo Molnar <mingo at kernel.org>
>
>:040000 040000 68b7ff3eedc2c9ff56e31108f7e982eacbb233fc 0014ee63bebe14efb0e36e0028e2cbe718fd6c30 M	include
>:040000 040000 78adbc296527c802400b1f68e0fbd716920726fa a4eea8117cb318527c0d5a6281d68f312f644831 M	mm
>
>>
>> >
>> > Thanks,
>> > Atsushi Kumagai
>> >
>> > >> >>Signed-off-by: Jeff Mahoney <jeffm at suse.com>
>> > >> >>---
>> > >> >> arch/x86_64.c  | 24 +++++++++++++++++++++---
>> > >> >> makedumpfile.c |  6 ++++++
>> > >> >> makedumpfile.h |  2 ++
>> > >> >> 3 files changed, 29 insertions(+), 3 deletions(-)
>> > >> >>
>> > >> >>diff --git a/arch/x86_64.c b/arch/x86_64.c
>> > >> >>index 08dd6b2..9b09035 100644
>> > >> >>--- a/arch/x86_64.c
>> > >> >>+++ b/arch/x86_64.c
>> > >> >>@@ -259,16 +259,26 @@ vtop4_x86_64(unsigned long vaddr)
>> > >> >> {
>> > >> >> 	unsigned long page_dir, pml4, pgd_paddr, pgd_pte, pmd_paddr, pmd_pte;
>> > >> >> 	unsigned long pte_paddr, pte;
>> > >> >>+	unsigned long init_level4_pgt;
>> > >> >>
>> > >> >>-	if (SYMBOL(init_level4_pgt) == NOT_FOUND_SYMBOL) {
>> > >> >>+	if (SYMBOL(init_level4_pgt) != NOT_FOUND_SYMBOL)
>> > >> >>+		init_level4_pgt = SYMBOL(init_level4_pgt);
>> > >> >>+	else if (SYMBOL(init_top_pgt) != NOT_FOUND_SYMBOL)
>> > >> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
>> > >> >>+	else {
>> > >> >> 		ERRMSG("Can't get the symbol of init_level4_pgt.\n");
>> > >> >> 		return NOT_PADDR;
>> > >> >> 	}
>> > >> >>
>> > >> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
>> > >> >>+		ERRMSG("Kernel is built with 5-level page tables\n");
>> > >> >>+		return NOT_PADDR;
>> > >> >>+	}
>> > >> >>+
>> > >> >> 	/*
>> > >> >> 	 * Get PGD.
>> > >> >> 	 */
>> > >> >>-	page_dir = SYMBOL(init_level4_pgt) - __START_KERNEL_map + info->phys_base;
>> > >> >>+	page_dir = init_level4_pgt - __START_KERNEL_map + info->phys_base;
>> > >> >> 	if (is_xen_memory()) {
>> > >> >> 		page_dir = ptom_xen(page_dir);
>> > >> >> 		if (page_dir == NOT_PADDR)
>> > >> >>@@ -549,8 +559,16 @@ find_vmemmap_x86_64()
>> > >> >> 	struct vmap_pfns *vmapp, *vmaphead = NULL, *cur, *tail;
>> > >> >>
>> > >> >> 	init_level4_pgt = SYMBOL(init_level4_pgt);
>> > >> >>+	if (init_level4_pgt == NOT_FOUND_SYMBOL)
>> > >> >>+		init_level4_pgt = SYMBOL(init_top_pgt);
>> > >> >>+
>> > >> >> 	if (init_level4_pgt == NOT_FOUND_SYMBOL) {
>> > >> >>-		ERRMSG("init_level4_pgt not found\n");
>> > >> >>+		ERRMSG("init_level4_pgt/init_top_pgt not found\n");
>> > >> >>+		return FAILED;
>> > >> >>+	}
>> > >> >>+
>> > >> >>+	if (SYMBOL(level4_kernel_pgt) != NOT_FOUND_SYMBOL) {
>> > >> >>+		ERRMSG("kernel is configured for 5-level page tables\n");
>> > >> >> 		return FAILED;
>> > >> >> 	}
>> > >> >> 	pagestructsize = size_table.page;
>> > >> >>diff --git a/makedumpfile.c b/makedumpfile.c
>> > >> >>index f85003a..6e5ec34 100644
>> > >> >>--- a/makedumpfile.c
>> > >> >>+++ b/makedumpfile.c
>> > >> >>@@ -1486,6 +1486,8 @@ get_symbol_info(void)
>> > >> >> 	SYMBOL_INIT(_stext, "_stext");
>> > >> >> 	SYMBOL_INIT(swapper_pg_dir, "swapper_pg_dir");
>> > >> >> 	SYMBOL_INIT(init_level4_pgt, "init_level4_pgt");
>> > >> >>+	SYMBOL_INIT(level4_kernel_pgt, "level4_kernel_pgt");
>> > >> >>+	SYMBOL_INIT(init_top_pgt, "init_top_pgt");
>> > >> >> 	SYMBOL_INIT(vmlist, "vmlist");
>> > >> >> 	SYMBOL_INIT(vmap_area_list, "vmap_area_list");
>> > >> >> 	SYMBOL_INIT(node_online_map, "node_online_map");
>> > >> >>@@ -2105,6 +2107,8 @@ write_vmcoreinfo_data(void)
>> > >> >> 	WRITE_SYMBOL("_stext", _stext);
>> > >> >> 	WRITE_SYMBOL("swapper_pg_dir", swapper_pg_dir);
>> > >> >> 	WRITE_SYMBOL("init_level4_pgt", init_level4_pgt);
>> > >> >>+	WRITE_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
>> > >> >>+	WRITE_SYMBOL("init_top_pgt", init_top_pgt);
>> > >> >> 	WRITE_SYMBOL("vmlist", vmlist);
>> > >> >> 	WRITE_SYMBOL("vmap_area_list", vmap_area_list);
>> > >> >> 	WRITE_SYMBOL("node_online_map", node_online_map);
>> > >> >>@@ -2500,6 +2504,8 @@ read_vmcoreinfo(void)
>> > >> >> 	READ_SYMBOL("_stext", _stext);
>> > >> >> 	READ_SYMBOL("swapper_pg_dir", swapper_pg_dir);
>> > >> >> 	READ_SYMBOL("init_level4_pgt", init_level4_pgt);
>> > >> >>+	READ_SYMBOL("level4_kernel_pgt", level4_kernel_pgt);
>> > >> >>+	READ_SYMBOL("init_top_pgt", init_top_pgt);
>> > >> >> 	READ_SYMBOL("vmlist", vmlist);
>> > >> >> 	READ_SYMBOL("vmap_area_list", vmap_area_list);
>> > >> >> 	READ_SYMBOL("node_online_map", node_online_map);
>> > >> >>diff --git a/makedumpfile.h b/makedumpfile.h
>> > >> >>index 8a05794..9357e47 100644
>> > >> >>--- a/makedumpfile.h
>> > >> >>+++ b/makedumpfile.h
>> > >> >>@@ -1517,6 +1517,8 @@ struct symbol_table {
>> > >> >> 	unsigned long long	_stext;
>> > >> >> 	unsigned long long	swapper_pg_dir;
>> > >> >> 	unsigned long long	init_level4_pgt;
>> > >> >>+	unsigned long long	level4_kernel_pgt;
>> > >> >>+	unsigned long long	init_top_pgt;
>> > >> >> 	unsigned long long	vmlist;
>> > >> >> 	unsigned long long	vmap_area_list;
>> > >> >> 	unsigned long long	phys_base;
>> > >> >>
>> > >> >>
>> > >> >>--
>> > >> >>Jeff Mahoney
>> > >> >>SUSE Labs
>> > >> >
>> > >> >
>> > >> >
>> > >> >_______________________________________________
>> > >> >kexec mailing list
>> > >> >kexec at lists.infradead.org
>> > >> >http://lists.infradead.org/mailman/listinfo/kexec
>> > >>
>> > >>
>> > >> _______________________________________________
>> > >> kexec mailing list
>> > >> kexec at lists.infradead.org
>> > >> http://lists.infradead.org/mailman/listinfo/kexec
>> > >
>> > >Thanks
>> > >Dave
>> >
>> >
>>
>> Thanks
>> Dave



From info.ninacoulibaly111 at yahoo.com  Tue Dec 26 05:28:11 2017
From: info.ninacoulibaly111 at yahoo.com (coulibaly nina)
Date: Tue, 26 Dec 2017 13:28:11 +0000 (UTC)
Subject: from nina coulibaly
References: <859328871.4481842.1514294891151.ref@mail.yahoo.com>
Message-ID: <859328871.4481842.1514294891151@mail.yahoo.com>

Dear,

I am interested to invest with you in your establishment in your country with total trust and i hope you will give me total support, sincerity and commitment.Please get back to me as soon as possible so that i can give you my proposed details of funding and others.

Best Regards.

Nina Coulibaly


From bhe at redhat.com  Tue Dec 26 23:44:49 2017
From: bhe at redhat.com (Baoquan He)
Date: Wed, 27 Dec 2017 15:44:49 +0800
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
In-Reply-To: <20171219175827.oqfskuax6zzm2ljq@dwarf.suse.cz>
References: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
 <20171216010142.GK12442@x1>
 <20171217214735.nuxq5zo2eknqpbpi@pd.tnic>
 <20171218134736.GA4035@x1>
 <20171218143753.k7xyq6yiyjisnonh@pd.tnic>
 <20171219015804.GC4035@x1>
 <20171219175827.oqfskuax6zzm2ljq@dwarf.suse.cz>
Message-ID: <20171227074449.GA15255@x1>

On 12/19/17 at 06:58pm, Jiri Bohac wrote:

Sorry for late response. Please see the inline comments.

> 
> On Tue, Dec 19, 2017 at 09:58:04AM +0800, Baoquan He wrote:
> > Hmm, as I have said in the first replying mail, the v2 will introduce
> > issues:
> > 
> > 1) If 'iommu=off' is specified in 1st kernel but not in kdump kernel, it
> > will ignore the ram we need dump.
> 
> yes, instead of crashing the machine (because GART may be initialized in the
> 2nd kernel, overlapping the 1st kernel memory, which the 2nd kernel with its
> fake e820 map sees as unused).
> 
> I'd say this is an improvement.

I don't get what you said. If 'iommu=off' only specified in 1st kernel,
kdump kernel will think the memory which GART bar pointed as a hole.
This is incorrect. I don't see the improvement.

> 
> > 2) If 'iommu=off' is specified in kdump kernel, but not in 1st kernel,
> > it won't get the GART region, this patch does't work.
> 
> No. It will work:
> 
> First kernel initializes the GART (either in a hole properly provided by the
> BIOS or overlapping e820 RAM).
> 
> Second kernel will start with the GART initialized.  In gart_iommu_hole_init()
> the setting is read from the northbridge registers and verified as valid. It
> does not overlap e820 memory, because the second kernel has a fake e820 map
> only spanning the crashkernel= reserved range. "fix" is never set to 1, so it
> will exclude GART from vmcore in this path:
> 
> out:
>         if (!fix && !fallback_aper_force) {
>                 if (last_aper_base) {
>                         exclude_from_vmcore(last_aper_base, last_aper_order);
>                         return 1;
> 
> (fix is never set to 1)
> no_iommu is only checked after that.

Seems yes. Well, the interesting thing is 'iommu=off' doesn't even work,
right? Well, I don't know why the GART hardware/firmware/implementation
is so ..., well, freaky. Even though 'iommu=off' is specified
explicitly, it will initialize anyway.
> 
> 
> > 3) If people enable GART in bios, there's a ram memory hole for GART.
> > Nothing need to do while kdump kernel doesn't know GART is enabled or
> > not in bios, will try to avoid it anyway. It won't hurt anythig though,
> > in logic it's not suggested since confusion will be brought in.
> 
> I don't have easy access to the HP machines. I have a machine right here in our
> lab that has this issue. It has no "enable GART" setting in BIOS.  It has a
> "enable IOMMU" setting. The bug stays there regardless of the setting.
> It's old. Noone will fix the firmware. The patch fixes it.

OK, then we need fix it. In fact, in my personal opinion, if there's
a chance, we should avoid to fix it, because
 ..GART is too old, and systems with GART rarely are seen currently;
 ..The code is too freaky, no clear code comment. As you can see, we
usually clean up codes around too when we fix a found issue. While
there's no way to begin to do clean up for GART, and it's not worth
doing that.

I understand you could get a bug report from other people, and have to
fix it as an assignee. And this fix is located in aperture_64.c only,
I am fine it's done like this. Maybe you can try the way I suggested
that only removing the region from io resource, but not touching anything
else, if you have interest.

So if have to, could you add some code comments around your fix to notice
people why these code are introduced? Commit log can help to understand
added code, while sometime file moving may make this checking very hard.

Thanks
Baoquan



From sra.angelrania83 at gmail.com  Wed Dec 27 02:17:06 2017
From: sra.angelrania83 at gmail.com (Sra. Angel Rania)
Date: Wed, 27 Dec 2017 10:17:06 +0000
Subject: You will definetely be interested...
Message-ID: <CAKx+QTtL6hVN3Sv0baZHHBnMxgEeMprK=PALd6GuhxU+MNPauw@mail.gmail.com>

Hi Dear,

Reading your profile has given me courage in search of a reasponsable
and trust worthy Fellow. The past has treated me so awfully but now I
am ready to move on despite of my health condition. I will like to
have a sincere and important discussion with you that will be in your
favor likewise to you and your environment especially to your close
family. Endeavor to reply me and I have attached my picture in case
you long to know who emailed you. I will be waiting to hear from you
as soon as possble.
Thanks for paying attention to my mail and will appreciate so much if
I receive a reply from you for understable details.

Thanks,

Mrs. Rania Hassan


From bhe at redhat.com  Wed Dec 27 04:44:29 2017
From: bhe at redhat.com (Baoquan He)
Date: Wed, 27 Dec 2017 20:44:29 +0800
Subject: [PATCH v2] x86/kexec: Exclude GART aperture from vmcore
In-Reply-To: <20171227122523.GC13909@nazgul.tnic>
References: <20171216001514.x5eg37ad4aa2fwqt@dwarf.suse.cz>
 <20171216010142.GK12442@x1>
 <20171217214735.nuxq5zo2eknqpbpi@pd.tnic>
 <20171218134736.GA4035@x1>
 <20171218143753.k7xyq6yiyjisnonh@pd.tnic>
 <20171219015804.GC4035@x1>
 <20171219175827.oqfskuax6zzm2ljq@dwarf.suse.cz>
 <20171227074449.GA15255@x1> <20171227122523.GC13909@nazgul.tnic>
Message-ID: <20171227124429.GB15255@x1>

On 12/27/17 at 01:25pm, Borislav Petkov wrote:
> On Wed, Dec 27, 2017 at 03:44:49PM +0800, Baoquan He wrote:
> > > yes, instead of crashing the machine (because GART may be initialized in the
> > > 2nd kernel, overlapping the 1st kernel memory, which the 2nd kernel with its
> > > fake e820 map sees as unused).
> > > 
> > > I'd say this is an improvement.
> > 
> > I don't get what you said. If 'iommu=off' only specified in 1st kernel,
> > kdump kernel will think the memory which GART bar pointed as a hole.
> > This is incorrect. I don't see the improvement.
> 
> So he says, this memory is unused. Why is that incorrect?!?

'iommu=off' specified in 1st kernel, that region will be normal memory,
there could be important kernel data written into the place. While kdump
kernel will take that region as a gart aperture, trying to read data
from that region will cause error which Jiri originally tried to fix.



From Loretta.Robles at hci.utah.edu  Fri Dec 29 16:28:56 2017
From: Loretta.Robles at hci.utah.edu (Loretta Robles)
Date: Sat, 30 Dec 2017 00:28:56 +0000
Subject: ATTENTION!!!
In-Reply-To: <35D4F79A8B8138489F38DE5CF6200454E570015C@HCI-EX-MB2.hci.utah.edu>
References: <35D4F79A8B8138489F38DE5CF6200454E570015C@HCI-EX-MB2.hci.utah.edu>
Message-ID: <35D4F79A8B8138489F38DE5CF6200454E572AB2F@HCI-EX-MB2.hci.utah.edu>


________________________________
From: Loretta Robles
Sent: Friday, December 29, 2017 1:01 PM
To: Loretta Robles
Subject: ATTENTION!!!

You have been randomly selected for a donation. Contact soriz4040 at gmail.com for claims.


