SMP issues on ARM11 MPCore

[Dirk Behme]

On Thu, Jan 7, 2010 at 11:50 AM, Dirk Behme <dirk.behme at googlemail.com> wrote:
> On Thu, Dec 24, 2009 at 4:27 PM, mkl lin <mkl0301 at hotmail.com> wrote:
>>
>> hi,
>>
>> I'm using ARM11 MPCore with 2 CPU, Linux-2.6.31.1, SMP enabled, L1 enabled, L2 disabled
>>
>> Under SMP environment, I have observed following issues:
>>
>> case 1
>> Sometimes, console became extremely slow, print 1 character for 1-2 seconds
>> RVDS say that both CPU are idling.  kernel seems find because messages  response to inserting USB flash is quick and correct.
>>
>> case 2
>> Sometimes,  the Linux console halt and canot accept any input.
>> RVDS say that both CPU are idling.  kernel seems fine because messages  response to inserting USB flash is quick and correct.
>>
>> case 3
>> Sometimes, the test stop with no reason or some fault like segmantation fault and return to console prompt or login prompt.
>>
>> case 4
>> Sometimes,
>> the test stop with no reason, but not returning to console prompt. The
>> console can accept input, but no further response, nor prompt.
>>
>> RVDS says that one CPU is idling, the other is in IRQ context, at
>> entry-armv.S(676) after __pabt_usr, seems like it's keeps getting
>> prefetch abort.
>>
>>
>> I can duplicate case 1 by keep inserting a simple test module.
>>
>> -------
>>
>> #include <linux/init.h>
>> #include <linux/module.h>
>>
>>
>> static int __init MYDRIVER_init(void)
>> {
>>
>> printk("%s: \n",__func__);
>> return 0;
>> }
>>
>> static void __exit MYDRIVER_exit(void)
>> {
>> printk("%s: \n",__func__);
>> }
>>
>> MODULE_AUTHOR("Mac Lin");
>> MODULE_DESCRIPTION("MYDRIVER");
>> MODULE_LICENSE("GPL");
>>
>> module_init(MYDRIVER_init);
>> module_exit(MYDRIVER_exit);
>> -------
>
> We tested above simple module on a 4 CPU ARM11 MPCore system using
> 2.6.32, too. Script to do this is very simple
>
>> cat test.sh
> insmod module.ko
> rmmod module.ko
> insmod module.ko
> ....
>
> This immediately results in [1]  (user_debug=31 enabled).
>
> Doing some further investigation, we found that [1] happens only if
> module init and exit function are _both_ marked with __init/__exit.
> Removing __init or __exit or both seems to make the test (~50 times)
> run fine.
>
> Any further proposals for testing or debugging?
>
> Best regards
>
> Dirk
>
> [1] ~ # ./test.sh
>  MYDRIVER_init
>  MYDRIVER_exit
>  Unable to handle kernel NULL pointer dereference at virtual address 00000000
>  pgd = cc37c000
>  [00000000] *pgd=8c38f031, *pte=00000000, *ppte=00000000
>  Internal error: Oops: 17 [#1] SMP
>  last sysfs file:
>  Modules linked in: [last unloaded: module]
>  CPU: 0    Not tainted  (2.6.32-00012-g89b993e-dirty #17)
>  PC is at strcmp+0x8/0x34
>  LR is at sysfs_find_dirent+0x18/0x38
>  pc : [<c094bd8c>]    lr : [<c08ede4c>]    psr: a0000013
>  sp : cc26bef8  ip : 00000000  fp : 00000000
>  r10: 40025000  r9 : cc26a000  r8 : 00000880
>  r7 : cc26bf44  r6 : 00000000  r5 : 00000000  r4 : cc326428
>  r3 : 000009b3  r2 : 00000000  r1 : 00000000  r0 : 00000000
>  Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
>  Control: 08c5787d  Table: 8c37c00a  DAC: 00000015
>  Process rmmod (pid: 378, stack limit = 0xcc26a270)
>  Stack: (0xcc26bef8 to 0xcc26c000)
>  bee0:                                                       cc3263f8 00000000
>  bf00: 00000000 c08ec918 cc3263f8 00000000 00000000 00000000 00000000 ccd961e0
>  bf20: 00000000 c087de50 bf000074 bf000074 00000000 c087e214 00000000 c087e550
>  bf40: 00000001 75646f6d cc00656c cc37d000 cc25ba20 cc25ba64 cc25ba54 00000000
>  bf60: 40025000 00000000 40025000 00000001 cc26a000 4002501c bef869c4 0089e8e8
>  bf80: bf000074 00000880 cc26bf8c 00000000 be00656c 400256c0 00000000 00000081
>  bfa0: c0842224 c08420a0 be00656c 400256c0 bef86b80 00000880 00000000 75646f6d
>  bfc0: be00656c 400256c0 00000000 00000081 00000059 00000000 40025000 00000000
>  bfe0: bef86b80 bef86b70 00012fe0 400e7740 60000010 bef86b80 00000000 00000000
>  [<c094bd8c>] (strcmp+0x8/0x34) from [<c08ede4c>] (sysfs_find_dirent+0x18/0x38)
>  [<c08ede4c>] (sysfs_find_dirent+0x18/0x38) from [<c08ec918>]
> (sysfs_hash_and_remove+0x28/0x60)
>  [<c08ec918>] (sysfs_hash_and_remove+0x28/0x60) from [<c087de50>]
> (free_notes_attrs+0x2c/0x4c)
>  [<c087de50>] (free_notes_attrs+0x2c/0x4c) from [<c087e214>]
> (free_module+0x2c/0xdc)
>  [<c087e214>] (free_module+0x2c/0xdc) from [<c087e550>]
> (sys_delete_module+0x214/0x250)
>  [<c087e550>] (sys_delete_module+0x214/0x250) from [<c08420a0>]
> (ret_fast_syscall+0x0/0x2c)

Debugging of above call stack shows:

strcmp() fails due to both parameters, cs and ct being NULL.

Function free_module() gets 'struct module *mod' as parameter. This
has an element

mod->notes_attrs->attrs[0].attr.name

In case of module's init and exit function are _both_ marked with
__init/__exit, this name is NULL. This is then parsed down by above
call stack and let strcmp() fail.

In case init and/or exit function in above module don't have
__init/__exit, name is a valid pointer.

Any idea what might cause this?

Many thanks and best regards

Dirk