[3389] in linux-scsi channel archive
Re: Race in AHA1542 detection
daemon@ATHENA.MIT.EDU (Keith Owens)
Mon Feb 23 18:55:34 1998
From: Keith Owens <kaos@ocs.com.au>
To: Eric Youngdale <eric@andante.jic.com>
cc: linux-scsi@vger.rutgers.edu
In-reply-to: Your message of "Mon, 23 Feb 1998 09:09:46 CDT."
<Pine.LNX.3.95.980223090042.16823E-100000@andante.jic.com>
Date: Tue, 24 Feb 1998 09:12:20 +1100
On Mon, 23 Feb 1998 09:09:46 -0500 (EST),
Eric Youngdale <eric@andante.jic.com> wrote:
> First of all, I don't really know what the APIC patches were for.
>Could someone explain this, and tell me when it went into the kernel?
>Also, what does pirq=0 mean?
Copied to the list in case it is useful for other SCSI drivers. Any
driver relying on cli/sti may have problems with IO-APIC.
Ingo Molnar's patches to allow interrupts to be serviced by *any* cpu
on an SMP machine. Once more step (a big one) in multi threading the
kernel. Experimental patches came out around 2.1.84, it is now
integrated into the kernel. pirq=0 was originally required to activate
this feature for my motherboard, it is no longer required. See
Documentation/IO-APIC.txt. /proc/interrupts looks like this
Without apic. All interrupts on one processor.
CPU0 CPU1
0: 8075 0 XT PIC timer
1: 147 0 XT PIC keyboard
2: 0 0 XT PIC cascade
3: 3 0 XT PIC serial
4: 4 0 XT PIC serial
10: 7 0 XT PIC aha1542
11: 5084 0 XT PIC ncr53c8xx
13: 4 0 XT PIC fpu
15: 2890 0 XT PIC NE2000
NMI: 0
IPI: 0
With apic. Most interrupts can be handled by either cpu.
CPU0 CPU1
0: 10573 31038 IO-APIC timer
1: 189 712 IO-APIC keyboard
2: 0 0 XT PIC cascade
3: 1 2 IO-APIC serial
4: 5 0 IO-APIC serial
10: 6 8 IO-APIC aha1542
11: 6388 6284 IO-APIC ncr53c8xx
13: 4 0 XT PIC fpu
15: 35 46 IO-APIC NE2000
NMI: 0
IPI: 0
> Nonetheless, the thing that confuses me is that the mailboxes are
>set up before the interrupt handler is registered. Thus I cannot quite
>see how things could get fouled up in this fashion, but until I know more
>about what the APIC stuff is doing, I am flying blind.
>
> Maybe I am reading the stuff below wrong, but it almost looks like
>both processors are running the aha1542_detect() routine. If this is
>true, it is almost certainly not correct.
setup_mailboxes, aha1542_out, aha1542_detect are all running on cpu(0),
pid(1). aha1542_detect calls request_irq on cpu(0) then proceeds to do
the DMA work. With apic, after the irq has been assigned, cpu(1) takes
an interrupt before aha1542_detect has ended.
c01c1d27 setup_mailboxes +<13/e4> (6.31) cpu(0) pid(1)
c01c1043 aha1542_out +<13/e0> (1.17) cpu(0) pid(1)
c010cfa1 __global_save_flags +<d/40> (1.34) cpu(0) pid(1)
c010ce23 __global_cli +<f/13c> (96.23) cpu(0) pid(1)
c010cfe1 __global_restore_flags +<d/6c> (0.93) cpu(0) pid(1)
c010cf5d __global_sti +<d/44> (45.28) cpu(0) pid(1)
c010cfa1 __global_save_flags +<d/40> (1.36) cpu(0) pid(1)
c010ce23 __global_cli +<f/13c> (2.99) cpu(0) pid(1)
c010d5ae do_IRQ +<e/64> (2.77) cpu(1) pid(0)
c01c250f aha1542_detect +<2a7/42c> (2.98) cpu(0) pid(1)
c010d4ac do_ioapic_IRQ +<10/104> (2.62) cpu(1) pid(0)
c010d700 request_irq +<10/8c> (2.67) cpu(0) pid(1)
c0113afd ack_APIC_irq +<d/28> (52.23) cpu(1) pid(0)
c012bd6f kmalloc_wrapper +<f/3c> (1.30) cpu(0) pid(1)
c0128170 real_kmalloc +<10/188> (2.83) cpu(0) pid(1)
c012bad4 alloc_addr +<10/100> (2.56) cpu(0) pid(1)
c010d617 setup_x86_irq +<13/ec> (1.52) cpu(0) pid(1)
c010cfa1 __global_save_flags +<d/40> (1.07) cpu(0) pid(1)
c010ce23 __global_cli +<f/13c> (2.47) cpu(0) pid(1)
c010c871 set_8259A_irq_mask +<d/44> (1.42) cpu(0) pid(1)
c010c8b5 unmask_generic_irq +<d/48> (1.20) cpu(0) pid(1)
c0113105 enable_IO_APIC_irq +<11/3c> (1.04) cpu(0) pid(1)
c01130b5 io_apic_read +<d/24> (8.49) cpu(0) pid(1)
c01130d9 io_apic_write +<d/28> (1.27) cpu(0) pid(1)
c010cfe1 __global_restore_flags +<d/6c> (1.02) cpu(0) pid(1)
c010ce23 __global_cli +<f/13c> (1.88) cpu(0) pid(1)
c01c2551 aha1542_detect +<2e9/42c> (1.57) cpu(0) pid(1)
c011972a request_dma +<e/50> (2.29) cpu(0) pid(1)
c01c25a1 aha1542_detect +<339/42c> (4.72) cpu(0) pid(1)
c01c2624 aha1542_detect +<3bc/42c> (0.87) cpu(0) pid(1)
c010cfe1 __global_restore_flags +<d/6c> (0.95) cpu(0) pid(1)
c010cf5d __global_sti +<d/44> (1.49) cpu(0) pid(1)
**** Flags restored, cpu(0) is still in aha1542_detect and
**** aha1542_intr_handle starts running on cpu(1).
c01c262f aha1542_detect +<3c7/42c> (2.71) cpu(0) pid(1)
c010d050 handle_IRQ_event +<10/60> (2.92) cpu(1) pid(0) <=== interrupt on cpu(1)
c01c2634 aha1542_detect +<3cc/42c> (5.07) cpu(0) pid(1)
c01c1437 aha1542_intr_handle +<13/2a8> (7.68) cpu(1) pid(0)
c0120300 request_region +<10/8c> (6.15) cpu(0) pid(1)
c010cfa1 __global_save_flags +<d/40> (3.10) cpu(1) pid(0)
c0120293 find_gap +<13/70> (3.25) cpu(0) pid(1)
c010ce23 __global_cli +<f/13c> (4.88) cpu(1) pid(0)
c010cfa1 __global_save_flags +<d/40> (4.36) cpu(0) pid(1)
c010cfe1 __global_restore_flags +<d/6c> (2.83) cpu(1) pid(0)
c010ce23 __global_cli +<f/13c> (2.68) cpu(0) pid(1)
c011ae8b printk +<13/184> (2.29) cpu(1) pid(0)
The normal fix to this problem is to use spinlocks instead of cli/sti,
that is Linus's preferred direction. I can do the patch, but where to
store the spinlock? Should it be global for all 1542's in a machine or
local to each card? My feeling is local to each card which probably
means adding a spinlock to the Scsi_Host structure. Before I do that
and affect every other SCSI driver, it might be a good idea to agree on
how to handle apic.
