Mailing Lists: Apple Mailing Lists
Image of Mac OS face in stamp
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Zero-copy PXI-X driver for G5 Mac, process access to 7 GB memory

Josh de Cesare writes:
> Drew,
> While I can not comment on when/if the AppleMacRiscPCI driver
> will/would be released into open source, I can comment a bit about
> how the DART works. Really we don't want developers to worry about
> the specifics of how the DART works, because it can change. The
> IOMapper is the interface that we are supporting.
> The DART translates all memory accesses from
> HyperTransport/PCI devices in the range of 0 to 2 GB to a 36 bit
> physical memory address. The translations are managed by page. The
> driver for the DART manages the 2 GB space with a power of 2
> allocator. This allows most allocations/mappings the be handle in a
> single contiguous extent. If a client requests a disk read on 1 MB
> of its memory, those pages could be anywhere in physical memory. The
> Mapper will use the DART to create a single 1 MB extent that a PCI
> device can use for DMA. Even though the memory is not really
> contiguous, the driver for the PCI device will usually be able to
> treat it as if it is contiguous.
> The DART is only able to map 2GB of memory for active DMA. It
> is not possible to give a HT/PCI device access to more than 2 GB of
> memory at a given time.
> If you can, please describe the environment you are working
> in. There may be ways around the limitations of the current system.
>
> Josh

Consider a cluster of a 8GB SMP machine, each with 2 4GB processes
that want to make most or all of their address space available to
other procesess on the network for OS-bypass, remote memory access
(via Myrinet, or some other high performance interconnect). Rather
than being able to make (4GB - epsilon) available at the start of the
app, they'll need to call into the device driver to map and unmap
their remote memory. Or, more likely, they'll just have to make a
small segment of memory available and copy to/from it.

This is the sort of thing that works seemlessly on other large memory
platforms -- x86, AMD64, alpha, itanium, etc. The only exception is
Sun (and they've recently corrected it in their latest PCI chipsets).

The alpha (using tru64 and linux) has an IOMMU similar to the DART.
It can facilitate efficent scatter/gather DMA as you describe above.
It has similar limitations. Only a "small" amount of memory can be
mapped through it at any one time.

However, if you indicate to the tru64 ddi that you're a 64-bit capable
device, you get to use the "monster window". When you translate a
virtual address into a DMA address, you end up with an address that
ALWAYS has a high bit set (1<<40 on most chipsets). The PCI devices
DMA's to/from that address. The iommu notices the high bit, strips it
off, and treats the remaining address as a real physical address.

Is there a magic bit that can be setup in the DART so as to enable an
alpha-style monster window? Or does the DART itself only understand
32-bit PCI addresses?

I'd much rather just be able to read the DART docs. That would save a
lot of bandwidth. I suspect that it can do what I want, but it would
be nice if I knew... ;)


Drew
_______________________________________________
darwin-drivers mailing list | email@hidden
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-drivers
Do not post admin requests to the list. They will be ignored.
References: 
 >Re: Zero-copy PXI-X driver for G5 Mac, process access to 7 GB memory (From: Josh de Cesare <email@hidden>)



Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Contact Apple | Terms of Use | Privacy Policy

Copyright © 2007 Apple Inc. All rights reserved.