On Feb 29, 2012, at 12:00 PM, Chris Murphy < email@hidden> wrote: I've been curious about the difference between disk and rdisk for some time, when it comes to performance. While dd is not my metric for overall performance, I notice that when using dd to create and image from a disk or volume, or write an image onto a disk or volume, there is a massive difference in performance. Maybe twice the data rate using rdisk compared to disk, across all forms of media, SSD, HDD, and even USB sticks.
One of the key differences between /dev/disk and /dev/rdisk, when you access them from user space, is that /dev/disk is buffered. The read/write path for /dev/disk breaks up the I/O into 4KB chunks, which it reads into the buffer cache, and then copies into the user space buffer (and then issues the next 4KB read...). This is nice in that you can do unaligned reads and writes, and it just works. In contrast, /dev/rdisk basically just passes the read or write straight to the device, which means the start and end of the I/O need to be aligned on sector boundaries.
If you do a read or write larger than one sector to /dev/rdisk, that request will be passed straight through. The lower layers may break it up (eg., USB breaks it up into 128KB pieces due to the maximum payload size in the USB protocol), but you generally can get bigger and more efficient I/Os. (When streaming, like via dd, 128KB to 1MB are pretty good sizes to get near-optimal performance on current non-RAID hardware.)
The caching being done by /dev/disk's read and write paths is very simple, almost brain dead. It caches even if not strictly necessary (like if the device could memory map and directly transfer into your app's buffer). It does small (4KB) I/Os, which leads to a lot of per-I/O overhead. It does not do any read ahead or write behind. However, df reports mounted volumes using disk, not rdisk, yet they don't seem to exhibit the same performance "penalty" when using Finder, or some other application, to read/write files.
That's because the file system in the kernel (or a KEXT) doesn't go through the same caching layer that a read() or write() of the /dev/disk does. In effect, mounting a file system on /dev/disk gets the device information, and ends up issuing I/Os to the underlying driver. There is a caching layer (known as the Cluster I/O layer) inside the kernel. Local file systems generally go through the Cluster I/O layer rather than issuing I/Os to the device directly (at least for file content). Cluster I/O can generate large, contiguous I/Os (as long as the file system is good about telling Cluster I/O which pieces of the file are contiguous). It also does read ahead and write behind. If your I/O buffer is properly aligned, it can even arrange for the device to store the data directly in the memory pages where your buffer is mapped, avoiding an extra cross-address-space memory copy.
Finder is good about issuing non-cached I/Os and supplying properly aligned buffers during a file copy. This means that the data is read from one device directly into the buffer, and written straight from there to the destination device. This generates big, efficient I/Os and eliminates a lot of memory copies, as well as avoiding polluting the buffer cache with data that is unlikely to be used again soon. I do not require an exhaustive explanation, it's more a curiosity than needing to apply something in practice.
Curiosity is good! :-)
-Mark
|