Apple

On Mar 5, 2015, at 7:30 AM, Paul Davis <email@hidden> wrote:

Yes, the block size dependence is clear and shared across all platforms (though the shape of the curve is different depending on the platform and the filesystem).

I modified the test to print out the total time and number of seeks:

printf ("seeks: %llu: bytes: %llu total_time: %f\n", cnt * nfiles, (nfiles * _read), total_time/1000000.0);

WD Red on USB 3
122.768 MB/sec read (128731578 bytes/sec)

256 files 10MB each -> 2684354560 total bytes to read
Time required to stream 2684354560 bytes -> 20.85 seconds

Blocksize 2621440
Number of seeks 1024
Time taken: 27.16 seconds
Time available for seeks: 27.16 - 20.85 = 6.31 seconds
6.31 / 1024 -> 6.2 ms seek time

Blocksize 65536
Number of seeks 40960
Time taken: 365.88 seconds
Time available for seeks: 365.88 - 20.85 = 345.03 seconds
345.03 / 40960 -> 8.42 ms seek time

Blocksize 32768
Number of seeks 81920
Time taken: 928.40 seconds
Time available for seeks: 928.40 - 20.85 = 907.55 seconds
907.55 / 81920 -> 11.078 ms seek time

So, with smaller and smaller block sizes, the seek time is creeping up a bit. However, it’s still well within spec of the drive. Also there might be some things unaccounted for in the simplistic calculation, such as adjacent track seeking. But that would only get you closer to the average seek time for all block sizes.

How is it that the other OS’s can do it faster with caching disabled? I maintain that there must be some form of caching going on.

Would you mind posting the drive model, filesystem, and the output of the test, with the above printf included?

However, on OS X, even when you use "optimal" block sizes (which appear to be on the order of 2MB-4MB (roughly twice as large as is necessary on Windows or Linux to get the the disk streaming capacity), the system is *still* not capable of maintaining sustained bandwidth necessary for high track counts, because it drops *way* below that

bandwidth for too long to reasonably buffer. At least that's what we've seen from testing so far.

I think that’s a different problem then. Have you tried higher thread priorities? Why does it seem to work on mine? Or is this test just averaging through it? We could modify the test to print out a list of times and then graph that.

Thanks for trying the test! Yours are the first set of results we've seen that are close to what you'd expect, which is strange.

BTW if anyone wants to try this without the glib dependency compile with:

gcc -o readtest readtest.c

but first replace this:

#include <glib.h>

with this:

#if __APPLE__

#import <mach/mach_time.h> // for mach time

#import <sys/syslimits.h> // for PATH_MAX

typedef int64_t gint64;

// From gmain.c

gint64 g_get_monotonic_time (void)

{

static mach_timebase_info_data_t timebase_info;

if (timebase_info.denom == 0)

{

/* This is a fraction that we must use to scale

* mach_absolute_time() by in order to reach nanoseconds.

* We've only ever observed this to be 1/1, but maybe it could be

* 1000/1 if mach time is microseconds already, or 1/1000 if

* picoseconds. Try to deal nicely with that.

mach_timebase_info (&timebase_info);

/* We actually want microseconds... */

if (timebase_info.numer % 1000 == 0)

timebase_info.numer /= 1000;

else

timebase_info.denom *= 1000;

/* We want to make the numer 1 to avoid having to multiply... */

if (timebase_info.denom % timebase_info.numer == 0)

{

timebase_info.denom /= timebase_info.numer;

timebase_info.numer = 1;

}

else

{

/* We could just multiply by timebase_info.numer below, but why

* bother for a case that may never actually exist...

* Plus -- performing the multiplication would risk integer

* overflow. If we ever actually end up in this situation, we

* should more carefully evaluate the correct course of action.

mach_timebase_info (&timebase_info); /* Get a fresh copy for a better message */

fprintf (stderr, "Got weird mach timebase info of %d/%d. Please file a bug against GLib.",

timebase_info.numer, timebase_info.denom);

}

return mach_absolute_time () / timebase_info.denom;

}

#else

#include <glib.h>

#endif

References:
	>Re: large scale (audio) file I/O on OS X : help or insight requested (From: Hamilton Feltman <email@hidden>)
	>Re: large scale (audio) file I/O on OS X : help or insight requested (From: Paul Davis <email@hidden>)