Dan Creswell wrote:
>The trouble is that each OS is exhibiting different policies and a naive
>user is unaware of these different policies leading to assumptions about
>performance as perhaps demonstrated by this thread. Equally, the
>original poster may actually be hitting a performance issue. How do we
>know which it is?
We can't tell much. There's not nearly enough information about the
original poster's machine. We don't know the machine's model,
configuration, OS version, or Java version.
However, based on my tests with a puny 233 MHz G3 iMac, the originally
reported read-speed figures strike me as exceptionally low. As I wrote
earlier:
Expressing the performance as MB/s, and assuming 100KB
for each file (on average):
250 ms is about 0.40 MB/s
450 ms is about 0.22 MB/s
Those rates aren't even half the uncached read speed I got with files on a
USB thumb-drive plugged into a USB 1.0 bus on a very puny machine (Bondi
iMac, 233 MHz G3, 160 MB RAM). I purposely chose the USB device because it
was the slowest thing I had (due to bus speed, not device speed). Yet it's
over twice as fast at JUST uncached reads. Something else is most likely
making a major contribution to the poster's original speed problem. But
what? It could be anything. It may even be pathological swapping, which
isn't under a Java app's control at all.
I did my tests with a test app pretty much the same as the first benchmark
posted in this thread. I mainly added more reported results, and a single
shared buffer so GC wouldn't trigger inopportunely.
I think the actual read() speed could only be significant to the original
poster if it's an *exceedingly* slow disk, AND the data is uncached. That
seems extraordinary (or perhaps it's just a lousy network share).
I strongly suspect the primary factor for the actual original problem (slow
cached image files) is something else entirely, but without further
information I can't even begin to guess what it might be, nor how to remedy
it.
>The above test results, therefore, are interesting ...
No argument there.
>...because it suggests
>that Linux aggressively caches recently written data whilst OS X is more
>resistant...
It depends on what you mean by "aggressive" and "resistant".
I see no evidence of a difference in aggressiveness between the two
configurations. They both seem to have very similar strategies, if not
identical ones:
- They both use unallotted RAM as extra disk-cache.
- They both cache write-then-read from files.
- They both cache initial reads so re-reads don't hit the disk at all.
- They both have lazy writeback from cache.
- They both have a 'sync' to force a full media write.
The main difference I see is that the Linux config is simply faster at
moving data from the disk cache into the Java byte[] when using read().
I'd call that a difference in memory-to-memory copy speed, not a difference
in aggressiveness or resistance.
I also think it's interesting how the read-and-check times are similar
between the two machines, implying there's not much difference in JVM speed
for the code that does the check. Then if you subtract out the time taking
doing just reads (read-and-ignore), it tends to indicate that the Mac OS X
JVM is computing faster yet is reading slower. Other interpretations are
also possible.
The calculated numbers represent the time in ms to perform the check itself
(the doCheck() method) over all bytes read (8MB * 51 = 408MB):
88 - 32 = 56 for G5 config
79 - 6 = 73 for Linux config
The read-and-ignore difference might be only a Java difference, subject to
exactly how read() is implemented in each JVM. Nothing says it has to call
the C function fread(), or read(), or anything else.
Or it might be an OS difference, subject to how disk blocks are read in,
cached, and copied to Java's buffers. The other postings discussin grep's
speed tends to support this, or the next possibility...
It might be largely hardware-dependent, subject to RAM speed, DMA speed,
bus speed, and other hardware factors. Similar CPU clock-rate is no
guarantee that the rest of a system has corresponding speed.
Or it might be a combination of the above factors.
>In conclusion, whilst I think your points are valid your delivery comes
>across somewhat aggressive (whether you intend it or not) ..
Aggressive or merely obnoxious? ;-) Point taken, though.
If I offended Mr. McDougall by my snarkiness, I apologize.
No offense was intended.
If any of my other comments offended anyone, I apologize.
No offense was intended.
To further this constructively, here's the modifed disk-read test I used,
and the file-creating app I wrote to create the test-files in the first
place. They're both pretty simple and straightforward, not comprehensive
or exhaustive. All modifications are released into the public domain,
as-is and without warranty.
- - - - -
import java.io.*;
import java.text.*;
public class FileIO
{
public static DecimalFormat DP2 = new DecimalFormat( "0.00" );
public long length; // in bytes, total amount read
public long elapsed; // elapsed time in ms
public double rate; // in MB/s
private byte[] buf;
public
FileIO( byte[] buf )
{ this.buf = buf; }
/** Do test, report results, leave public fields with results. */
public void
test( File testable )
{
if ( ! testable.isFile() )
return;
try
{
elapsed = testRead( testable );
int readKB = (int) (length / 1024);
int bufKB = buf.length / 1024;
String speed = DP2.format( rate );
System.out.print ("read: "+ readKB + " KB in " + elapsed + " ms" );
System.out.print( ", by " + bufKB + " KB" );
System.out.print( ", at " + speed + " MB/s" );
System.out.println();
}
catch ( IOException why )
{ why.printStackTrace( System.out ); }
}
/** Read entire file, doing nothing with the data (read-and-ignore). */
private long
testRead( File readable )
throws IOException
{
// Opening the file is NOT measured in elapsed time.
InputStream in = new FileInputStream( readable );
this.length = readable.length();
byte[] buffer = buf;
int block = buffer.length;
// A block is smaller of the buffer's length or the file's length.
if ( block > length )
block = (int) length;
// Start timing...
long millis = System.currentTimeMillis();
// Always read until EOF returned from read().
while ( in.read( buffer, 0, block ) >= 0 )
{ ; } // empty loop body
this.elapsed = System.currentTimeMillis() - millis;
// CAUTION: file not closed if read() throws exception.
in.close();
// Calculate rate as MB/s
double megabytes = (double) length / (1024 * 1024);
double seconds = elapsed / 1000.0D;
this.rate = megabytes / seconds;
return ( elapsed );
}
/** Args provide one file or directory to test. */
public static void
main( String[] args )
{
if ( args.length != 1)
{
System.out.println( "## Please provide a file or directory to test." );
System.exit( 1 );
}
// Common 2 MB buffer. ## FIXME: get size in KB from a property.
byte[] buf = new byte[ 2 * 1024 * 1024 ];
// Aggregate statistics.
// longEnough is to avoid Infinity MB/s or serious jitter errors
int longEnough = 10; // measured in ms
int count = 0;
double min, max, sum;
min = 1E99;
max = sum = 0;
File[] files = listFiles( new File( args[0] ) );
for ( int i = 0; i < files.length; ++i )
{
FileIO test = new FileIO( buf );
test.test( files[ i ] );
if ( test.elapsed >= longEnough )
{
double rate = test.rate;
++count;
sum += rate;
if ( rate > max )
max = rate;
if ( rate < min )
min = rate;
}
}
System.out.print ("# overall -- min: "+ DP2.format( min ) );
System.out.print( ", mean: " + DP2.format( sum / count ) );
System.out.print( ", max: " + DP2.format( max ) + " MB/s" );
System.out.println();
identify();
}
/** Compatible with JDK 1.1 for hysterical raisins. */
private static File[]
listFiles( File dir )
{
File[] files;
if ( dir.isDirectory() )
{
String[] names = dir.list();
files = new File[ names.length ];
for ( int i = 0; i < names.length; ++i )
{ files[ i ] = new File( dir, names[ i ] ); }
}
else
files = new File[] { dir };
return ( files );
}
/** Identify OS and JVM. */
private static void
identify()
{
String jvm = System.getProperty( "java.version", "?" );
jvm = System.getProperty( "java.runtime.version", jvm );
String os = System.getProperty( "os.name", "OS" );
String osv = System.getProperty( "os.version", "?" );
System.out.println( "# on " + os + ": " + osv + ", Java: " + jvm );
}
}
- - - - -
import java.io.*;
public class FileMake
{
/** Args provide one pathname pattern to create files. */
public static void
main( String[] args )
{
// optional: -Dkb=FileSizeInKB -Dcount=NumberOfFiles
int kb = Integer.getInteger( "kb", 100 ).intValue();
int count = Integer.getInteger( "count", 10 ).intValue();
String path = args[ 0 ];
byte[] buf = new byte[ kb * 1024 ];
System.out.println( "Creating " + count + " files of " + kb + " KB" );
for ( int i = 0; i < count; ++i )
{ fill( path + "-" + i, buf ); }
}
private static void
fill( String path, byte[] buf )
{
try
{
FileOutputStream out = new FileOutputStream( path );
out.write( buf );
out.close();
}
catch ( IOException why )
{ why.printStackTrace( System.out ); }
// ## FIXME does not close file on failures
}
}
- - - - -
-- GG
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Java-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/java-dev/email@hidden
This email sent to email@hidden
