Mailing Lists: Apple Mailing Lists
Image of Mac OS face in stamp
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Lightning fast file reader?

Along these same lines I am trying to write a "file engine" to read our
custom data files. The first try was to completely MemoryMap the entire
file which generally worked until I had a few 500MB files happen back
to back. That is when I found out that just because I null out the
memory map and call to garbage collect the actual memory that was used
outside the JVM is NOT released in a reasonable amount of time. That
lead to Ram -> HD swapping and the "engine" slowed to a crawl. So I
went searching around the net for something else and came upon a
"Buffered Random Access File" implementation.
/*
* RandomAccessFile.java. By Russ Rew, based on
* BufferedRandomAccessFile by Alex McManus, based on Sun's source code
* for java.io.RandomAccessFile. For Alex McManus version from which
* this derives, see his <a href="http://www.aber.ac.uk/~agm/Java.html";>
* Freeware Java Classes</a>.
*/

This works fine but is based on java 1.3.1 java.io.* classes. I was
wondering if any one else has something that utilizes the java.nio.*
classes. The way is which the data is accessed is READ-ONLY and
generally random.. in other words the program may need to jump to byte
position 123456 and read any number of bytes in a row with some
skipping over a few bytes here and there along the way.

Any pointers or help would be much appreciated.

Mike Jackson
On Apr 5, 2004, at 5:47 AM, Tim Boudreau wrote:

> Marginally faster might be a read-only Document implementation
> directly over a memory-mapped byte buffer. I've been working on a
> similar approach for NetBeans' output window, on a branch. Functional
> prototype code (but use a JTextArea, not a JTextPane - it's a work in
> progress, and currently with StyledDocument it's not breaking
> lines...at all...no time right now to figure out why) of a work in
> progress:
>
> http://core.netbeans.org/source/browse/core/output2/src/org/netbeans/
> core/output2/Attic/OutWriter.java?rev=1.1.2.2&content-type=text/x-
> cvsweb-markup&showattic=1
>
> http://core.netbeans.org/source/browse/core/output2/src/org/netbeans/
> core/output2/Attic/OutputDocument.java?rev=1.1.2.2&content-type=text/
> x-cvsweb-markup&showattic=1
>
> You'd need to modify it a bit to do charset conversion - since I'm
> writing the file I'm displaying, I'm leaving it as UTF-16 for
> throughput reasons, at least until I can prove there's a benefit to
> converting while writing and reading.
>
> I tested the above prototype scrolling through a 240Mb file, and it
> was glitch free and dead fast. However, I haven't profiled it for
> small files - there the overhead of memory mapping the file could
> conceivably swamp any benefit - you'd have to profile it and see if
> there was any benefit. Probably the prototype could be simplified
> significantly further if you know the file is not going to be appended
> to.
>
> -Tim
>
>
> On Apr 4, 2004, at 8:24 AM, Craig Mattocks wrote:
>
>> I am trying to find a lightning fast way to read an ASCII text file
>> (30-100 K bytes) for display in a scrolling file previewer. So far,
>> I cannot do much better than the old readFile method (at the bottom
>> of the listing below). Can anyone do better?
>>
>> TIA,
>> Craig
>>
>> /**
>> * Program: FastStreamCopy.java
>> * Author : Craig Mattocks
>> * Date : April, 2004
>> * Description: Use Java 1.4 NIO channels and buffers
>> * to read a file fast.
>> */
>>
>> import java.io.*;
>> import java.nio.*;
>> import java.nio.channels.*;
>> import java.nio.charset.*;
>> import java.text.NumberFormat;
>>
>> public class FastStreamCopy
>> {
>> public static void main(String args[])
>> {
>> if (args.length != 1)
>> {
>> System.out.println("Usage: java FastStreamCopy filename");
>> System.exit(1);
>> }
>>
>> NumberFormat digits = NumberFormat.getInstance();
>> digits.setMaximumFractionDigits(3);
>>
>> long before;
>> long after;
>> double slowTime;
>> double fastTime;
>> double speedUp;
>>
>> String filename = args[0];
>> String contents;
>>
>> // Slow method
>> System.out.println("Reading file " + args[0] + " using slow
>> method");
>> before = System.currentTimeMillis(); // Start timing
>> // contents = slowStreamCopy(filename);
>> contents = readFile(filename);
>> after = System.currentTimeMillis(); // End timing
>> slowTime = after - before;
>> // System.out.println("File's contents:\n" + contents);
>>
>> // Fast method
>> System.out.println("Reading file " + args[0] + " using fast
>> method");
>> before = System.currentTimeMillis(); // Start timing
>> contents = fastStreamCopy(filename);
>> after = System.currentTimeMillis(); // End timing
>> fastTime = after - before;
>> // System.out.println("File's contents:\n" + contents);
>>
>> // Comparison
>> speedUp = 100d * slowTime/fastTime;
>> System.out.println("Slow method required " + slowTime + " ms.");
>> System.out.println("Fast method required " + fastTime + " ms.");
>> System.out.print("Speed up = " + digits.format(speedUp) + "% ");
>> System.out.println(speedUp > 100? "Good!":"Bad!");
>> }
>>
>> /**
>> Read the contents of a text file using a memory-mapped byte buffer.
>>
>> A MappedByteBuffer, is simply a special ByteBuffer. MappedByteBuffer
>> maps a region of a file directly in memory. Typically, that region
>> comprises the entire file, although it could map a portion. You must,
>> therefore, specify what part of the file to map. Moreover, as with
>> the other Buffer objects, no constructor exists; you must ask the
>> java.nio.channels.FileChannel for its map() method to get a
>> MappedByteBuffer.
>>
>> Direct buffers allocate their data directly in the
>> runtime environment memory, bypassing the JVM|OS boundary,
>> usually doubling file copy speed. However, they generally
>> cost more to allocate.
>> */
>> private static String fastStreamCopy(String filename)
>> {
>> String s = "";
>> FileChannel fc = null;
>> try
>> {
>> fc = new FileInputStream(filename).getChannel();
>>
>> // int length = (int)fc.size();
>>
>> MappedByteBuffer byteBuffer =
>> fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
>> // CharBuffer charBuffer =
>> Charset.forName("ISO-8859-1").newDecoder().decode(byteBuffer);
>>
>> // ByteBuffer byteBuffer = ByteBuffer.allocate(length);
>> // ByteBuffer byteBuffer = ByteBuffer.allocateDirect(length);
>> // CharBuffer charBuffer = byteBuffer.asCharBuffer();
>>
>> // CharBuffer charBuffer =
>> ByteBuffer.allocateDirect(length).asCharBuffer();
>> /*
>> int size = charBuffer.length();
>> if (size > 0)
>> {
>> StringBuffer sb = new StringBuffer(size);
>> for (int count=0; count<size; count++)
>> sb.append(charBuffer.get());
>> s = sb.toString();
>> }
>>
>> if (length > 0)
>> {
>> StringBuffer sb = new StringBuffer(length);
>> for (int count=0; count<length; count++)
>> {
>> sb.append(byteBuffer.get());
>> }
>> s = sb.toString();
>> }
>> */
>> int size = byteBuffer.capacity();
>> if (size > 0)
>> {
>> // Retrieve all bytes in the buffer
>> byteBuffer.clear();
>> byte[] bytes = new byte[size];
>> byteBuffer.get(bytes, 0, bytes.length);
>> s = new String(bytes);
>> }
>>
>> fc.close();
>> }
>> catch (FileNotFoundException fnfx)
>> {
>> System.err.println("File not found: " + fnfx);
>> }
>> catch (IOException iox)
>> {
>> System.err.println("I/O problems: " + iox);
>> }
>> finally
>> {
>> if (fc != null)
>> {
>> try
>> {
>> fc.close();
>> }
>> catch (IOException ignore)
>> {
>> // ignore
>> }
>> }
>> }
>> return s;
>> }
>>
>> private static String slowStreamCopy(String filename)
>> {
>> String s = "";
>>
>> FileReader in = null;
>> try
>> {
>> File file = new File(filename);
>>
>> int size = (int)file.length();
>>
>> char[] buffer = new char[size];
>> in = new FileReader(file);
>>
>> // int count = in.read(buffer, 0, size);
>> // if (count != -1)
>>
>> int count;
>> while ((count = in.read(buffer, 0, size)) >= 0)
>> {
>> s = new String(buffer, 0, count);
>> }
>> in.close();
>> }
>> /*
>> String line;
>> BufferedReader in = null;
>> try
>> {
>> File file = new File(filename);
>> int size = (int)file.length();
>> if (size > 0)
>> {
>> StringBuffer sb = new StringBuffer(size);
>> in = new BufferedReader(new InputStreamReader(new
>> FileInputStream(filename),"ISO-8859-1"));
>> while ((line = in.readLine()) != null)
>> {
>> sb.append(line);
>> }
>> in.close();
>> s = sb.toString();
>> }
>> }
>> */
>> catch (FileNotFoundException fnfx)
>> {
>> System.err.println("File not found: " + fnfx);
>> }
>> catch (IOException iox)
>> {
>> System.err.println("I/O problems: " + iox);
>> }
>> finally
>> {
>> if (in != null)
>> {
>> try
>> {
>> in.close();
>> }
>> catch (IOException ignore)
>> {
>> // ignore
>> }
>> }
>> }
>> return s;
>> }
>>
>> public static String readFile(String filename)
>> {
>> String s = "";
>> FileInputStream in = null;
>> // FileReader in = null;
>> try
>> {
>> File file = new File(filename);
>> byte[] buffer = new byte[(int) file.length()];
>> // char[] buffer = new char[(int) file.length()];
>> in = new FileInputStream(file);
>> // in = new FileReader(file);
>> in.read(buffer);
>> s = new String(buffer);
>> in.close();
>> }
>> catch (FileNotFoundException fnfx)
>> {
>> System.err.println("File not found: " + fnfx);
>> }
>> catch (IOException iox)
>> {
>> System.err.println("I/O problems: " + iox);
>> }
>> finally
>> {
>> if (in != null)
>> {
>> try
>> {
>> in.close();
>> }
>> catch (IOException ignore)
>> {
>> // ignore
>> }
>> }
>> }
>> return s;
>> }
>>
>> }
>> ------------------------------------------
>> "We scorch the earth, set fire to the sky.
>> We steep so low, to reach so high."
>> - "Red Hill Mining Town", Joshua Tree, U2
>> ------------------------------------------
>> _______________________________________________
>> java-dev mailing list | email@hidden
>> Help/Unsubscribe/Archives:
>> http://www.lists.apple.com/mailman/listinfo/java-dev
>> Do not post admin requests to the list. They will be ignored.
> _______________________________________________
> java-dev mailing list | email@hidden
> Help/Unsubscribe/Archives:
> http://www.lists.apple.com/mailman/listinfo/java-dev
> Do not post admin requests to the list. They will be ignored.
_______________________________________________
java-dev mailing list | email@hidden
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/java-dev
Do not post admin requests to the list. They will be ignored.

References: 
 >Lightning fast file reader? (From: Craig Mattocks <email@hidden>)
 >Re: Lightning fast file reader? (From: Tim Boudreau <email@hidden>)



Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Contact Apple | Terms of Use | Privacy Policy

Copyright © 2007 Apple Inc. All rights reserved.