Re: fread multiple files

Subject: Re: fread multiple files

From: Michael Smith <email@hidden>

Date: Fri, 25 Sep 2009 17:49:54 -0700

Delivered-to: email@hidden

On Sep 25, 2009, at 12:03 PM, email@hidden wrote:

An MPI code I use outputs 3 * N files, where N is the number of
processors. Then I am using a serial code to post-process the results.
The post-processor is single-threaded and for every time instant reads
a chunk of data from each of 3 *N files. I noticed that most of the
time (nearly 50%) is spent in:

__spin_lock
pthread_mutex_lock
fread

This - the use of fread - is almost certainly your problem.

Your post-processor should not be using the stdio interfaces. It's probable that it's also reading small amounts of data at a time, which would account for the large amount of time spent inside the locked region.

If you fix your post-processor to use the read(2) interface, and buffer data locally rather than pulling it through stdio, you should see a dramatic improvement in performance.

= Mike

Ars longa, vita brevis, occasio praeceps, experimentum periculosum, iudicium difficile -- Hippocrates

_______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (email@hidden) Help/Unsubscribe/Update your Subscription: This email sent to email@hidden