Mailing Lists: Apple Mailing Lists
Image of Mac OS face in stamp
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Performance problem creating large files

The machine for the following round of testing was a
PowerMac G4/450MHz/8GB disk/512MB RAM/MacOS X 10.1.5/MacOS 9.2.2.

When ported to MacOS Classic CFM, the test program takes just around
1 second when running under MacOS 9.2.2 natively.
If I then run another test program against the generated file, it looks
like sectors have been allocated for the entire file. Yes, I'm parsing
the data structures that are stored on the disk.

When running the same Classic CFM binary under the MacOS X 10.1.5 Classic
Environment with MacOS 9.2.2, the program takes 72 seconds. The processor
load bounces between 20 and 40%. (Based on CPU Monitor.) The machine is
more than responsive the entire time. The call to close()/FSClose is
where 99% of the clock time is spent.

Now, if I build a MacOS Carbon Mach-O target, I wind up with it taking
the same amount of time. Around 71 seconds.

Now, if I build a MacOS Carbon CFM target, I again wind up with it taking
the same amount of time. Around 71 seconds.

On the BSD side for this machine, it took 71.13 seconds to execute a
command line version of the program. Therefore, the time to create the file
is nearly equivalent when using either the Classic Environment File Manager
APIs or BSD file operations when running under MacOS X 10.1.5.
I have seen the same problem in 10.0.0 and 10.2.1.

Through the examples above, I've demonstrated that running this test
program against a HFS+ volume under MacOS 9.2.2 natively results in
very quick execution. About 1 second. And, under MacOS X, any attempt
to create a large file using BSD or Classic or Carbon results in
a significant (and I might add unacceptable) delay.

--Steve


On Thu, Dec 12, 2002 at 06:03:06PM -0800, Umesh Vaishamapayan wrote:
> HFS+ does not have sparse files. That's why your test program takes
> forever. [close() has to zerofill the file to ensure you do not have
> access to stale data on the disk].
>
> The same program run on UFS will execute a *lot* quicker because it
> supports "holes"...
> However, if you seek top 1GB - 1 and write a byte, UFS and Linux will
> *not* preallocate the disk space.
>
> I am guessing that #3126545 would be closed as "behaves correctly"...
>
>
> --Umesh
>
> On Thursday, December 12, 2002, at 04:39 PM, Steven Bytnar wrote:
>
> >On other OSes, like Linux 2.4.18 x86 and MacOS 8.6 & 9.1, the test code
> >below takes almost no time. All the test code does is create a file,
> >seeks to 1GB -1, writes one byte and then closes the file.
> >
> >bash-2.05b$ time ./l
> >Creating 1GB test file...
> >Writting at 1GB...
> >Closing file...
> >Task took 0.726 sec to run.
> >Done. Large file created.
> >
> >real 0m0.004s
> >user 0m0.010s
> >sys 0m0.000s
> >
> >
> >On Darwin 6.1 (OS X 10.2.1) it takes forever!
> >Creating 1GB test file...
> >Writting at 1GB...
> >Closing file...
> >Task took 38.26823 sec to run.
> >Done. Large file created.
> >0.000u 5.940s 0:38.03 15.6% 0+0k 0+14io 0pf+0w
> >
> >What's the deal? Are there other BSD or Cocoa or Carbon APIs that
> >I can call to avoid this performance problem?
> >All of the time is spent on the close(), so I suspect that the
> >close system call is where the process is getting blocked.
> >
> >I need this functionality so I can pre-allocate large scratch files
> >that are used later in the software's lifetime. It is not feasible to
> >create these files dynamically at this point in time.
> >
> >--Steve (filed as bug #3126545)
> >
> >
> >/*
> > * Assuming this is named largetest.c, build with:
> > * cc -o largetest largetest.c
> > */
> >#include <stdio.h>
> >#include <sys/types.h>
> >#include <sys/stat.h>
> >#include <fcntl.h>
> >#include <unistd.h>
> >#include <errno.h>
> >#include <string.h>
> >#include <sys/time.h>
> >
> >#define ERROR(x,y) { if ( (x) == -1) { perror(y); exit(errno); } }
> >
> >int main(int argn, char **argv)
> >{
> > const char *file;
> > int fd, len;
> > long long count;
> > ssize_t result;
> > long long offset;
> > struct timeval start, end, dif;
> >
> > if (argn < 2)
> > file = "largefile.junk";
> > else
> > file = argv[1];
> >
> > gettimeofday(&start, NULL);
> >
> > fd = open(file, O_CREAT | O_TRUNC | O_RDWR , 00644);
> > ERROR(fd, "Failed to open file");
> >
> > printf("Creating 1GB test file...\n");
> > count = (1*1024*1024*1024) - 1;
> > offset = lseek(fd, count, SEEK_CUR);
> > ERROR(offset, "Failed to set current offset");
> >
> > printf("Writting at 1GB...\n");
> > result = write(fd, "b", 1); /* Write data at 1GB point */
> > ERROR(result, "Failed to write at 1 GB mark");
> >
> > printf("Closing file...\n");
> > result = close(fd);
> > ERROR(result, "Failed to close file");
> >
> > gettimeofday(&end, NULL);
> >
> > timersub(&end, &start, &dif);
> > printf("Task took %ld.%ld sec to run.\n", dif.tv_sec, dif.tv_usec);
> >
> > printf("Done. Large file created.\n");
> >}
> >_______________________________________________
> >darwin-development mailing list | email@hidden
> >Help/Unsubscribe/Archives:
> >http://www.lists.apple.com/mailman/listinfo/darwin-development
> >Do not post admin requests to the list. They will be ignored.
_______________________________________________
darwin-development mailing list | email@hidden
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-development
Do not post admin requests to the list. They will be ignored.
References: 
 >Performance problem creating large files (From: Steven Bytnar <email@hidden>)
 >Re: Performance problem creating large files (From: Umesh Vaishamapayan <email@hidden>)



Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Contact Apple | Terms of Use | Privacy Policy

Copyright © 2007 Apple Inc. All rights reserved.