Mailing Lists: Apple Mailing Lists
Image of Mac OS face in stamp
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Performance problem creating large files

This analysis seems perfectly consistent with Umesh's explanantion:

> [close() has to zerofill the file to ensure you do not have access to stale data on the disk]

Mac OS 9.2.2, being a single-user operating system didn't zerofill files, and
after extending the logical eof, stale disk data could be read. Mac OS X,
being a multi-user operating system should, and does, zerofill for
security considerations.

-Kevin


Steven Bytnar <email@hidden> wrote:

> The machine for the following round of testing was a
> PowerMac G4/450MHz/8GB disk/512MB RAM/MacOS X 10.1.5/MacOS 9.2.2.
>
> When ported to MacOS Classic CFM, the test program takes just around
> 1 second when running under MacOS 9.2.2 natively.
> If I then run another test program against the generated file, it looks
> like sectors have been allocated for the entire file. Yes, I'm parsing
> the data structures that are stored on the disk.
>
> When running the same Classic CFM binary under the MacOS X 10.1.5 Classic
> Environment with MacOS 9.2.2, the program takes 72 seconds. The processor
> load bounces between 20 and 40%. (Based on CPU Monitor.) The machine is
> more than responsive the entire time. The call to close()/FSClose is
> where 99% of the clock time is spent.
>
> Now, if I build a MacOS Carbon Mach-O target, I wind up with it taking
> the same amount of time. Around 71 seconds.
>
> Now, if I build a MacOS Carbon CFM target, I again wind up with it taking
> the same amount of time. Around 71 seconds.
>
> On the BSD side for this machine, it took 71.13 seconds to execute a
> command line version of the program. Therefore, the time to create the file
> is nearly equivalent when using either the Classic Environment File Manager
> APIs or BSD file operations when running under MacOS X 10.1.5.
> I have seen the same problem in 10.0.0 and 10.2.1.
>
> Through the examples above, I've demonstrated that running this test
> program against a HFS+ volume under MacOS 9.2.2 natively results in
> very quick execution. About 1 second. And, under MacOS X, any attempt
> to create a large file using BSD or Classic or Carbon results in
> a significant (and I might add unacceptable) delay.
>
> --Steve
>
>
> On Thu, Dec 12, 2002 at 06:03:06PM -0800, Umesh Vaishamapayan wrote:
> > HFS+ does not have sparse files. That's why your test program takes
> > forever. [close() has to zerofill the file to ensure you do not have
> > access to stale data on the disk].
> >
> > The same program run on UFS will execute a *lot* quicker because it
> > supports "holes"...
> > However, if you seek top 1GB - 1 and write a byte, UFS and Linux will
> > *not* preallocate the disk space.
> >
> > I am guessing that #3126545 would be closed as "behaves correctly"...
> >
> >
> > --Umesh
> >
> > On Thursday, December 12, 2002, at 04:39 PM, Steven Bytnar wrote:
> >
> > >On other OSes, like Linux 2.4.18 x86 and MacOS 8.6 & 9.1, the test code
> > >below takes almost no time. All the test code does is create a file,
> > >seeks to 1GB -1, writes one byte and then closes the file.
> > >
> > >bash-2.05b$ time ./l
> > >Creating 1GB test file...
> > >Writting at 1GB...
> > >Closing file...
> > >Task took 0.726 sec to run.
> > >Done. Large file created.
> > >
> > >real 0m0.004s
> > >user 0m0.010s
> > >sys 0m0.000s
> > >
> > >
> > >On Darwin 6.1 (OS X 10.2.1) it takes forever!
> > >Creating 1GB test file...
> > >Writting at 1GB...
> > >Closing file...
> > >Task took 38.26823 sec to run.
> > >Done. Large file created.
> > >0.000u 5.940s 0:38.03 15.6% 0+0k 0+14io 0pf+0w
> > >
> > >What's the deal? Are there other BSD or Cocoa or Carbon APIs that
> > >I can call to avoid this performance problem?
> > >All of the time is spent on the close(), so I suspect that the
> > >close system call is where the process is getting blocked.
> > >
> > >I need this functionality so I can pre-allocate large scratch files
> > >that are used later in the software's lifetime. It is not feasible to
> > >create these files dynamically at this point in time.
> > >
> > >--Steve (filed as bug #3126545)
> > >
> > >
> > >/*
> > > * Assuming this is named largetest.c, build with:
> > > * cc -o largetest largetest.c
> > > */
> > >#include <stdio.h>
> > >#include <sys/types.h>
> > >#include <sys/stat.h>
> > >#include <fcntl.h>
> > >#include <unistd.h>
> > >#include <errno.h>
> > >#include <string.h>
> > >#include <sys/time.h>
> > >
> > >#define ERROR(x,y) { if ( (x) == -1) { perror(y); exit(errno); } }
> > >
> > >int main(int argn, char **argv)
> > >{
> > > const char *file;
> > > int fd, len;
> > > long long count;
> > > ssize_t result;
> > > long long offset;
> > > struct timeval start, end, dif;
> > >
> > > if (argn < 2)
> > > file = "largefile.junk";
> > > else
> > > file = argv[1];
> > >
> > > gettimeofday(&start, NULL);
> > >
> > > fd = open(file, O_CREAT | O_TRUNC | O_RDWR , 00644);
> > > ERROR(fd, "Failed to open file");
> > >
> > > printf("Creating 1GB test file...\n");
> > > count = (1*1024*1024*1024) - 1;
> > > offset = lseek(fd, count, SEEK_CUR);
> > > ERROR(offset, "Failed to set current offset");
> > >
> > > printf("Writting at 1GB...\n");
> > > result = write(fd, "b", 1); /* Write data at 1GB point */
> > > ERROR(result, "Failed to write at 1 GB mark");
> > >
> > > printf("Closing file...\n");
> > > result = close(fd);
> > > ERROR(result, "Failed to close file");
> > >
> > > gettimeofday(&end, NULL);
> > >
> > > timersub(&end, &start, &dif);
> > > printf("Task took %ld.%ld sec to run.\n", dif.tv_sec, dif.tv_usec);
> > >
> > > printf("Done. Large file created.\n");
> > >}
> > >_______________________________________________
> > >darwin-development mailing list | email@hidden
> > >Help/Unsubscribe/Archives:
> > >http://www.lists.apple.com/mailman/listinfo/darwin-development
> > >Do not post admin requests to the list. They will be ignored.
> _______________________________________________
> darwin-development mailing list | email@hidden
> Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-development
> Do not post admin requests to the list. They will be ignored.
_______________________________________________
darwin-development mailing list | email@hidden
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/darwin-development
Do not post admin requests to the list. They will be ignored.
References: 
 >Re: Performance problem creating large files (From: Steven Bytnar <email@hidden>)



Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Contact Apple | Terms of Use | Privacy Policy

Copyright © 2007 Apple Inc. All rights reserved.