August 2005 - Darwin-dev - lists.apple.com

libmx odcctools
by plumber 18 Aug '05

18 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com i can link libsystem as libmx but i prefer to have lmx it'll be very kind Best Regards -----BEGIN PLUMBER CODE BLOCK----- Version: 1.0 ¬ | | > Regards Plumber || GNU Darwin. ¬ \\\ plumber.gnu-darwin.org ¬ ¬ | | > OOP --\///\ (0)=(0) Darwin/Power Mac G4. ¬ | | > ( __ò ó__ ) The best for Fede ¬ \\\ federicafontana.it ¬ ------END PLUMBER CODE BLOCK------ -----BEGIN GEEK CODE BLOCK----- Version: 3.1 GMU/S d+@ s: a- K+++++ PGP++++ K---- dpu s--:-- C++++ C-- B L X U++++ P+++ C-- P! L--- E+++ W+++ P! L--- N+++ o-- K+++++ ------END GEEK CODE BLOCK------ Plain, Loving, Useful, Mischievous, Brutal, Explosive, Responsible _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Hi there , someone has a pkg for darwin-7-x86 of the libmx(odcctools) ? i've got an minor ld error at build but i need libmx for tests ( short time i've to leave for my real job :) ) I"It's making my eyes better. And it's legal. I could walk up to the President and blow smoke in his stupid monkey face, and he'd just have to sit there groovin' on it!" Homer J Simpson, Springfield, Illinois, Massachusetts nobody knows :) This email sent to site_archiver(a)lists.apple.com

1 0

directio, O_DIRECT, "mount -o sync", O_SYNC, O_DSYNC and all that
by Reto Kramer 17 Aug '05

17 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Thanks, - Reto _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm looking for a way to (1) enable direct IO on OS X (10.4) and (2) a way to write w/o updating metadata (i.e. last access time). All in the context of the ever popular tx log file use case. - are there plans to support O_DIRECT, or mount -o forcedirectio (disable cache)? - are there plans to support the O_DSYNC open flag (avoid some metadata update, i.e. last access time)? To help me understand what is available (what I found at least) in 10.4, I appreciate your help with the following 3 questions: (a) is the sync mount option equivalent to the directio flag, I can e.g. get on Solaris?, or (b) if I open a file with O_SYNC (in fcntl.h but not "man 2 open" page) is this the same as writing to a file on a FS that I mounted with "mount -o sync" ? (c) is writing (write system call) to a "-o sync" mounted FS (HFS+) equivalent to fsync'ing the fd (I only care about a single tx log file), except that I save a system call? I.e. can I be assured that (module the fcntl F_FULLSYNC call) the data is flushed to disk if I write to a "-o sync" HFS, or "D_SYNC" opened file? This email sent to site_archiver(a)lists.apple.com

1 0

Re: memory management question
by Andrew Gallatin 17 Aug '05

17 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Brian Barrett writes: > > So the actual question to the list - what is the best way to do this > on Darwin? Currently, we do something really evil, similar to what See "Re: overriding malloc " by Mike Smith on April 15, 2005. http://lists.apple.com/archives/darwin-dev/2005/Apr/msg00050.html I have never gotten around to looking closely at this for Mpich-GM, because GM is essentially EOL, and the current strategy, while awkward, works. Drew _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

memory management question
by Brian Barrett 17 Aug '05

17 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Hi all - 1) register region of memory to send from / receive to 2) perform send/recv 3) unregister region of memory Thanks, Brian -- Brian Barrett Graduate Student, Open Systems Lab, Indiana University http://www.osl.iu.edu/~brbarret/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I have a question about Darwin's memory manager I hope someone can help with. I'm working on Open MPI, an implementation of the Message Passing Interface for high performance computing. Basically, a nice, portable interface for writing high performance parallel codes. We support a number of high speed networks including InfiniBand (IB) and Myrinet, both of which use OS bypass remote memory access for better performance. The problem I'm running into is with "registering" memory with the communication library. Essentially, both IB and the Myrinet software (also known as gm) have a usage similar to: The registration serves both the purpose of allowing the communication library (IB or GM) to figure out all the virtual to physical memory address translations and to pin/wire the pages so that they don't move during the registration period. Registration costs are around 10us and deregistration costs are around 200us for a single page and the cost scale is sub-linear but not constant as registration sizes increase. For a network with sub-5us end to end communication cost, this is too high for most uses. Due to the high registration/deregistration costs and given that most HPC applications have a high locality, it would be really nice to cache the registrations so that we only unregister memory when either the application is done or some threshold of # of registered pages is reached. The problem with this is that the user is free to do something like: 1) malloc out a chunk of memory 2) call MPI_Send out of said buffer (which means MPI registered, sends, but doesn't unpin because we're caching) 3) calls free on the chunk of memory which means the user buffer is being returned to the OS while the communication library thinks the memory is still pinned. From this point, bad things usually happen - exactly what depends on the OS. I believe Darwin will block the thread, waiting for the memory to be unpinned before free() returns. Which basically means free() isn't going to return. On Linux, we provide an override for the weak symbols for munmap and friends, and use either the glibc malloc hooks or an ldpreload to intercept free() and realloc() so that we can unregister any pinned memory before we let the OS actually free it. Ideally, rather than intercepting free() and realloc(), we'd like some callback / notification / etc. that memory was actually being given back to the OS. That way, we don't have to go through our registration LRU lists and all that So the actual question to the list - what is the best way to do this on Darwin? Currently, we do something really evil, similar to what MPICH-gm (the Myricom distributed MPI for Myrinet) does. We copied scalable_malloc.c out of the Darwin libSystem sources and added a hook to deregister our memory wherever vm_deallocate() is called. MPI applications are generally compiled with "wrapper compilers" that get all the right CFLAGS, LDFLAGS, LIBS, etc for that particular MPI implementation, so we can add the right linker flags to make sure that our version of scalable_malloc.o is used instead of the one in libSystem. This, of course, is pretty fragile - if Apple changes scalable_malloc.c, we have to adjust to that. We have to force a flat namespace for the user application, etc. So, is there a better way to get notification that memory is about to leave the process and give us an opportunity to deregister the memory (if needed)? This email sent to site_archiver(a)lists.apple.com

1 0

Re: Error in online instructions
by Markus Hitter 17 Aug '05

17 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Am 16.08.2005 um 20:27 schrieb Justin C. Walker: On Aug 16, 2005, at 10:34 , Scott Mansfield wrote: At the line hdiutil create -size 4g -type UDIF -fs HFSX -volname Builds \ -uid 0 -gid 0 -attach Builds.dmg [snip] heck, yet another $0.02 :-) Markus - - - - - - - - - - - - - - - - - - - Dipl. Ing. Markus Hitter http://www.jump-ing.de/ _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm new to darwin build and just ran through the installation steps. In doing so I encountered something a wee confusing. http://opendarwin.org/projects/darwinbuild/doc/tools/ index.html#install It's not a bug, nor is it a feature. The backslash is a common documentation trick when a line is too long and needs a visual break; It indicates "type all this in." It's actually more than a documentation trick. It's a common shell "escape", and means "treat the next line as part of this line when evaluating the command". One possible reason why it failed for the original poster is, any trailing character, even a trailing space makes this machanism fail. Trailing spaces are quite common if you copy&paste from HTML documents. This email sent to site_archiver(a)lists.apple.com

1 0

dlopen and d2i_DHparams
by Steve Checkoway 17 Aug '05

17 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com $ cat foobar.cpp #include <openssl/dh.h> #include <openssl/asn1.h> #include <openssl/bn.h> #include <cstdio> #include <dlfcn.h> DH *dh2 = DHparams_dup(dh); dlclose(handle); return 0; } $ make foo bar baz g++ -o foo -lcrypto foobar.cpp g++ -flat_namespace -undefined suppress -bind_at_load \ -o bar foobar.cpp MACOSX_DEPLOYMENT_TARGET=10.3 g++ -undefined dynamic_lookup \ -bind_at_load -o baz foobar.cpp $ ./foo $ ./bar dyld: Symbol not found: _d2i_DHparams Referenced from: /Users/steve/temp/dlopen/./bar Expected in: flat namespace Trace/BPT trap $ ./baz dyld: Symbol not found: _d2i_DHparams Referenced from: /Users/steve/temp/dlopen/./baz Expected in: dynamic lookup Trace/BPT trap $ cat hash.cpp #include <openssl/sha.h> #include <cstdio> #include <cstring> #include <dlfcn.h> puts(hello); SHA1((unsigned char *)hello, strlen(hello), md); for(int i = 0; i < SHA_DIGEST_LENGTH; ++i) printf("%02hhx", md[i]); puts(""); dlclose(handle); return 0; } $ make hash g++ -flat_namespace -undefined suppress -bind_at_load \ -o hash hash.cpp $ ./hash Hello World! 2ef7bde608ce5404e97d5f042f95f89f1c232871 - Steve _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com I'm getting strange behavior when I try to dlopen libcrypto.dylib and then use d2i_DHparams. #ifdef DHparams_dup # undef DHparams_dup #endif #define DHparams_dup (x) \ (DH *)ASN1_dup((int (*)(...)) i2d_DHparams, \ (char *(*)(...))d2i_DHparams, (char *)(x)) int main() { void *handle = dlopen("libcrypto.dylib", RTLD_LAZY| RTLD_GLOBAL); DH *dh = DH_new(); BN_hex2bn(&dh->p, "FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD129024" "E088A67CC74020BBEA63B139B22514A08798E3404DDEF9519B3CD" "3A431B302B0A6DF25F14374FE1356D6D51C245E485B576625E7EC" "6F44C42E9A637ED6B0BFF5CB6F406B7EDEE386BFB5A899FA5AE9F" "24117C4B1FE649286651ECE65381FFFFFFFFFFFFFFFF"); dh->g = BN_new(); BN_set_word(dh->g, DH_GENERATOR_2); When I run ./foo it runs correctly. When I run ./bar and ./baz dyld fails with the symbol not found. However, it works with other functions such as SHA1. int main() { void *handle = dlopen("libcrypto.dylib", RTLD_LAZY| RTLD_GLOBAL); unsigned char md[SHA_DIGEST_LENGTH]; char hello[] = "Hello World!"; What is the difference between SHA1 and d2i_DHparams that would cause the first program to fail but the second one to succeed? smime.p7s

1 0

Re: Error in online instructions
by Shantonu Sen 16 Aug '05

16 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com <http://opendarwin.org/mailman/listinfo/darwinbuild> Shantonu On Aug 16, 2005, at 10:30 AM, Will Hickie wrote: _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The same place where you read the instructions has a link to the darwinbuild mailing list, which is dedicated to discussion of darwinbuild and building Darwin. This may not be the right place to report this, so my apologies in advance if it isn't. :) This email sent to site_archiver(a)lists.apple.com

1 0

Re: Error in online instructions
by Justin C. Walker 16 Aug '05

16 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Aug 16, 2005, at 10:34 , Scott Mansfield wrote: At the line hdiutil create -size 4g -type UDIF -fs HFSX -volname Builds \ -uid 0 -gid 0 -attach Builds.dmg For example: $ ls -l \ > /tmp/ total 3560 -rw-r--r-- 1 justin wheel 1511 Aug 14 17:15 files -rwxr-xr-x 1 justin wheel 280 Aug 14 17:23 foo -rw-r--r-- 1 justin wheel 474 Aug 13 10:29 foo.cc -rwxr-xr-x 1 justin wheel 201452 Aug 16 09:46 gmp1 -rw-r--r-- 1 justin wheel 764 Aug 16 09:44 gmp1.c -rw-r--r-- 1 justin wheel 662 Aug 16 09:44 gmp2.c drwxrwxrwx 3 root wheel 102 Aug 14 17:20 printers $ Regards, Justin -- Justin C. Walker, Curmudgeon at Large Institute for General Semantics ----------- My wife 'n kids 'n dog are gone, I can't get Jesus on the phone, But Ol' Milwaukee's Best is my best friend. ----------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm new to darwin build and just ran through the installation steps. In doing so I encountered something a wee confusing. http://opendarwin.org/projects/darwinbuild/doc/tools/ index.html#install [snip] It's not a bug, nor is it a feature. The backslash is a common documentation trick when a line is too long and needs a visual break; It indicates "type all this in." It's actually more than a documentation trick. It's a common shell "escape", and means "treat the next line as part of this line when evaluating the command". Note the "> " on the line after the "\"; this is the "secondary" shell prompt, indicating that the shell is waiting for the rest of the line. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Error in online instructions
by Scott Mansfield 16 Aug '05

16 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com At http://opendarwin.org/projects/darwinbuild/doc/tools/index.html#install the line hdiutil create -size 4g -type UDIF -fs HFSX -volname Builds \ -uid 0 -gid 0 -attach Builds.dmg includes a \ after '-volname Builds'. Removing the \ causes all to work well. Cheers, Scott _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… I'm new to darwin build and just ran through the installation steps. In doing so I encountered something a wee confusing. If this line is typed in to the shell, hdiutil throws an error stating only one volume name can be specified. This may not be the right place to report this, so my apologies in advance if it isn't. :) Will It's not a bug, nor is it a feature. The backslash is a common documentation trick when a line is too long and needs a visual break; It indicates "type all this in." This email sent to site_archiver(a)lists.apple.com

1 0

Error in online instructions
by Will Hickie 16 Aug '05

16 Aug '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0