December 2005 - Darwin-dev - lists.apple.com

Re: Trylocks in the kernel
by Peter Lovell 15 Dec '05

15 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Dec 14, 2005, at 7:00 PM, Sam Vaughan wrote: On 15/12/2005, at 7:03 AM, darwin-dev-request(a)lists.apple.com wrote: On Dec 13, 2005, at 9:21 AM, Brian Bergstrand wrote: On Dec 13, 2005, at 12:23 AM, Ryan McGann wrote: The new Tiger locking KPIs seem to lack a trylock variant for any of the lock types (mutex, spinlock, read/write). A mutex try variant is available in the Unsupported KPI. It's probably safe to use this for 8.0, but may break in the next major release. All of the try variants are in the kernel proper, so one has to wonder why they are not exported in the KPI. Well they also have a way to promote read-write locks to exclusive from shared exported in the kernel, but not in the KPIs (lck_rw_exclusive_to_shared and lck_rw_shared_to_exclusive). That bug was closed :-( It would be really nice to have those KPIs back in the supported list, if anybody is listening... Hear Hear. _lck_mtx_try_lock _lck_rw_done _lck_rw_lock_exclusive_to_shared_gen _lck_rw_try_lock_exclusive _lck_rw_try_lock_shared _lck_spin_try_lock Regards.....Peter _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Amongst the 30 kernel symbols we need that are preventing us from using the 10.4 KPI I see the following locking functions: My bugs have been marked "Duplicate", so someone's aware of the issue. All the same I think we need to push this at WWDC next year. A locking API that doesn't export try functions and requires you to keep track of whether an rwlock is held shared or exclusive when you unlock it simply isn't complete in my opinion. Sam I add my vote to this request. I don't care what Yoda says, there *does* need to be "try". This email sent to site_archiver(a)lists.apple.com

1 0

Re: General Guidelines on bulk data through Posix Sockets
by Andrew White 15 Dec '05

15 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mozilla Thunderbird 0.9 (Macintosh/20041103) Mark Gilbert wrote: For interest, compare the 'mtu' values on the various interfaces: % ifconfig lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280 stf0: flags=0<> mtu 1280 en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078 ppp0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1444 Note that recvfrom can also return under-sized data. >From 'man recvfrom': --- If no messages are available at the socket, the receive call waits for a message to arrive, unless the socket is nonblocking (see fcntl(2)) in which case the value -1 is returned and the external variable errno set to EAGAIN. The receive calls normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested; this behavior is affected by the socket-level options SO_RCVLOWAT and SO_RCVTIMEO described in getsockopt(2). --- -- Andrew White -------------------------------------------------------------------------- This email and any attachments may be confidential. They may contain legally privileged information or copyright material. You should not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages. We do not accept liability in connection with computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment. This notice should not be removed. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… The remote machine requests and frame and my app delivers it. The frame rate is slow (a few frames per sec) Are you using TCP or UDP? I dont know, sorry... I use the regular POSIX sockets stuff, bind, connect, read etc If you're using listen & connect, then you're using TCP. UDP doesn't use "listen", and connect doesn't do much. More definitively, look at your "socket" call. TCP uses SOCK_STREAM and IPPROTO_TCP, while UDP uses SOCK_DGRAM and IPPROTO_UDP. Sending a 30k "datagram" is interesting, since you will inevitably suffer fragmentation. A standard ethernet supports discrete packets of only ~1500 bytes or less. As such, large datagrams must be split up into smaller packets for transmission, and then reassembled at the destination. That is exactly what I see. Interestingly when I was running client and server on the same CPU there was no fragmentation of the message, but when I put them on different machines things change considerably as you describe. When on the same server, the networking stack simply copies the entire message from the outgoing buffer of one socket to the incoming buffer of the other. This is true even if you use a physical interface rather than loopback; the datagram never hits the hardware so it is never fragmented. got it. can you elaborate on the functions I need to do this with TCP ? aha. "recvfrom" . I shall look this up in the big book. Using TCP, I'd probably recommend abstracting out the packet nature of your communications and treat the link as a bi-directional data stream. Messages are formatted as 'length:data'. The receiver reads a 4 byte length byte and then slurps in the actual data. I don't think you'll have flushing problems as long as you use write rather than stdio. If you finish reading (or more rarely, writing) early and you don't mind blocking, you just call read again with the remainder until you have it all. If you do mind blocking or want to handle multiple clients, then you need a full TCP server model with a separate TCP connection for each client, and track where each client (and client socket) is up to. In this case, 'select' is your friend. If you use UDP, then recvfrom and sendto are more applicable, but you also need to worry about recovery from lost packets (rare) and manual packetisation (common). This email sent to site_archiver(a)lists.apple.com

1 0

Re: General Guidelines on bulk data through Posix Sockets
by Andrew White 15 Dec '05

15 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Mozilla Thunderbird 0.9 (Macintosh/20041103) > continue reading until 'read' returned-1 (which would be the end of the > reply), I now see a return of -1 part way through the 30k reply, and it > picks up again later. -- Andrew White -------------------------------------------------------------------------- This email and any attachments may be confidential. They may contain legally privileged information or copyright material. You should not read, copy, use or disclose them without authorisation. If you are not an intended recipient, please contact us at once by return email and then delete both messages. We do not accept liability in connection with computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment. This notice should not be removed. _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Mark Gilbert wrote: Before I rework this code to deal with this, I wanted to get any general guidelines on the best way to handle exchange of larger packets of data via sockets. Any particular tricks ? pitfalls ? Firstly, are you using a stream (semi-continuous flow of information) or datagram (explicit discrete messages) model? Are you using TCP or UDP? The stream model generally uses TCP, which includes a bunch of overhead to ensure reliable, in-order delivery and to avoid link saturation. The datagram model can use either TCP or UDP, but UDP has historically had this model in mind. UDP simply attempts to deliver single "packets" of data, and doesn't have the same reliability model that TCP does. Sending a 30k "datagram" is interesting, since you will inevitably suffer fragmentation. A standard ethernet supports discrete packets of only ~1500 bytes or less. As such, large datagrams must be split up into smaller packets for transmission, and then reassembled at the destination. TCP avoids this by simply buffering everything in a large sequential buffer and peeling off appropriate sized chunks for packetisation at an appropriate rate to avoid losing packets. UDP doesn't do this, so it simply partitions the datagram, sends a rush of packets, and hopes they all get through. If some don't then the packet cannot be reassembled and the whole packet is discarded. In particular, whereas previously I could read 1 byte blocking, then This is perfectly normal. 'read' only returns what has currently been received, and delivering 30kb of data is not atomic. It takes a little bit of time for the system to tell your app to respond to delivered data, so you may get more than one packet in the receive buffer before your app gets a chance to read it. You also need to be careful with 'read' and datagrams, since it deals in data, not messages. Even with small packets, it's quite possible to have more than one in the buffer, and if you use a big enough destination buffer then read could slurp them all up in one hit. If you really need per-packet / datagram handling, you should use 'recvfrom' instead. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Libtool (ld) problem in makefile. Is the GCC 4.0 distribution broken?
by Martin Costabel 15 Dec '05

15 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com User-agent: Thunderbird 1.5 (Macintosh/20051025) Paul Archibald wrote: [] $(TARGET): $(OBJECTS) # this is the original libtool line libtool -dynamic -M -single_module -o $@ $^ -lstdc++ -lc -lgcc > map strip -x $@ Don't use libtool directly for linking, use g++. The result of making this is... >>> begin output >>> ld: for architecture ppc ld: Undefined symbols: __ZN9__gnu_cxx18__exchange_and_addEPVii __ZNSs4_Rep20_S_empty_rep_storageE __ZNSs4_Rep9_S_createEmmRKSaIcE __ZSt18_Rb_tree_decrementPSt18_Rb_tree_node_base __ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_ __ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_ __ZNSt15_List_node_base4hookEPS_ __ZNSt15_List_node_base6unhookEv __ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base libtool: internal link edit command failed make: *** [../libJEngine.jnilib] Error 1 These symbols are all defined in the g++-4.0 dynamic libstdc++. /usr/lib/libstdc++.a -> gcc/darwin/default/libstdc++.a /usr/lib/libgcc.a -> gcc/darwin/default/libgcc.a and, in /usr/lib/gcc/darwin default -> 3.3 -- Martin _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… [] So, what is going on here? What are these symbols? Are they STL/template symbols? (Do I care a lot? Not really, I just want this to work.) [] Where are these undefined symbols defined? Am I missing a library or a header, and if so, which one and how would I find out? (Even if the problem IS the GCC distribution, I am still curious how I would determine where to find the definitions of undefined symbols. Metrowerks supplies a "Find Library" facility which can be used to find symbols. Does GCC have something like that?) A simple recursive "grep" in /usr/lib shows where they are mentioned, and with "nm" you can see where they are defined. (One thing that helped a lot was to use the Disk Utility.app to repair the file permissions on my drive. Before I did that (thanks to whoever suggested that, I forget where I saw it), libtool choked on the -llstdc++ -lc -lgcc section of the makefile, asserting that lstdc++ and gcc were not found and could not be included. After fixing file permissions, that error went away. I think everyone should run fix permissions.) Nothing could be further from the truth. Whoever gave you this advice didn't know that repair permissions is actually broken on Tiger. What it did for you was to make some symbolic links in /usr/lib that should only be there if you chose gcc-3.3 as compiler and not if you selected gcc-4.0. Two of these links are All of these links disappear if you run "gcc_select 4.0" and they reappear (correctly) if you run "gcc_select 3.3", but also (incorrectly) if you run repair permissions. These symlinks do, indeed, make the "-lstdc++ -lgcc" flags for /usr/bin/libtool not choke, by linking to the static gcc-3.3 versions of libstdc++ and libgcc. But in this case, not choking is a bug. What should be linked in are the dynamic version of libstdc++ and the static version of libgcc, both from gcc-4.0, both to be found in /usr/lib/gcc/powerpc-apple-darwin8/4.0.1/, and both automatically (without need of a -l flag) linked in if you use g++ for linking instead of libtool. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Trylocks in the kernel
by Sam Vaughan 15 Dec '05

15 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On 15/12/2005, at 7:03 AM, darwin-dev-request(a)lists.apple.com wrote: On Dec 13, 2005, at 9:21 AM, Brian Bergstrand wrote: On Dec 13, 2005, at 12:23 AM, Ryan McGann wrote: The new Tiger locking KPIs seem to lack a trylock variant for any of the lock types (mutex, spinlock, read/write). A mutex try variant is available in the Unsupported KPI. It's probably safe to use this for 8.0, but may break in the next major release. All of the try variants are in the kernel proper, so one has to wonder why they are not exported in the KPI. Well they also have a way to promote read-write locks to exclusive from shared exported in the kernel, but not in the KPIs (lck_rw_exclusive_to_shared and lck_rw_shared_to_exclusive). That bug was closed :-( It would be really nice to have those KPIs back in the supported list, if anybody is listening... Hear Hear. _lck_mtx_try_lock _lck_rw_done _lck_rw_lock_exclusive_to_shared_gen _lck_rw_try_lock_exclusive _lck_rw_try_lock_shared _lck_spin_try_lock Sam _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… Amongst the 30 kernel symbols we need that are preventing us from using the 10.4 KPI I see the following locking functions: My bugs have been marked "Duplicate", so someone's aware of the issue. All the same I think we need to push this at WWDC next year. A locking API that doesn't export try functions and requires you to keep track of whether an rwlock is held shared or exclusive when you unlock it simply isn't complete in my opinion. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Libtool (ld) problem in makefile. Is the GCC 4.0 distribution broken?
by Justin C. Walker 14 Dec '05

14 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com On Dec 14, 2005, at 10:25 , Paul Archibald wrote: A couple of things spring to mind, although I'm not the expert here: 1) The "-I" can't have a space following it ("-I/...") Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large Institute for General Semantics -------- Some people have a mental horizon of radius zero, and call it their point of view. -- David Hilbert -------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… $(TARGET): $(OBJECTS) # this is the original libtool line libtool -dynamic -M -single_module -o $@ $^ -lstdc++ -lc -lgcc > map strip -x $@ %.o: %.cpp # this is the original cc line cc -c -O2 -o $@ $^ -I /System/Library/Frameworks/JavaVM.framework/ Headers # I have tried this, too, with the same results # g++ -c -O2 -o $@ $^ -I /System/Library/Frameworks/ JavaVM.framework/Headers 2) Don't use 'cc' to link (or, I think compile, but that may be handled by internal smarts) C++. Use 'c++' or 'g++' (the same thing). If you just pass 'cc' a bunch of .o's, it's got no clue what you want, so it treats the args as .o's compiled from 'c's, and links against the .c runtime. This email sent to site_archiver(a)lists.apple.com

1 0

Libtool (ld) problem in makefile. Is the GCC 4.0 distribution broken?
by Paul Archibald 14 Dec '05

14 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Greetings Comrades, >>> begin makefile >>> TARGET=../libJEngine.jnilib OBJECTS=\ JEngine_Model.o\ JEngine_Parser.o\ # >> more *.o files here << engine/model.o\ engine/imc.o <<< end makefile <<< The result of making this is... >>> begin output >>> ld: for architecture ppc ld: Undefined symbols: __ZN9__gnu_cxx18__exchange_and_addEPVii __ZNSs4_Rep20_S_empty_rep_storageE __ZNSs4_Rep9_S_createEmmRKSaIcE __ZSt18_Rb_tree_decrementPSt18_Rb_tree_node_base __ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_ __ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_ __ZNSt15_List_node_base4hookEPS_ __ZNSt15_List_node_base6unhookEv __ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base libtool: internal link edit command failed make: *** [../libJEngine.jnilib] Error 1 <<< end output <<< _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… First of all, let me apologize if this posting belongs elsewhere, I am not a regular user. If this belongs on another list, let me know. Also, my problem involves Java, although I believe the Java element is not the main issue. (This is an update to a previous posting from several months ago. I have been busy with other aspects of the project, and this part has been lying fallow. Now it is getting in the way, and needs to be addressed.) Here's the situation: I have a makefile that was used to build a shared library (a Java Native Interface library, to be specific). This makefile worked when I had Panther installed, but it broke when I upgraded to Tiger. $(TARGET): $(OBJECTS) # this is the original libtool line libtool -dynamic -M -single_module -o $@ $^ -lstdc++ -lc -lgcc > map strip -x $@ %.o: %.cpp # this is the original cc line cc -c -O2 -o $@ $^ -I /System/Library/Frameworks/JavaVM.framework/ Headers # I have tried this, too, with the same results # g++ -c -O2 -o $@ $^ -I /System/Library/Frameworks/JavaVM.framework/ Headers So, what is going on here? What are these symbols? Are they STL/ template symbols? (Do I care a lot? Not really, I just want this to work.) My experiments with the makefile, including changing cc to gcc or g+ +, or trying to use glibtool instead of libtool, have been unsuccessful. I have tried setting my GCC version to use 3.3 instead of 4.0 (sudo gcc_select 3.3). No luck. Is the GCC distribution broken? Is libtool the culprit, or is it gcc or ld? Are there any known fixes? Where are these undefined symbols defined? Am I missing a library or a header, and if so, which one and how would I find out? (Even if the problem IS the GCC distribution, I am still curious how I would determine where to find the definitions of undefined symbols. Metrowerks supplies a "Find Library" facility which can be used to find symbols. Does GCC have something like that?) (One thing that helped a lot was to use the Disk Utility.app to repair the file permissions on my drive. Before I did that (thanks to whoever suggested that, I forget where I saw it), libtool choked on the -llstdc++ -lc -lgcc section of the makefile, asserting that lstdc+ + and gcc were not found and could not be included. After fixing file permissions, that error went away. I think everyone should run fix permissions.) This email sent to site_archiver(a)lists.apple.com

1 0

Re: smartcard info
by Garth Cummings 14 Dec '05

14 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Programmatic backtrace in userspace application...
by Shawn Erickson 14 Dec '05

14 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=VtAQrbqzw1YP7ILVg6ATbfrCtiWHhgifv0iKYJurReWoep7A0FX0JsM0+/I+7pu7DVuHmnBGq620Ew5yQ/RWhwjqxeug8u9H67EYXwL6jP/ajgXC3i079dxfhA8olq0BnS/uSEPe50jdfXxNWCwYcFMGUcT4CRRpv9QIOylZKys= I am looking for a simple way to log a backtrace to help me understand runtime callers of some code that I may have to modify (based on who is calling it, etc.). I found the following source code example but before I spend time pulling it apart and putting it back together so I can use it in my situation... I have a couple of questions. <http://developer.apple.com/samplecode/MoreIsBetter/listing41.html> 1) Does the system (10.4 and later) provide a backtrace logging facility for MACHO binaries that can be called during runtime in an arbitrary thread? (without faulting the application) 2) If not then is the MoreBacktrace.c code still correct enough for PowerPC based systems using 10.4 and ok to use with gcc 3.3 / 4.0 C++ compiled code? -Shawn _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0

Re: smartcard info
by Kevin Van Vechten 14 Dec '05

14 Dec '05

site_archiver(a)lists.apple.com Delivered-To: darwin-dev(a)lists.apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-dev mailing list (Darwin-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-dev/site_archiver%40lists.app… This email sent to site_archiver(a)lists.apple.com

1 0