April 2006 - Darwin-kernel - lists.apple.com

Re: Socket filtering
by Peter Lovell 28 Apr '06

28 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Apr 26, 2006, at 7:49 PM, Eric Long wrote: I'm not sure what you are trying to accomplish here. When the socket is closed, any uninjected data should be discarded. You are asking for a hook to use as a warning that the socket will close. Do you mean a hook before the client calls close on the socket? I'm not sure how one would predict such a thing. A hook after the client calls close seems useless because the client is done calling read, so there is no point in injecting data. Cheers.....Peter _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Well, I'm scanning data looking for pattern matches. I don't necessarily know the protocol, so I don't know for sure how much data to expect. If I find a partial match in a packet, I want to see if the next packet contains the end of the data I am looking for, so I swallow the packet and wait for the next packet. If it turns out there is no next packet, I've got a problem. If I discard the data, I've messed up the stream. I need to properly reinject it before the socket closes. Eric And of course you might split the packet, sending what seems OK and retaining the rest until you know. Assuming the protocol allows splits, that is. Of course, if the send and receive sides are closely dependent, holding any data might cause a logical deadlock between the local and remote ends. But if you're looking for a more contained hunk then it may just be buffer boundaries and you'll soon get the remaining piece. This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Eric Long 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Microsoft-Entourage/11.1.0.040913 > A few observations: Thanks for your help. I've received a lot of good feedback here, so I think I'll just go chew on it for awhile. Thanks to all for the comments. Eric _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Michael Smith 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Apr 27, 2006, at 10:14 AM, Eric Long wrote: If he's just monitoring data content, though, he should make a local copy for however big his buffer size is, and just let it through, unless he's trying to catch bad data before it gets in, and wants to check data spanning a buffer boundary, rather than just monitoring. But if the buffer that was swallowed doesn't itself contain the data you're trying to stop, then sending a partial buffer, and keeping a local copy in the kext for the spanning check, rather than swallowing it, is still an OK thing to do (assuming the data isn't malicious unless complete - and if it's malicious when incomplete, then the check should have matched already on the packet that was swallowed. This suggests that your "maliciousness" analysis could use some work. But if the socket closes and I don't get the data back into the stream before that, the transaction ends up incomplete and its my fault. I What sort of "match"? If I knew there was no more data to come, I wouldn't want to hold the packet. It sounds a lot to me like your basic analysis algorithm needs work. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… That's a bit broad of an assumption. The previous packet might have just a couple of bytes that match. It's hard to say how much data is quantitatively malicious if its sensitive data. I can thwart a transaction when I know I've matched a pattern I'm looking for, but if I don't hold packets, a partial data transfer might actually be detrimental. However, it also suggests that you are at least in part parsing a structured exchange between two parties. I'm scanning streams that other processes are conducting using an NKE socket filter. If I swallow a packet it is only intended to be temporary, until I can either get more data and determine to reinsert it based on the new data or arbitrarily set a timeout to reinsert it if no more data was received. I assume that there's more to it than this; if you decide to "thwart" the transaction you're not going to reinsert the packet. I think this is the part that's confusing us. Who is closing the socket? Are you talking about a remote close, or a local close? swallowed the packet only because I found a partial match at the end of its data holding out the possibility that more data was coming that might demonstrate a true match. Let me hazard a guess; you are trying to find specific data in email or HTTP transactions, perhaps as part of the "icSPYWARE & ANTI PHISHING" suite, using a naive string matching algorithm and attempting to avoid knowing anything about the framing of the data in these transactions. If you were to avail yourself of said framing information, or better still, do this at a level where the framing was implicit, none of this would be an issue. Information systems tend to be structured, and working within the structure is by definition easier than ignoring it. Frequently the answer to "why is X hard?" is "because you should not be doing X". This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Justin C. Walker 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Apr 27, 2006, at 10:14 , Eric Long wrote: If he's just monitoring data content, though, he should make a local copy for however big his buffer size is, and just let it through, unless he's trying to catch bad data before it gets in, and wants to check data spanning a buffer boundary, rather than just monitoring. But if the buffer that was swallowed doesn't itself contain the data you're trying to stop, then sending a partial buffer, and keeping a local copy in the kext for the spanning check, rather than swallowing it, is still an OK thing to do (assuming the data isn't malicious unless complete - and if it's malicious when incomplete, then the check should have matched already on the packet that was swallowed. If I knew there was no more data to come, I wouldn't want to hold the packet. Regards, Justin -- Justin C. Walker, Curmudgeon-At-Large, Director Institute for the Enhancement of the Director's Income -------- The path of least resistance: it's not just for electricity any more. -------- _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… That's a bit broad of an assumption. The previous packet might have just a couple of bytes that match. It's hard to say how much data is quantitatively malicious if its sensitive data. I can thwart a transaction when I know I've matched a pattern I'm looking for, but if I don't hold packets, a partial data transfer might actually be detrimental. I'm scanning streams that other processes are conducting using an NKE socket filter. If I swallow a packet it is only intended to be temporary, until I can either get more data and determine to reinsert it based on the new data or arbitrarily set a timeout to reinsert it if no more data was received. But if the socket closes and I don't get the data back into the stream before that, the transaction ends up incomplete and its my fault. I swallowed the packet only because I found a partial match at the end of its data holding out the possibility that more data was coming that might demonstrate a true match. A few observations: - if the underlying protocol is stream-based (like TCP), you will see a packet terminating the connection, coming from the local endpoint. You won't know whether this packet is "graceful" or "prejudicial" :-} If "graceful", you should push the data out; otherwise, not. Oh, and not all stream protocols distinguish (leaving it to higher layers to complete a handshake to terminate). - if the underlying protocol is datagram-based, you are hosed. It is possible that the local endpoint-owner will call a "disconnect" procedure for the socket, but that need not translate into a real packet. Even if there is a "hook" to give you control in this case, most UDP- spewers don't do it, so the only way you know that there aren't any more packets coming is that there aren't any more packets coming :-} In short, for real streams, you can fake it by looking for terminating (e.g., FIN) packets. Otherwise, you have no idea what comes next, or when. It also occurs to me that by holding onto a packet, waiting for "the next one", you are screwing around with the transport's idea of timing, and potentially harming the connection, and perhaps the data it carries. Just keeping a copy of the current packet and pushing the packet out doesn't seem like what you want (unless you just want to see what passes by), so it's not clear to me how to get the 100% solution... This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Eric Long 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Microsoft-Entourage/11.1.0.040913 > If he's just monitoring data content, though, he should make a local > copy for however big his buffer size is, and just let it through, > unless he's trying to catch bad data before it gets in, and wants to > check data spanning a buffer boundary, rather than just monitoring. > > But if the buffer that was swallowed doesn't itself contain the data > you're trying to stop, then sending a partial buffer, and keeping a > local copy in the kext for the spanning check, rather than swallowing > it, is still an OK thing to do (assuming the data isn't malicious > unless complete - and if it's malicious when incomplete, then the > check should have matched already on the packet that was swallowed. That's a bit broad of an assumption. The previous packet might have just a couple of bytes that match. It's hard to say how much data is quantitatively malicious if its sensitive data. I can thwart a transaction when I know I've matched a pattern I'm looking for, but if I don't hold packets, a partial data transfer might actually be detrimental. I'm scanning streams that other processes are conducting using an NKE socket filter. If I swallow a packet it is only intended to be temporary, until I can either get more data and determine to reinsert it based on the new data or arbitrarily set a timeout to reinsert it if no more data was received. But if the socket closes and I don't get the data back into the stream before that, the transaction ends up incomplete and its my fault. I swallowed the packet only because I found a partial match at the end of its data holding out the possibility that more data was coming that might demonstrate a true match. If I knew there was no more data to come, I wouldn't want to hold the packet. Eric _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

RE: Socket filtering
by Sanjay Mishra 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Thread-index: AcZpjEjyk4U3BCl3RraADZ2EH76y6gAb09ew Thread-topic: Socket filtering There are two situations here. I don't know which you are talking about. One, each such session should have complete pattern matching. If any part of the data is not matched I have to drop complete data. Second, It is possible that the task can be done in multiple sessions.So I will retain the pattern matched data and will discard only unmatched data. I will close my side of the connection if I don't get any data for a long time. Better still I will send "Are you alive?" message to him now and then if I am not getting any data for some specified time and I am expecting data to come. Which kind of situation is this? --Sanjay -----Original Message----- From: darwin-kernel-bounces+samishra=quark.com(a)lists.apple.com [mailto:darwin-kernel-bounces+samishra=quark.com@lists.apple.com] On Behalf Of Eric Long Sent: Thursday, April 27, 2006 5:20 AM To: Josh Graessley Cc: Darwin Kernel Subject: Re: Socket filtering > > I'm not sure what you are trying to accomplish here. > > When the socket is closed, any uninjected data should be discarded. > > You are asking for a hook to use as a warning that the socket will > close. Do you mean a hook before the client calls close on the socket? > I'm not sure how one would predict such a thing. A hook after the > client calls close seems useless because the client is done calling > read, so there is no point in injecting data. Well, I'm scanning data looking for pattern matches. I don't necessarily know the protocol, so I don't know for sure how much data to expect. If I find a partial match in a packet, I want to see if the next packet contains the end of the data I am looking for, so I swallow the packet and wait for the next packet. If it turns out there is no next packet, I've got a problem. If I discard the data, I've messed up the stream. I need to properly reinject it before the socket closes. Eric _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/samishra%40quark.co m This email sent to samishra(a)quark.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Terry Lambert 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Unless it was a notification from something like: shutdown(s,SHUT_WR); -- Terry Ronnie On Apr 26, 2006, at 5:46 PM, Michael Smith wrote: On Apr 26, 2006, at 4:49 PM, Eric Long wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/rgmisra%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/tlambert%40apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… (Not that I think that's likely). In that case, the local client would expect to still get data, but not send it. That probably wouldn't do what he thinks he wants, though -- the problem is that you could get an infinite amount of data after that, and it might include the data you don't want to get through, so you can't just flush the packet on out. If he's just monitoring data content, though, he should make a local copy for however big his buffer size is, and just let it through, unless he's trying to catch bad data before it gets in, and wants to check data spanning a buffer boundary, rather than just monitoring. But if the buffer that was swallowed doesn't itself contain the data you're trying to stop, then sending a partial buffer, and keeping a local copy in the kext for the spanning check, rather than swallowing it, is still an OK thing to do (assuming the data isn't malicious unless complete - and if it's malicious when incomplete, then the check should have matched already on the packet that was swallowed. I expect that sending it on would be his way of discarding the buffer without having to actually do the necessary record keeping (just expecting the socket close code to do it for him, instead). If that's the case, then the issue is a race - and what he needs to do is a notification interlock on the close, so that the data gets propagated or dropped - if not by the close of the socket, then by himself. Otherwise he might end up with a slow leak, especially if his pattern matching is pretty slow. On Apr 26, 2006, at 6:23 PM, Ronnie Misra wrote: Perhaps the client is closing the socket because they time out waiting for the packet that you've swallowed? Anyway, as Mike points out, once the client has decided to close the socket, you're too late. And there is nothing the system could do to warn you before the client makes the decision to close the socket... You are asking for a hook to use as a warning that the socket will close. Do you mean a hook before the client calls close on the socket? I'm not sure how one would predict such a thing. A hook after the client calls close seems useless because the client is done calling read, so there is no point in injecting data. Well, I'm scanning data looking for pattern matches. I don't necessarily know the protocol, so I don't know for sure how much data to expect. If I find a partial match in a packet, I want to see if the next packet contains the end of the data I am looking for, so I swallow the packet and wait for the next packet. If it turns out there is no next packet, I've got a problem. If I discard the data, I've messed up the stream. I need to properly reinject it before the socket closes. Why? The socket is closing, nobody will ever read the data. Just throw it away; that's all the system would do if you were to reinject it anyway... This email sent to rgmisra(a)apple.com This email sent to tlambert(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Ronnie Misra 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com Ronnie On Apr 26, 2006, at 5:46 PM, Michael Smith wrote: On Apr 26, 2006, at 4:49 PM, Eric Long wrote: = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Perhaps the client is closing the socket because they time out waiting for the packet that you've swallowed? Anyway, as Mike points out, once the client has decided to close the socket, you're too late. And there is nothing the system could do to warn you before the client makes the decision to close the socket... You are asking for a hook to use as a warning that the socket will close. Do you mean a hook before the client calls close on the socket? I'm not sure how one would predict such a thing. A hook after the client calls close seems useless because the client is done calling read, so there is no point in injecting data. Well, I'm scanning data looking for pattern matches. I don't necessarily know the protocol, so I don't know for sure how much data to expect. If I find a partial match in a packet, I want to see if the next packet contains the end of the data I am looking for, so I swallow the packet and wait for the next packet. If it turns out there is no next packet, I've got a problem. If I discard the data, I've messed up the stream. I need to properly reinject it before the socket closes. Why? The socket is closing, nobody will ever read the data. Just throw it away; that's all the system would do if you were to reinject it anyway... _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/rgmisra% 40apple.com This email sent to rgmisra(a)apple.com This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Michael Smith 27 Apr '06

27 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com On Apr 26, 2006, at 4:49 PM, Eric Long wrote: You are asking for a hook to use as a warning that the socket will close. Do you mean a hook before the client calls close on the socket? I'm not sure how one would predict such a thing. A hook after the client calls close seems useless because the client is done calling read, so there is no point in injecting data. = Mike _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… Well, I'm scanning data looking for pattern matches. I don't necessarily know the protocol, so I don't know for sure how much data to expect. If I find a partial match in a packet, I want to see if the next packet contains the end of the data I am looking for, so I swallow the packet and wait for the next packet. If it turns out there is no next packet, I've got a problem. If I discard the data, I've messed up the stream. I need to properly reinject it before the socket closes. Why? The socket is closing, nobody will ever read the data. Just throw it away; that's all the system would do if you were to reinject it anyway... This email sent to site_archiver(a)lists.apple.com

1 0

Re: Socket filtering
by Eric Long 26 Apr '06

26 Apr '06

site_archiver(a)lists.apple.com Delivered-To: darwin-kernel(a)lists.apple.com User-agent: Microsoft-Entourage/11.1.0.040913 > > I'm not sure what you are trying to accomplish here. > > When the socket is closed, any uninjected data should be discarded. > > You are asking for a hook to use as a warning that the socket will > close. Do you mean a hook before the client calls close on the > socket? I'm not sure how one would predict such a thing. A hook after > the client calls close seems useless because the client is done > calling read, so there is no point in injecting data. Well, I'm scanning data looking for pattern matches. I don't necessarily know the protocol, so I don't know for sure how much data to expect. If I find a partial match in a packet, I want to see if the next packet contains the end of the data I am looking for, so I swallow the packet and wait for the next packet. If it turns out there is no next packet, I've got a problem. If I discard the data, I've messed up the stream. I need to properly reinject it before the socket closes. Eric _______________________________________________ Do not post admin requests to the list. They will be ignored. Darwin-kernel mailing list (Darwin-kernel(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/darwin-kernel/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0