Re: Using Flex/Lex in a Cocoa project
Re: Using Flex/Lex in a Cocoa project
- Subject: Re: Using Flex/Lex in a Cocoa project
- From: "Michael Ash" <email@hidden>
- Date: Mon, 18 Aug 2008 23:57:02 -0400
On Mon, Aug 18, 2008 at 4:55 PM, Ricky Sharp <email@hidden> wrote:
>
> On Aug 18, 2008, at 3:40 PM, mm w wrote:
>
>> to avoid the splitting problem
>>
>> (c < 128) ? "%c" : "\\ux", c);
>
> I'm not sure what this solves.
>
> Per Michael's e-mail below, this is indeed a difficult problem. UTF-8 is
> just a particular scheme to store Unicode strings. Operating on individual
> bytes in such streams will most likely not make any sense.
>
> What I would do is pick some normalized form and operate on that data. For
> a recent feature at my day job, we normalized all input CSV files to
> UTF-16BE. We were able to handle all of our customer data so far. The
> final solution still isn't 100% Unicode-savvy (e.g. it does crap-out with
> surrogate pairs), but we have unit tests to expose/document such
> limitations. And, customer data doesn't yet have such things.
Note that depending on what kind of results you want, even if all of
your data is within the BMP, this *still* won't save you.
As a really basic example, consider a simple, obvious character like
é. (That's an e with an acute accent on it if you're having unicode
trouble in your e-mail client.) That can be represented as two
separate unicode code points, a plain old ASCII e followed by a
combining accent mark. If you should happen to split the string on the
accent mark, such that the e goes into the first half and the
combining accent mark goes into the second half, you get a really
unintuitive result. What appears to the user to be a single character
gets suddenly blown in two. Worse, if you happen to insert a string in
the middle, you could end up applying that acute accent to some
*other* letter instead.
And if you think this is bad, you should see how Unicode deals with Korean.
If you're using NSString, you can find good places to split using the
-rangeOfComposedCharacterSequenceAtIndex: method. I believe that it
will also deal with surrogate pairs, not only "normal" composed
character sequences.
Ultimately if you're doing any manipulation of Unicode, some large
amount of knowledge about Unicode needs to be in the system somewhere.
If your code is running on a Mac then you can use the knowledge that
NSString has about Unicode to help out, sometimes. But alas, due to
how Unicode is designed, there's simply no way to safely manipulate
strings beyond very basic operations like concatenation unless you
either make the code know a lot about Unicode or place overly strong
constraints on the system, such as only splitting on line breaks or
carriage returns (or commas).
Yeah, the situation kind of sucks, but it's what we're stuck with.
Thankfully Foundation and CoreFoundation do a lot to hide the messy,
ugly details from us.
Mike
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden