Re: String Encoding Detection (Revisited)
Re: String Encoding Detection (Revisited)
- Subject: Re: String Encoding Detection (Revisited)
- From: Dustin Voss <email@hidden>
- Date: Thu, 7 Aug 2003 12:00:56 -0700
On Thursday, August 7, 2003, at 10:11 AM, Dustin Voss wrote:
On Thursday, August 7, 2003, at 01:44 AM, Francisco Tolmasky wrote:
Ok, so I recently posted a question about auto-detecting string
encodings, and also looked through the archives. Basically there's
no way unless it is unicode and has a BOM. I still want an
auto-detect feature though, like BBEdit's. So basically, how do I
check for a BOM (I check TextEdit's code, couldn't find it, found
lots of other stuff though). Anyways, other than that and doing some
weird spell checking thing someone suggested (Using spellchecker to
see if the string makes sense or not, which would be pretty useless
if it's code or anything other than pure sentences), are there any
other "tricks"?
And when all else fails and I resort to just using an encoding, which
one should I choose mac os roman, ascii, utf-8?
I don't know about tricks, but the BOM will be one of the following:
UTF-16 BE: FE FF
UTF-16 LE: FF FE
UTF-8: EF BB BF
Additionally, valid UTF-8 follows these rules
(
http://www.faqs.org/rfcs/rfc2279.html):
1. Any byte C0-FD will always be followed by at least one byte 80-BF.
2. Any byte 00-7F will never be followed by a byte 80-BF.
3. There will never be a byte FE or a byte FF.
Valid UTF-16 follows these rules (
http://www.ietf.org/rfc/rfc2781.txt)
once endian-ness is taken care of:
1. A word D800-DBFF will always be followed by one word DC00-DFFF.
2. A word DC00-DFFF will only follow a word D800-DBFF.
3. Words FFFF and FFFE are invalid.
There is some open-source Unicode validation code at
http://oss.software.ibm.com/icu/. It is more thorough than the above,
since I think it covers unassigned characters and invalid surrogate
pairs as well.
I now return you to your regularly scheduled programming. (Heh.
"Programming.")
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.