Re: Binary floating point format
Re: Binary floating point format
- Subject: Re: Binary floating point format
- From: Pascal Bourguignon <email@hidden>
- Date: Fri, 31 Mar 2017 03:53:14 +0200
> On 31 Mar 2017, at 02:25, Carl Hoefs <email@hidden> wrote:
>
> I have megabytes of raw legacy science datasets that I'm trying to read into my app and ingest into an array of doubles. The data is supposed to be organized as a stream of 8-byte doubles. I do not know how these datasets were generated, so I don't know what format (big/little endian, byte swapped, etc) they are in.
>
> If I read the data directly into double variables, they evaluate as very small (E-92, etc).
>
> I've tried all of these functions:
>
> NSSwapDouble()
> NSSwapBigDoubleToHost()
> NSSwapLittleDoubleToHost()
> NSConvertSwappedDoubleToHost()
> CFConvertDoubleSwappedToHost()
>
> The above functions generally return invalidly large values (E+87, etc).
>
> Here is a hex dump of 4 binary doubles:
>
> 49BF7DE372533C05 A8C02FE3135B4F09 86C22FE37E630B05 27C2C4E3E258BA08
>
> It seems there's some structure to them, as the last byte is always in the range of 03 to A0, so maybe they somehow correspond to IEEE 754 double-precision binary floating point format?
>
> Is there yet another function I could use to parse this binary data?
Do these values look more sensible?
Perhaps it’s IBM floating points?
cl-user> (let ((ds '(#x1122334455667788 #x49BF7DE372533C05
#xA8C02FE3135B4F09 #x86C22FE37E630B05 #x27C2C4E3E258BA08)))
(dolist (swap '(identity swap64/8 swap64/32 swap-32-32))
(dolist (n ds (terpri))
(format t "~8,'0X ~~> ~8,'0X -> ~A~%" n (funcall swap n)
(ibm-to-double-float (funcall swap n))))))
1122334455667788 ~> 1122334455667788 -> 2.3754432874621412D-67
49BF7DE372533C05 ~> 49BF7DE372533C05 -> 5.202153205871582D0
A8C02FE3135B4F09 ~> A8C02FE3135B4F09 -> -1.0481728277757355D-39
86C22FE37E630B05 ~> 86C22FE37E630B05 -> -1.3051642321919604D-80
27C2C4E3E258BA08 ~> 27C2C4E3E258BA08 -> 6.365827421850181D-41
1122334455667788 ~> 8877665544332211 -> -1.7249758893089965D-78
49BF7DE372533C05 ~> 53C5372E37DBF49 -> 1.0356655888799012D-81
A8C02FE3135B4F09 ~> 94F5B13E32FC0A8 -> 2.5774949195090032D-77
86C22FE37E630B05 ~> 50B637EE32FC286 -> 3.933545844594491D-82
27C2C4E3E258BA08 ~> 8BA58E2E3C4C227 -> 6.637651455481117D-78
1122334455667788 ~> 5566778811223344 -> 6.01657272434688D+14
49BF7DE372533C05 ~> 72533C0549BF7DE3 -> 2.798654988457155D+50
A8C02FE3135B4F09 ~> 135B4F09A8C02FE3 -> 1.1405782651894492D-64
86C22FE37E630B05 ~> 7E630B0586C22FE3 -> 7.98839220202526D+64
27C2C4E3E258BA08 ~> E258BA0827C2C4E3 -> -1.5431197015853318D+31
1122334455667788 ~> 4433221188776655 -> 7.11677876097383D-6
49BF7DE372533C05 ~> E37DBF49053C5372 -> -7.647223718743149D+31
A8C02FE3135B4F09 ~> E32FC0A8094F5B13 -> -1.0059537510577916D+32
86C22FE37E630B05 ~> E32FC286050B637E -> -1.4436818052519877D+31
27C2C4E3E258BA08 ~> E3C4C22708BA58E2 -> -2.3622313801204285D+32
Or perhaps it’s Microsoft floating points?
cl-user> (let ((ds '(#x1122334455667788 #x49BF7DE372533C05
#xA8C02FE3135B4F09 #x86C22FE37E630B05 #x27C2C4E3E258BA08)))
(dolist (swap '(identity swap64/8 swap64/32 swap-32-32))
(dolist (n ds (terpri))
(format t "~8,'0X ~~> ~8,'0X -> ~A~%" n (funcall swap n)
(ibm-to-double-float (funcall swap n))))))
1122334455667788 ~> 1122334455667788 -> 1.2202591425418212D-34
49BF7DE372533C05 ~> 49BF7DE372533C05 -> -1.0380788666241747D-17
A8C02FE3135B4F09 ~> A8C02FE3135B4F09 -> -4.1271856580565436D+11
86C22FE37E630B05 ~> 86C22FE37E630B05 -> -24.273383128545337D0
27C2C4E3E258BA08 ~> 27C2C4E3E258BA08 -> -6.145829211972407D-28
1122334455667788 ~> 8877665544332211 -> 123.69986928104575D0
49BF7DE372533C05 ~> 53C5372E37DBF49 -> 3.4590018121487237D-38
A8C02FE3135B4F09 ~> 94F5B13E32FC0A8 -> 6.093638471033148D-37
86C22FE37E630B05 ~> 50B637EE32FC286 -> 2.5601652562851923D-38
27C2C4E3E258BA08 ~> 8BA58E2E3C4C227 -> -2.7381261883853355D-37
1122334455667788 ~> 5566778811223344 -> 5.117393619226236D-14
49BF7DE372533C05 ~> 72533C0549BF7DE3 -> 2.5181109391411485D-5
A8C02FE3135B4F09 ~> 135B4F09A8C02FE3 -> 6.5995823691006745D-34
86C22FE37E630B05 ~> 7E630B0586C22FE3 -> 0.11086086575985371D0
27C2C4E3E258BA08 ~> E258BA0827C2C4E3 -> 1.3414732167249665D+29
1122334455667788 ~> 4433221188776655 -> 3.034632414629399D-19
49BF7DE372533C05 ~> E37DBF49053C5372 -> 3.1412382858272243D+29
A8C02FE3135B4F09 ~> E32FC0A8094F5B13 -> 2.175711360216376D+29
86C22FE37E630B05 ~> E32FC286050B637E -> 2.1758016487161476D+29
27C2C4E3E258BA08 ~> E3C4C22708BA58E2 -> -2.4357511146913354D+29
You have to choose the byte swapping.
https://en.wikipedia.org/wiki/IBM_Floating_Point_Architecture#Double-precision_64-bit <https://en.wikipedia.org/wiki/IBM_Floating_Point_Architecture#Double-precision_64-bit>
If that’s the case, then the first (big-endian) byte order is probably the right one,
but the example data you gave may be wrong. Instead of dumping the file 8 octets by 8 octets, you should dump it octet by octet to preserve the byte order.
https://en.wikipedia.org/wiki/Microsoft_Binary_Format#Technical_details <https://en.wikipedia.org/wiki/Microsoft_Binary_Format#Technical_details>
In that case, then I would assume little-endian is perhaps the right byte order.
[pjb@despina :0.0 ~]$ od -t x8 foo
0000000 400921fb54442d18 4083a28c59d5433b
0000020 40feadfb4c5d390c 4177f7ec53a8d491
0000040
[pjb@despina :0.0 ~]$ od -t x1 foo
0000000 18 2d 44 54 fb 21 09 40 3b 43 d5 59 8c a2 83 40
0000020 0c 39 5d 4c fb ad fe 40 91 d4 a8 53 ec f7 77 41
0000040
--
__Pascal J. Bourguignon__
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden