[Fed-Talk] Apple Mail "Hanging" Reading DoD CAC-Signed E-mail (Problem with "trustd")
[Fed-Talk] Apple Mail "Hanging" Reading DoD CAC-Signed E-mail (Problem with "trustd")
- Subject: [Fed-Talk] Apple Mail "Hanging" Reading DoD CAC-Signed E-mail (Problem with "trustd")
- From: Basil Decina <email@hidden>
- Date: Fri, 28 Jul 2017 16:20:09 -0400
Apple Mail Users…
I’m sending this in case others have hit a problem reading certain DoD
CAC-signed e-mail using Apple Mail...
Here’s an update to the CAC issue that I’ve been having with Apple Mail since
February 2017 (when I upgraded to Sierra). It was briefly discussed under the
thread "Best Alternative to Apple Mail that does S/MIME” as I was looking to
abandon Apple Mail (yes, the problem was that bad). Alternatives to Apple Mail
were not jumping out and my issue didn’t seem to be shared by others (taking 30
minutes to open certain CAC-signed messages).
But finally, after almost 6 months, I have a work-around to make Apple Mail
usable again. I’m sharing this with others if they see similar problems.
Background and Problem...
A change to cert validation processing was made in Sierra (sometime between the
last El Capitan 10.11.6 and early Sierra 10.12.3). After upgrading to macOS
10.12.3 (from 10.11.6) back in February 2017, my Apple Mail took "forever" to
download/read/open/touch certain messages (messages would not download, viewing
panes of existing messages would spin with the “Loading” graphic and message
moves would stall). The problem was exacerbated when I tried to move blocks of
messages from one folder to another or rebuild folders. After literally
spending more than 2 solid weeks of run-time rebuilding a few hundred thousand
messages (20+GBytes) I managed to isolate the problem to only CAC-signed
messages (one might think the sheer number of messages I squirreled away was
the problem — but it was later discovered to be unrelated). After another few
months, I finally traced the problem down to specific messages that were
CAC-signed (or encrypted) with a cert whose embedded e-mail address no longer
matched the sender address (“From:”) *AND* whose certs were signed by “DoD Root
CA 2”. These “problem” messages were mostly from people who moved to
“mail.mil" but kept using their their older certs, although those who had newer
“mail.mil” certs but continued to use their old e-mail “From:” were also
affected. But, interestingly enough, certs signed by “DoD Root CA 3” where not
affected — even if their cert address did not match their “From:” address.
Lastly, the imprecise “forever” unit of time was measured to be about 30
minutes — so every time Apple Mail did anything to one of the “problem”
messages, the process hung for 30 minutes. (1000’s of messages with this
condition translated to many 100’s of hours.)
I was living with this, trying to fix it in my spare time, until I upgraded to
macOS 10.12.6 earlier this week which made the problem worse — it essentially
bricked Apple Mail. “Forever” changed from 30 minutes to infinity (or at least
24 hours as problem messages would not open even after Mail was running for 24
hours) and Apple Mail became unusable. So whatever the problem introduced in
Sierra was, it became worse in the 10.12.6 update.
Desperation and Cry for Help...
At this point the major annoyance became a work stoppage and I found out how to
do dangerous things like turning off “System Integrity Protection”! (Previous
month-long attempts at standard rebuilding, re-installing macOS and running
various disk utilities programs did not help.) I needed a surgical instrument
to rattle the bits floating around my computer. I started removing
impossible-to-remove files (like the deprecated “X509Anchors” Keychain which
had an old “DoD Root CA 2” cert) and making other changes but no luck. Then I
got really desperate — I asked for help. Specifically from one of our Unix
kernel folks to try things like DTrace to isolate exactly which process was
causing the problem.
However, like all good quiet, backroom Unix types, once he started looking at
the problem and tried “this and that”, he found the root cause and fixed it —
or at least dampened the effects to make my Apple Mail usable again. He traced
the problem to “trustd”, an apparently new (or at least majorly re-written)
daemon in Sierra.
Problem Discovered...
“trustd” is bad. Maybe good intent, but bad implementation. “trustd” was
found to be running somewhat rampant — more than one instance at a time and
looping on retries to retrieve information (CRLs, CA Certs, etc) from old CAs
that pointed to stale web sites. Instead of realizing that old cert
information was no longer available and to stop trying, “trustd” continued to
reach out to non-existent sites and create unexplainable processing logjams.
For example, mismatched message/cert addresses under “DoD Root CA 2” wreaked
havoc for “trustd" but mismatched message/cert addresses under “DoD Root CA 3”
did not.
Work-around...
In the end, we cleaned up (deleted) a number of old certs (intermediate and CA
certs) whose embedded URLs were dead. The simple act of removing expired certs
suddenly changed “forever” from 30-minutes (and infinity) down to 2-3 seconds.
So… The problem is still there — mismatched addresses under “DoD Root CA 2”
are still causing processing delays to Apple Mail (and also Keychain) — but 2-3
seconds per message (and now “instant” Keychain response) is a whole lot better
than waiting between 30-minutes and “forever” to read a message.
Real Fix...
It would be good if “traced” where either removed or re-written to better
handle bad information. [Don’t know if Apple developers still read this
thread.]
In the meantime, if you have this problem, removing old certs is a good
work-around (note this is a very manual process — especially since “Keychain
First Aid” was removed years ago).
BTW, for the geeks in the crowd, enclosed are the notes that Ken Hornstein, our
making-macOS-useful-again person, wrote up…
Basil
-------------------------------------------------------------
Notes from Ken Hornstein in tracking down the “trustd” issue…
-------------------------------------------------------------
The root problem was Apple Mail would not display messages that were signed by
certificates issued by a particular CA ("DoD Root CA 2") AND had an email in
the certificate that did not match the sender’s email address. By "not
display", it would show the spinning “Loading..." indicator in the message
pane. Mismatched signed email messages displayed fine from other CAs, and
messages signed by certificates issued by "DoD Root CA 2" where the certificate
email address matched the sender’s email address also were fine.
A system call trace on Mail.app was not very useful; it wasn't doing anything
obvious, and it looked like it was performing some IPC and waiting for a
response from another program. However, a network trace showed some very
interesting information; the machine (which was only running Mail.app, no
other programs, and had been recently rebooted) was constantly contacting a
bunch of servers all over the Internet asking for certificate information.
Some sleuthing lead to a daemon called "trustd" which seems to be the
gatekeeper for certificate validation. A system call trace on the “trustd"
instance running under the user's userid showed that it was the one responsible
for issuing all of the requests from various servers for certificate
information. Further inspection showed that it was issuing the same requests
over and over for the same collection of certificates; even more unusual,
these were for certificates that were not related to the email in question at
all. It seemed like a lot of the requests for lists of intermediate
certificates, and the root certificates in question were expired so the servers
were returning "404 Not Found” for the requests.
Matching the request URLs for certificates in the keychain, the offending
certificates were deleted from the user's login keychain and “trustd" was then
restarted. This allowed the offending message in Mail.app to display after a
lengthy delay (maybe 30 seconds). It was discovered that trustd was still
issuing a series of certificate requests for a different set of expired
certificates, so those offending certificates were removed and “trustd" was
restarted. After every new round of certificate deletions, the delay to
display the offending message in Mail.app was reduced, until finally the
original message that could not be displayed could successfully be viewed with
minimal delay (1-2 seconds).
My guess is for whatever reason “trustd" decided that it needed to download
some sort of information for these expired root certificates and there is a bug
where it does not handle a "404" error properly and causes “trustd" to go into
a loop continually requesting the certificate information over and over and it
reached a point where all of the worker threads were busy processing these
requests, which left no workers free to respond with the other verification
requests from programs like Mail.app. Why this only affected certificates
issued from "DoD Root CA 2", I have no idea.
--Apple-Mail=_5A556A33-88BF-4148-8922-40A0E10F9F6F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset="utf-8"
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body
style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class=""><div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div
style="font-family: monospace; font-size: 11px;" class="">Apple Mail
Users…</div><div style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;"
class="">I’m sending this in case others have hit a problem reading certain DoD
CAC-signed e-mail using Apple Mail...</div><div style="font-family: monospace;
font-size: 11px;" class=""><br class=""></div><div style="font-family:
monospace; font-size: 11px;" class="">Here’s an update to the CAC issue that
I’ve been having with Apple Mail since February 2017 (when I upgraded to
Sierra). It was briefly discussed under the thread "Best Alternative
to Apple Mail that does S/MIME” as I was looking to abandon Apple Mail (yes,
the problem was that bad). Alternatives to Apple Mail were not jumping
out and my issue didn’t seem to be shared by others (taking 30 minutes to open
certain CAC-signed messages).</div><div style="font-family: monospace;
font-size: 11px;" class=""><br class=""></div><div style="font-family:
monospace; font-size: 11px;" class="">But finally, after almost 6 months, I
have a work-around to make Apple Mail usable again. I’m sharing this with
others if they see similar problems.</div><div style="font-family: monospace;
font-size: 11px;" class=""><br class=""></div><div style="font-family:
monospace; font-size: 11px;" class="">Background and Problem...</div><div
style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;" class="">A
change to cert validation processing was made in Sierra (sometime between the
last El Capitan 10.11.6 and early Sierra 10.12.3). After upgrading to
macOS 10.12.3 (from 10.11.6) back in February 2017, my Apple Mail took
"forever" to download/read/open/touch certain messages (messages
would not download, viewing panes of existing messages would spin with the
“Loading” graphic and message moves would stall). The problem was
exacerbated when I tried to move blocks of messages from one folder to another
or rebuild folders. After literally spending more than 2 solid weeks of
run-time rebuilding a few hundred thousand messages (20+GBytes) I managed
to isolate the problem to only CAC-signed messages (one might think the sheer
number of messages I squirreled away was the problem — but it was later
discovered to be unrelated). After another few months, I finally traced
the problem down to specific messages that were CAC-signed (or encrypted) with
a cert whose embedded e-mail address no longer matched the sender address
(“From:”) *AND* whose certs were signed by “DoD Root CA 2”. These
“problem” messages were mostly from people who moved to “mail.mil" but
kept using their their older certs, although those who had newer “mail.mil”
certs but continued to use their old e-mail “From:” were also affected.
But, interestingly enough, certs signed by “DoD Root CA 3” where not
affected — even if their cert address did not match their “From:” address.
Lastly, the imprecise “forever” unit of time was measured to be about 30
minutes — so every time Apple Mail did anything to one of the “problem”
messages, the process hung for 30 minutes. (1000’s of messages with this
condition translated to many 100’s of hours.)</div><div style="font-family:
monospace; font-size: 11px;" class=""><br class=""></div><div
style="font-family: monospace; font-size: 11px;" class="">I was living with
this, trying to fix it in my spare time, until I upgraded to macOS 10.12.6
earlier this week which made the problem worse — it essentially bricked Apple
Mail. “Forever” changed from 30 minutes to infinity (or at least 24 hours
as problem messages would not open even after Mail was running for 24 hours)
and Apple Mail became unusable. So whatever the problem introduced in
Sierra was, it became worse in the 10.12.6 update.</div><div
style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;"
class="">Desperation and Cry for Help...</div><div style="font-family:
monospace; font-size: 11px;" class=""><br class=""></div><div
style="font-family: monospace; font-size: 11px;" class="">At this point the
major annoyance became a work stoppage and I found out how to do dangerous
things like turning off “System Integrity Protection”! (Previous
month-long attempts at standard rebuilding, re-installing macOS and running
various disk utilities programs did not help.) I needed a surgical
instrument to rattle the bits floating around my computer. I started
removing impossible-to-remove files (like the deprecated “X509Anchors” Keychain
which had an old “DoD Root CA 2” cert) and making other changes but no luck.
Then I got really desperate — I asked for help. Specifically from
one of our Unix kernel folks to try things like DTrace to isolate exactly which
process was causing the problem.</div><div style="font-family: monospace;
font-size: 11px;" class=""><br class=""></div><div style="font-family:
monospace; font-size: 11px;" class="">However, like all good quiet, backroom
Unix types, once he started looking at the problem and tried “this and that”,
he found the root cause and fixed it — or at least dampened the effects to make
my Apple Mail usable again. He traced the problem to “trustd”, an
apparently new (or at least majorly re-written) daemon in Sierra.</div><div
style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;"
class="">Problem Discovered...</div><div style="font-family: monospace;
font-size: 11px;" class=""><br class=""></div><div style="font-family:
monospace; font-size: 11px;" class="">“trustd” is bad. Maybe good intent,
but bad implementation. “trustd” was found to be running somewhat rampant
— more than one instance at a time and looping on retries to retrieve
information (CRLs, CA Certs, etc) from old CAs that pointed to stale web sites.
Instead of realizing that old cert information was no longer available
and to stop trying, “trustd” continued to reach out to non-existent sites and
create unexplainable processing logjams. For example, mismatched
message/cert addresses under “DoD Root CA 2” wreaked havoc for “trustd"
but mismatched message/cert addresses under “DoD Root CA 3” did not.</div><div
style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;"
class="">Work-around...</div><div style="font-family: monospace; font-size:
11px;" class=""><br class=""></div><div style="font-family: monospace;
font-size: 11px;" class="">In the end, we cleaned up (deleted) a number of old
certs (intermediate and CA certs) whose embedded URLs were dead. The
simple act of removing expired certs suddenly changed “forever” from 30-minutes
(and infinity) down to 2-3 seconds.</div><div style="font-family: monospace;
font-size: 11px;" class=""><br class=""></div><div style="font-family:
monospace; font-size: 11px;" class="">So… The problem is still there —
mismatched addresses under “DoD Root CA 2” are still causing processing delays
to Apple Mail (and also Keychain) — but 2-3 seconds per message (and now
“instant” Keychain response) is a whole lot better than waiting between
30-minutes and “forever” to read a message.</div><div style="font-family:
monospace; font-size: 11px;" class=""><br class=""></div><div
style="font-family: monospace; font-size: 11px;" class="">Real Fix...</div><div
style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;"
class="">It would be good if “traced” where either removed or re-written to
better handle bad information. [Don’t know if Apple developers still read
this thread.]</div><div style="font-family: monospace; font-size: 11px;"
class=""><br class=""></div><div style="font-family: monospace; font-size:
11px;" class="">In the meantime, if you have this problem, removing old certs
is a good work-around (note this is a very manual process — especially since
“Keychain First Aid” was removed years ago).</div><div style="font-family:
monospace; font-size: 11px;" class=""><br class=""></div><div
style="font-family: monospace; font-size: 11px;" class="">BTW, for the geeks in
the crowd, enclosed are the notes that Ken Hornstein, our
making-macOS-useful-again person, wrote up…</div><div style="font-family:
monospace; font-size: 11px;" class=""><br class=""></div><div
style="font-family: monospace; font-size: 11px;" class="">Basil</div><div
style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;"
class=""><div
class="">-------------------------------------------------------------</div></div><div
style="font-family: monospace; font-size: 11px;" class="">Notes from Ken
Hornstein in tracking down the “trustd” issue…</div><div style="font-family:
monospace; font-size: 11px;"
class="">-------------------------------------------------------------</div><div
style="font-family: monospace; font-size: 11px;" class=""><br
class=""></div><div style="font-family: monospace; font-size: 11px;"
class="">The root problem was Apple Mail would not display messages that
were signed by certificates issued by a particular CA ("DoD Root
CA 2") AND had an email in the certificate that did not match the
sender’s email address. By "not display", it would show
the spinning “Loading..." indicator in the message pane.
Mismatched signed email messages displayed fine from other CAs, and
messages signed by certificates issued by "DoD Root CA 2" where
the certificate email address matched the sender’s email address also were
fine.<br class=""><br class="">A system call trace on Mail.app was not very
useful; it wasn't doing anything obvious, and it looked like it was
performing some IPC and waiting for a response from another program.
However, a network trace showed some very interesting information;
the machine (which was only running Mail.app, no other programs, and
had been recently rebooted) was constantly contacting a bunch of servers
all over the Internet asking for certificate information.<br class=""><br
class="">Some sleuthing lead to a daemon called "trustd" which seems
to be the gatekeeper for certificate validation. A system call trace
on the “trustd" instance running under the user's userid showed that it
was the one responsible for issuing all of the requests from various
servers for certificate information. Further inspection showed that
it was issuing the same requests over and over for the same collection of
certificates; even more unusual, these were for certificates that were
not related to the email in question at all. It seemed like a lot of
the requests for lists of intermediate certificates, and the root
certificates in question were expired so the servers were returning
"404 Not Found” for the requests.<br class=""><br class="">Matching
the request URLs for certificates in the keychain, the offending
certificates were deleted from the user's login keychain and “trustd"
was then restarted. This allowed the offending message in Mail.app
to display after a lengthy delay (maybe 30 seconds). It was
discovered that trustd was still issuing a series of certificate requests
for a different set of expired certificates, so those
offending certificates were removed and “trustd" was restarted.
After every new round of certificate deletions, the delay to display
the offending message in Mail.app was reduced, until finally the original
message that could not be displayed could successfully be viewed with
minimal delay (1-2 seconds).<br class=""><br class="">My guess is for whatever
reason “trustd" decided that it needed to download some sort of
information for these expired root certificates and there is a bug where
it does not handle a "404" error properly and causes
“trustd" to go into a loop continually requesting the
certificate information over and over and it reached a point where all of
the worker threads were busy processing these requests, which left no
workers free to respond with the other verification requests from programs
like Mail.app. Why this only affected certificates issued from "DoD
Root CA 2", I have no idea.<br class=""><br
class=""></div></div></body></html>
--Apple-Mail=_5A556A33-88BF-4148-8922-40A0E10F9F6F--
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Fed-talk mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden