Re: FYI: info on our scary Mac rebooting iOS testing crash.
Re: FYI: info on our scary Mac rebooting iOS testing crash.
- Subject: Re: FYI: info on our scary Mac rebooting iOS testing crash.
- From: Alex Zavatone via Cocoa-dev <email@hidden>
- Date: Fri, 18 Oct 2019 19:16:55 -0500
Wow was this ever fun.
Woo hoo!
Let this be an exercise in the dangers of memory leaks.
Our team uses VIPER with everything being strongly linked. Also in Objective-C,
there are cases where private instance variables are used, not the wrapping
properties, not to mention strongly referenced delegates. Who needs memory
management, right? Well, guess what? When you do this, it's really easy to leak
lots of objects and this means lots of mach ports are requested from Xcode and
this means that there are lots of mach ports requested by the WindowServer and
this means that the GPU driver, (SkyLight) requests more mach ports - until the
system says, "no, you can not have any more.” That happens around 260,000+
mach ports in the WindowServer. This restarts the GPU. And when the GPU
restarts, the system either restarts the Mac or crashes the user session out to
the login screen.
After a night fixing memory bugs until midnight, I ran our 280 KIF tests and no
more than 20,000 mach ports were requested. And the full test ran all the way
through without rebooting the Mac or crashing out to the login screen.
Memory leaks matter!
Oh, and if anyone wants video of watching the Mac reboot when the WindowServer
has over 260,000 mach ports allocated, I'll happily post a link.
Cheers,
Alex Zavatone
> On Oct 17, 2019, at 12:07 PM, Alex Zavatone via Cocoa-dev
> <email@hidden> wrote:
>
> As a background, I mentioned that we occasionally have these scary crashes
> while running our iOS tests that either reboot your Mac or crash the user
> process out to the login screen.
>
> Yesterday, I noticed that I could not run our tests at all without crashing
> around a certain area and on 3 different Macs, all with 24 GB or more of RAM.
>
> My speculation that it, might be related to mach ports was entirely correct
> as after 8 crashes, some debug logs finally appeared - but not all the time.
>
> Our iOS UI tests are run by KIF which uses XCTest as its foundation. When
> running these 280+ UI automation scripts in the Simulator, the amount of
> ports used by Xcode increased rapidly as the tests ran for 30 mins and the
> WindowServer process appeared to have about 20,000 more Mach ports allocated
> than Xcode over that time.
>
> Right before the crash happened, the WindowServer had allocated over 256,000
> ports and Xcode was just behind by a few thousand. Then, all operation
> pauses for a few seconds, the displays go black, and a few seconds later, the
> user login screen appears as the previous user session crashed out.
>
> What happened is that Xcode asked for more ports and asked the WindowServer
> to allocate more ports and the WindowServer asked the GPU driver for more
> ports and thread com.apple.SkyLight.mtl_submit to crash with either a SIGSEV
> or SIGILL. Either the WindowServer or the Skylight framework for MetalDevice
> and caused a GPU restart. The OS then crashed the user session out to the
> login screen.
>
> Now, the question. Why the hell is XCTest causing Xcode to request so many
> Mach ports and never release them until the app is quit or the system crashes?
>
> I tested this last night outside of our app with XCTest and KIF simply going
> back and forth between two screens in an iOS app several thousand times In
> the Simulator. The mach ports of both Xcode and the WindowServer go up and
> are not released until the app quits, but not nearly as fast as in our tests.
>
> Is this a mach port leak in XCTest?
>
> Does anyone know the details on the guts of this?
>
> Thanks a lot and I hope this helps someone else.
>
> Alex Zavatone
>
> Sent from my iThing.
> _______________________________________________
>
> Cocoa-dev mailing list (email@hidden)
>
> Please do not post admin requests or moderator comments to the list.
> Contact the moderators at cocoa-dev-admins(at)lists.apple.com
>
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden