Re: FYI: info on our scary Mac rebooting iOS testing crash.
Re: FYI: info on our scary Mac rebooting iOS testing crash.
- Subject: Re: FYI: info on our scary Mac rebooting iOS testing crash.
- From: 조성빈 via Cocoa-dev <email@hidden>
- Date: Sun, 20 Oct 2019 17:00:08 +0900
> 2019. 10. 19. 오전 9:17, Alex Zavatone via Cocoa-dev
> <email@hidden> 작성:
>
> Wow was this ever fun.
>
> Woo hoo!
>
> Let this be an exercise in the dangers of memory leaks.
>
> Our team uses VIPER with everything being strongly linked. Also in
> Objective-C, there are cases where private instance variables are used, not
> the wrapping properties, not to mention strongly referenced delegates. Who
> needs memory management, right? Well, guess what? When you do this, it's
> really easy to leak lots of objects and this means lots of mach ports are
> requested from Xcode and this means that there are lots of mach ports
> requested by the WindowServer and this means that the GPU driver, (SkyLight)
> requests more mach ports - until the system says, "no, you can not have any
> more.” That happens around 260,000+ mach ports in the WindowServer. This
> restarts the GPU. And when the GPU restarts, the system either restarts the
> Mac or crashes the user session out to the login screen.
>
> After a night fixing memory bugs until midnight, I ran our 280 KIF tests and
> no more than 20,000 mach ports were requested. And the full test ran all the
> way through without rebooting the Mac or crashing out to the login screen.
>
> Memory leaks matter!
>
> Oh, and if anyone wants video of watching the Mac reboot when the
> WindowServer has over 260,000 mach ports allocated, I'll happily post a link.
Just for curiosity: I would like the link :-)
> Cheers,
>
> Alex Zavatone
>
>> On Oct 17, 2019, at 12:07 PM, Alex Zavatone via Cocoa-dev
>> <email@hidden> wrote:
>>
>> As a background, I mentioned that we occasionally have these scary crashes
>> while running our iOS tests that either reboot your Mac or crash the user
>> process out to the login screen.
>>
>> Yesterday, I noticed that I could not run our tests at all without crashing
>> around a certain area and on 3 different Macs, all with 24 GB or more of RAM.
>>
>> My speculation that it, might be related to mach ports was entirely correct
>> as after 8 crashes, some debug logs finally appeared - but not all the time.
>>
>> Our iOS UI tests are run by KIF which uses XCTest as its foundation. When
>> running these 280+ UI automation scripts in the Simulator, the amount of
>> ports used by Xcode increased rapidly as the tests ran for 30 mins and the
>> WindowServer process appeared to have about 20,000 more Mach ports allocated
>> than Xcode over that time.
>>
>> Right before the crash happened, the WindowServer had allocated over 256,000
>> ports and Xcode was just behind by a few thousand. Then, all operation
>> pauses for a few seconds, the displays go black, and a few seconds later,
>> the user login screen appears as the previous user session crashed out.
>>
>> What happened is that Xcode asked for more ports and asked the WindowServer
>> to allocate more ports and the WindowServer asked the GPU driver for more
>> ports and thread com.apple.SkyLight.mtl_submit to crash with either a SIGSEV
>> or SIGILL. Either the WindowServer or the Skylight framework for
>> MetalDevice and caused a GPU restart. The OS then crashed the user session
>> out to the login screen.
>>
>> Now, the question. Why the hell is XCTest causing Xcode to request so many
>> Mach ports and never release them until the app is quit or the system
>> crashes?
>>
>> I tested this last night outside of our app with XCTest and KIF simply going
>> back and forth between two screens in an iOS app several thousand times In
>> the Simulator. The mach ports of both Xcode and the WindowServer go up and
>> are not released until the app quits, but not nearly as fast as in our
>> tests.
>> Is this a mach port leak in XCTest?
>>
>> Does anyone know the details on the guts of this?
>>
>> Thanks a lot and I hope this helps someone else.
>>
>> Alex Zavatone
>>
>> Sent from my iThing.
>> _______________________________________________
>>
>> Cocoa-dev mailing list (email@hidden)
>>
>> Please do not post admin requests or moderator comments to the list.
>> Contact the moderators at cocoa-dev-admins(at)lists.apple.com
>>
>> Help/Unsubscribe/Update your Subscription:
>>
>> This email sent to email@hidden
>
> _______________________________________________
>
> Cocoa-dev mailing list (email@hidden)
>
> Please do not post admin requests or moderator comments to the list.
> Contact the moderators at cocoa-dev-admins(at)lists.apple.com
>
> Help/Unsubscribe/Update your Subscription:
>
> This email sent to email@hidden
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden