September 2014 - Pro-apps-dev

Re: memory leak when rendering with Compressor from FCPX
by Paul Schneider 03 Sep '14

03 Sep '14

> I couldn't tell if you thought this was a bug in our code, or in Compressor. Should I file a radar? I was thinking that you could check your code and we could check our code :) Our built-in effects are handled mostly the same, but there are enough differences that there could be a problem on our side that would only manifest for third-party effects. A radar would be great. I don’t see an open radar about this issue currently. On Sep 3, 2014, at 5:27 AM, Paul Miller <paul(a)fxtech.com> wrote: > On 9/2/2014 4:38 PM, Paul Schneider wrote: >> >> Hi Paul and Gabe, >> >> Paul, your symptoms sound like the problem might be texture memory >> growing unbounded. I believe the GL driver will allocate memory for >> textures outside of the app’s address space. Are you viewing “My >> processes” or “All processes” in activity monitor? > > Hi Paul! > > Yes - that sounds like it may be the case. Both of the filter sets I tried are using the OpenGL path. > >> You might also try Open GL Profiler and/or vmmap to see if you can get a >> better view of texture and memory usage. > > Okay. > >> Gabe’s description of memory being released after every frame in FCP, >> but not until the end of the render in Compressor, is interesting. My >> uninformed guess is that as a headless process, ProMSRenderingTool >> either doesn’t have a run loop, or doesn’t service the run loop between >> frames when rendering. It’s possible that either your code or our code >> is scheduling some cleanup for idle time using dispatch_async or >> performSelector:afterDelay: or some other mechanism that relies on the >> run loop to be serviced. > > Interesting. Since it appears that FCP-native effects don't seem to be affected by this, are they implemented using the exact same FXPlug mechanisms as ours would be, or do they have a different path? > > I couldn't tell if you thought this was a bug in our code, or in Compressor. Should I file a radar? > > Cheers! > > -Paul > >> >> - Paul >> >> >> On Sep 2, 2014, at 12:36 PM, Paul Miller <paul(a)fxtech.com >> <mailto:paul@fxtech.com>> wrote: >> >>> On 9/2/2014 11:07 AM, Paul Miller wrote: >>>> On 9/2/2014 10:43 AM, Paul Miller wrote: >>>>> I watched a render today in Activity Monitor. While it was rendering, >>>>> Memory usage of ProMSRendererTool hit about 2GB and then was stable, >>>>> hovering around 1.81GB. compressord hung in there at about 128MB for the >>>>> whole time. >>>>> >>>>> After about 20 min I got the "Your system has run out of application >>>>> memory" dialog box. >>>> >>>> BTW - I tried again with a simple project with one long clip and a >>>> third-party filter that isn't one of mine, and got the same results. >>>> Despite ProMSRendererTool only showing between 1.5 and 1.9GB used in the >>>> Memory column, I can see Virtual Memory and Swap Used steadily rising, >>>> about 125MB/sec. >>>> >>>> I think there is a problem with Compressor. >>> >>> Of course, this doesn't happen when using a built-in effect, such as >>> FCP's Night Vision. ProMSRenderingTool hangs out at 440MB. Memory Used >>> creeps very very slowly. >>> >>> Perhaps the problem only occurs in 3rd party fxplugs? >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com >>> <mailto:Pro-apps-dev@lists.apple.com>) >>> Help/Unsubscribe/Update your Subscription: >>> https://lists.apple.com/mailman/options/pro-apps-dev/pschneider%40apple.com >>> >>> This email sent topschneider(a)apple.com <mailto:pschneider@apple.com> >> > > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) > Help/Unsubscribe/Update your Subscription: > https://lists.apple.com/mailman/options/pro-apps-dev/pschneider%40apple.com > > This email sent to pschneider(a)apple.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: memory leak when rendering with Compressor from FCPX
by Paul Miller 03 Sep '14

03 Sep '14

On 9/2/2014 4:38 PM, Paul Schneider wrote: Hi Paul and Gabe, Paul, your symptoms sound like the problem might be texture memory growing unbounded. I believe the GL driver will allocate memory for textures outside of the app’s address space. Are you viewing “My processes” or “All processes” in activity monitor? Hi Paul! You might also try Open GL Profiler and/or vmmap to see if you can get a better view of texture and memory usage. Okay. Gabe’s description of memory being released after every frame in FCP, but not until the end of the render in Compressor, is interesting. My uninformed guess is that as a headless process, ProMSRenderingTool either doesn’t have a run loop, or doesn’t service the run loop between frames when rendering. It’s possible that either your code or our code is scheduling some cleanup for idle time using dispatch_async or performSelector:afterDelay: or some other mechanism that relies on the run loop to be serviced. Cheers! -Paul - Paul On Sep 2, 2014, at 12:36 PM, Paul Miller <paul(a)fxtech.com <mailto:paul@fxtech.com>> wrote: On 9/2/2014 11:07 AM, Paul Miller wrote: On 9/2/2014 10:43 AM, Paul Miller wrote: I watched a render today in Activity Monitor. While it was rendering, Memory usage of ProMSRendererTool hit about 2GB and then was stable, hovering around 1.81GB. compressord hung in there at about 128MB for the whole time. After about 20 min I got the "Your system has run out of application memory" dialog box. BTW - I tried again with a simple project with one long clip and a third-party filter that isn't one of mine, and got the same results. Despite ProMSRendererTool only showing between 1.5 and 1.9GB used in the Memory column, I can see Virtual Memory and Swap Used steadily rising, about 125MB/sec. I think there is a problem with Compressor. Of course, this doesn't happen when using a built-in effect, such as FCP's Night Vision. ProMSRenderingTool hangs out at 440MB. Memory Used creeps very very slowly. Perhaps the problem only occurs in 3rd party fxplugs? _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com <mailto:Pro-apps-dev@lists.apple.com>) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/pschneider%40apple.com This email sent topschneider(a)apple.com <mailto:pschneider@apple.com> _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com Yes - that sounds like it may be the case. Both of the filter sets I tried are using the OpenGL path. Interesting. Since it appears that FCP-native effects don't seem to be affected by this, are they implemented using the exact same FXPlug mechanisms as ours would be, or do they have a different path? I couldn't tell if you thought this was a bug in our code, or in Compressor. Should I file a radar?

1 0

Re: memory leak when rendering with Compressor from FCPX
by Paul Schneider 02 Sep '14

02 Sep '14

_______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com

1 0

Re: memory leak when rendering with Compressor from FCPX
by Paul Miller 02 Sep '14

02 Sep '14

On 9/2/2014 11:07 AM, Paul Miller wrote: On 9/2/2014 10:43 AM, Paul Miller wrote: I watched a render today in Activity Monitor. While it was rendering, Memory usage of ProMSRendererTool hit about 2GB and then was stable, hovering around 1.81GB. compressord hung in there at about 128MB for the whole time. After about 20 min I got the "Your system has run out of application memory" dialog box. BTW - I tried again with a simple project with one long clip and a third-party filter that isn't one of mine, and got the same results. Despite ProMSRendererTool only showing between 1.5 and 1.9GB used in the Memory column, I can see Virtual Memory and Swap Used steadily rising, about 125MB/sec. I think there is a problem with Compressor. Perhaps the problem only occurs in 3rd party fxplugs? _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com Of course, this doesn't happen when using a built-in effect, such as FCP's Night Vision. ProMSRenderingTool hangs out at 440MB. Memory Used creeps very very slowly.

1 0

Re: memory leak when rendering with Compressor from FCPX
by Paul Miller 02 Sep '14

02 Sep '14

On 9/2/2014 10:43 AM, Paul Miller wrote: I watched a render today in Activity Monitor. While it was rendering, Memory usage of ProMSRendererTool hit about 2GB and then was stable, hovering around 1.81GB. compressord hung in there at about 128MB for the whole time. After about 20 min I got the "Your system has run out of application memory" dialog box. I think there is a problem with Compressor. _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com BTW - I tried again with a simple project with one long clip and a third-party filter that isn't one of mine, and got the same results. Despite ProMSRendererTool only showing between 1.5 and 1.9GB used in the Memory column, I can see Virtual Memory and Swap Used steadily rising, about 125MB/sec.

1 0

Re: memory leak when rendering with Compressor from FCPX
by Paul Miller 02 Sep '14

02 Sep '14

On 9/1/2014 3:23 AM, Gabriele de Simone wrote: Paul, Darrin, Some of our users are running into the same issue as well. The problem appears to be that ProMSRenderTool (invoked by Compressor) is allocating memory as part of each frame it renders, but the memory is not released until the entire operation is complete. This is *not* technically a memory leak because a reference to this memory is still being kept by the code, but all system memory is gradually exhausted. The system starts thrashing soon and eventually the app is killed. The issue doesn’t happen when you export from Final Cut Pro because memory footprint is not allowed to grow unchecked. Stuff from previous frames is released correctly. I couldn't connect to ProMSRendererTool from Instruments. Yet no process in the process list was using more than 2GB. I can't figure out what exactly is consuming all the memory. On Aug 29, 2014, at 8:55 PM, Darrin Cardani <dcardani(a)apple.com> wrote: On Aug 29, 2014, at 11:40 AM, Paul Miller <paul(a)fxtech.com> wrote: On 8/29/2014 11:28 AM, Darrin Cardani wrote: Paul, Can you give us any more information? How are you sending the project to Compressor? How complex is the sequence? Does it happen if your plug-in isn’t applied? Obviously Compressor is a different app and has different memory characteristics, but beyond that, it’s hard to do more than speculate. If you want to send us a project and a plug-in, we could try it out and see what we can find. You can probably run Compressor in the Leaks tool with Instruments and see if it gives you a common stack trace for where the leaked memory is coming from. If you do that, feel free to send me the results and I’ll take a look. Darrin - I was going to try that, though I don't know how to use Compressor yet. When I render through Compressor, does Compressor actually do the rendering? ie. does it have the Motion effects engine built into it? Is there a way to send a project to Compressor, then quit Compressor and re-launch it from Instruments, and have it pick up the rendering? Or can I "attach" Instruments to it like I can with the debugger? I don’t remember off the top of my head if Compressor renders things directly or has a daemon which does that. I will look into it. You can definitely attach instruments to a running process, though. There’s a popup near the top of the window next to the start/stop buttons that lists current processes and allows you to attach to one. Darrin _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/gds%40noiseindustries.… This email sent to gds(a)noiseindustries.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com I watched a render today in Activity Monitor. While it was rendering, Memory usage of ProMSRendererTool hit about 2GB and then was stable, hovering around 1.81GB. compressord hung in there at about 128MB for the whole time. After about 20 min I got the "Your system has run out of application memory" dialog box. Back in Activity Monitor, Memory Used was 7.99GB (Physical Memory is 8.00GB), Virtual Memory 71.94GB and Swap 22.15GB.

1 0

Re: memory leak when rendering with Compressor from FCPX
by Paul Miller 01 Sep '14

01 Sep '14

On 9/1/2014 3:23 AM, Gabriele de Simone wrote: Paul, Darrin, Some of our users are running into the same issue as well. The problem appears to be that ProMSRenderTool (invoked by Compressor) is allocating memory as part of each frame it renders, but the memory is not released until the entire operation is complete. This is *not* technically a memory leak because a reference to this memory is still being kept by the code, but all system memory is gradually exhausted. The system starts thrashing soon and eventually the app is killed. The issue doesn’t happen when you export from Final Cut Pro because memory footprint is not allowed to grow unchecked. Stuff from previous frames is released correctly. On Aug 29, 2014, at 8:55 PM, Darrin Cardani <dcardani(a)apple.com> wrote: On Aug 29, 2014, at 11:40 AM, Paul Miller <paul(a)fxtech.com> wrote: On 8/29/2014 11:28 AM, Darrin Cardani wrote: Paul, Can you give us any more information? How are you sending the project to Compressor? How complex is the sequence? Does it happen if your plug-in isn’t applied? Obviously Compressor is a different app and has different memory characteristics, but beyond that, it’s hard to do more than speculate. If you want to send us a project and a plug-in, we could try it out and see what we can find. You can probably run Compressor in the Leaks tool with Instruments and see if it gives you a common stack trace for where the leaked memory is coming from. If you do that, feel free to send me the results and I’ll take a look. Darrin - I was going to try that, though I don't know how to use Compressor yet. When I render through Compressor, does Compressor actually do the rendering? ie. does it have the Motion effects engine built into it? Is there a way to send a project to Compressor, then quit Compressor and re-launch it from Instruments, and have it pick up the rendering? Or can I "attach" Instruments to it like I can with the debugger? I don’t remember off the top of my head if Compressor renders things directly or has a daemon which does that. I will look into it. You can definitely attach instruments to a running process, though. There’s a popup near the top of the window next to the start/stop buttons that lists current processes and allows you to attach to one. Darrin _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/gds%40noiseindustries.… This email sent to gds(a)noiseindustries.com _______________________________________________ Do not post admin requests to the list. They will be ignored. Pro-apps-dev mailing list (Pro-apps-dev(a)lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/pro-apps-dev/site_archiver%40lists.… This email sent to site_archiver(a)lists.apple.com Very interesting Gabriele! Our customer says that the problem doesn't happen when rendering with FCPX-native filters, though I don't know which ones they tried. Can anyone else confirm>

1 0