I actually got it to happen in the viewer window this time.
It really looks like as if no enough VRAM was left for Quartz
Composer. Try quitting all other running programs and make sure you
don't have any external screen connected to your laptop, then
relaunch QC and see if that still happens.