Does Quartz Composer support any 3-dimensional texture
formats (equivalent to Direct3D .dds files)?
Doesn't look like it. All the built-in image support is based on 2d
images (width and height only). There's nothing stopping you from
writing your own plugin that uses 3d textures while rendering, but
passing around such data may be problematic.
