I want to mimic iTunes's behavior, which does this. I want the Dock
menu to list the window only if it's open, but since the Window
menu will always have an item to show the main window, I don't need
it listed again.
You could exclude the window from both locations with -
setExcludedFromWindowsMenu:, then manage the window list in the dock
menu yourself. This should be pretty simple if you just have a couple
of non-document windows like iTunes.