Or alternately, create a small overlay window that draws the
triangle and put it in a higher layer than the menu's backing window
is. Judging from some drawing glitches I got, that's what the dock
is actually doing.
That's precisely what I did for my application, both for certain menu
bar items and a little Docklet -- here's a screenshot from Tiger:
