As I seem to recall I had an Xvfb-based custom X server that had input break quite horribly between the 1.3 and 1.4 X server branches (although it might have been 1.2). I was using the X server as a backing server, so that I could have another process using shared memory copy the window contents, without using the Composite extension. The entire root window (which was very large and offscreen) was mmap shared memory. It was an experiment, and it worked, but it was also not perfect, and the cost of all of the buffer copying and alpha compositing only allowed about a maximum of about 80-90 FPS, and when playing videos it was often much less. I also had a weird sort of window manager that would organize the windows on the Xvfb server, so that the large root Xvfb root window would allow more applications on screen. In other words it was a hack to enable alpha compositing and other effects. I had rotating windows and other silly stuff to emulate another window system I worked on. The backing server contents were copied, translated, scaled, etc. to the primary X server via the shared memory. I also used Xvfb with another project for X11 compatibility with a non-X11 window system. I was trying to integrate that with a window system I had written, so that I could have compatibility. I never did track down what the input code changes were that broke the input for these 2 projects, and I lost interest. Anyway, you're not alone. Perhaps the 1.6 branch will or has fixed Xvfb. The code in Xvfb is fairly simple, so hopefully the underlying problem won't stay broken for long.