Are you sure you're talking about window mode and not screen/display mode?
That's the only one that works for me with --disable-direct-composition
I have NVIDIA GeForce GTX 750
Actually this was originally reported to me by a player who uses an ATI adapter.
A long time ago when I tested it, it worked even in game mode (more than a year ago), but somewhere since then it got broken.
What settings can cause this?
I tried all the options available on OBS, none of them seemed to help.
Personally I do my recordings with NVIDIA built-in game recorder, works better than anything else, but it also needs --disable-direct-composition to work, and not everyone got a GTX 600+ for that.