I've already managed to find a fix. I'm posting here so others afflicted by the same problem may be able to find a potential solution.

I recently upgraded to 11.4 and also the Nvidia 270.41.06 driver. I found that on logging out X would fail to restart with a seg fault logged in /var/log/kdm.log concerning doListFontsWithInfo (stack trace attached). I also found that I could not manually restart kdm by changing run-levels. I also found that if I did a startx as root, the session would start, but as soon as I focused on a KDE konsole it would error again. A reboot fixed the problem until the next logout.

After some googling I found that disabling the use of the xfs font server by commenting out the entire #Section "Files" in /etc/xorg.conf fixed this problem. Reference here:

X server 1.9.3 crash on Fedora 14, GeForce 8600 GTS - nV News Forums

So if you encounter a similar problem, see if disabling the local X servers use of xfs fixes it.

Backtrace:
0: /usr/bin/Xorg (xorg_backtrace+0x28) [0x463678]
1: /usr/bin/Xorg (0x400000+0x6ae89) [0x46ae89]
2: /lib64/libc.so.6 (0x7f4278ecc000+0x32b30) [0x7f4278efeb30]
3: /usr/bin/Xorg (doListFontsWithInfo+0x1c1) [0x4301f1]
4: /usr/bin/Xorg (ProcessWorkQueue+0x21) [0x44d5d1]
5: /usr/bin/Xorg (WaitForSomething+0x6a) [0x463eca]
6: /usr/bin/Xorg (0x400000+0x2f432) [0x42f432]
7: /usr/bin/Xorg (0x400000+0x25ace) [0x425ace]
8: /lib64/libc.so.6 (__libc_start_main+0xfd) [0x7f4278eeabfd]
9: /usr/bin/Xorg (0x400000+0x25679) [0x425679]
Segmentation fault at address 0x6abab380

Fatal server error:
Caught signal 11 (Segmentation fault). Server aborting