3.11 The X11 Window System
X11 [Scheifler and Gettys, 1992] is the dominant windowing system used on UNIX systems. It uses the network for communication between applications and the I/O devices (the screen, the mouse, and so on), which allows the applications to reside on different machines. This is the source of much of the power of X11. It is also the source of great danger.
The fundamental concept of X11 is the somewhat disconcerting notion that the user's terminal is a server. This is quite the reverse of the usual pattern, in which the per-user, small, dumb machines are the clients, requesting services via the network from assorted servers. The server controls all of the interaction devices. Applications make calls to this server when they wish to talk to the user. It does not matter how these applications are invoked; the window system need not have any hand in their creation. If they know the magic tokensthe network address of the serverthey can connect.
In short, we give away control of our mouse, keyboard, and screen.
Applications that have connected to an X11 server can do all sorts of things. They can detect keypresses, dump the screen contents, generate synthetic keypresses for applications that will permit them, and so on. In other words, if an enemy has connected to your keyboard you can kiss your computer assets good-bye. It is possible for an application to grab sole control of the keyboard when it wants to do things like read a password. Few users use that feature. Even if they did, another mechanism that can't be locked out will let you poll the keyboard up/down status map.
The problem is now clear. An attacker anywhere on the Internet can probe for X11 servers. If they are unprotected, as is often the case, this connection will succeed, generally without notification to the user. Nor is the port number difficult to guess; it is almost always port 6000 plus a very small integer, usually zero.
One application, the window manager, has special properties. It uses certain unusual primitives so that it can open and close other windows, resize them, and so on. Nevertheless, it is an ordinary application in one very important sense: It, too, issues network requests to talk to the server.
A number of protection mechanisms are present in X11. Not all are particularly secure. The first level is host address-based authentication. The server retrieves the network source address of the application and compares it against a list of allowable sources; connection requests from unauthorized hosts are rejected, often without any notification to the user. Furthermore, the granularity of this scheme is tied to the level of the requesting machine, not an individual. There is no protection against unauthorized users connecting from that machine to an X11 server. IP spoofing and hijacking tools are available on the Internet.
A second mechanism uses a so-called magic cookie. Both the application and the server share a secret byte string; processes without this string cannot connect to the server. But getting the string to the server in a secure fashion is difficult. One cannot simply copy it over a possibly monitored network cable, or use NFS to retrieve it. Furthermore, a network eavesdropper could snarf the magic cookie whenever it was used.
A third X11 security mechanism uses a cryptographic challenge/response scheme. This could be quite secure; however, it suffers from the same key distribution problem as does magic cookie authentication. A Kerberos variant exists, but of course it's only useful if you run Kerberos. And there's still the issue of connection-hijacking.
The best way to use X11 these days is to confine it to local access on a workstation, or to tunnel it using ssh or IPsec. When you use ssh, it does set up a TCP socket that it forwards to X11, but the socket is bound to 127.0.0.1, with magic cookie authentication using a local, randomly generated key on top of that. That should be safe enough.
3.11.1 xdm
How does the X server (the local terminal, remember) tell remote clients to use it? In particular, how do X terminals log you in to a host? An X terminal generates an X Display Manager Control Protocol (XDMCP) message and either broadcasts it or directs it to a specific host. These queries are handled by the xdm program, which can initiate an xlogin screen or offer a menu of other hosts that may serve the X host.
Generally, Xdm itself runs as root, and has had some security problems in the past (e.g., CERT Vendor-Initiated Bulletin VB-95:08). Current versions are better, but access to the xdm service should be limited to hosts that need it. There are configuration files that tell xdm whom to serve, but they only work if you use them. Both xauth and xhost should be used to restrict access to the X server.