This tutorial will show you how to remove KDE, Gnome and return to pure xfce

http://www.psychocats.net/ubuntu/purexfce