I've been messing with spice for the past week now and I can make a few observations:
Spice is not all that hard to get going using virt-manager with KVM, but all the ways I've seen it used are entirely based around KVM. So I wouldn't myself even try to set up spice seperate from KVM as it would involve an intimate knowlege of it's workings that I have no idea about, such as compiling the spice software components from source. As there aren't any compiled packages (RPM's DEB's etc) that I'm aware of that aren't designed to be used with KVM/Qemu.
My suspicion is that the spice server module is dependent on qemu/kvm.
Also, spice needs the KVM emulated "QXL" video card driver in order to function properly. It does work without using the QXL driver, but is very slow. On windows it's much much faster with it, and so far, much to my annoyance

, I haven't noticed any difference in performance in linux (Ubuntu, Fedora, & Sabayon) with the QXL driver installed... I think I'll ask around here, and if all else fails try and get a copy of Red Hat linux to test it on, but I suppose it will exhibit much the same behavior as Fedora.