I've got Gentoo on one partition to play around with and the compile times are pretty quick. Never been bothered too much by any long waiting times. Any delay is probably more to do with my ancient 20Gb HD that I've been nursing for what seems like ages.
You certainly can install 32-bit packages such as Firefox. I've never done it though as I'm in an "ALL OR NOTHING" kind of mood with 64-bit. I either do it 100%, or I stick with 32-bit throughout.
Must admit that if I require a reinstall for any reason in the near future, it will be to FC4 32-bit at the moment. There are just too many things that I miss. The repos also seem to be WAY behind in supporting a raft of things (perhaps they just won't work on 64-bit - like WINE). I'd like to mess around with WINE in the next few months, so 32-bit FC4 appears to be winging it's way closer to my machine.
If you can live without these frills (or wait for the OpenSource version of them) then the system is very good. But it's the frills that make a system fun to use I think. The core needs to be rock solid, but you need to have some frills to give you that special smile.