I've been noticing this recently that pavucontrol and pulseaudio process will show up on the top of the KDE system activity list for memory users. For example I've just seen pavucontrol using an RSS of 885 MB. I've also recently seen pulseaudio process in the GB area.
If I kill off the process and restart it, the RSS used goes back down to normal (which makes sense). But it can get high enough to push my machine (Fedora 17, 64bit) into the swap (which doesn't happen too often with 8GB of ram)
I guess it's this bug:
http://pulseaudio.org/ticket/901