well, if you wish to download the source for a specific kernel version (assuming it is still in the repos) you just put the entire package name (along with version) instead of just kernel.
For example, to download the source for 3.5.0-4 for Fedora 17,
yumdownloader --source kernel-3.5.0-4.fc17.src.rpm
Or you can just go over to koji and select whichever you want, just drill down for what version. then source is in the src directory, the kernels are in either the i386 or x86_64 directory depending upon which you want.