Will there be any side effects if you remove the X Windowing System? Well, yes, you might say so. You won't have a GUI any more, just a CLI. And as far as removing Gnome, I suspect that this would happen when you removed X, as Gnome has X as a dependency. What I don't understand is why, by the Sacred Chair of Steve Ballmer, you wuld need to uninstall X and Gnome to install Asterisk IP Telephony.
Registered Linux user #470359 and permanently recovered BOFH.
Any advice in this post is worth exactly what you paid for it.