Actually, one spoiler does affect performance. Put the big wing on the Phoenix and you can fly. Or it can get caught on some low bridges.
I've been trying to get a spoiler-Infernus for 3-4 days (3 times, none, wasted money) just because it looks nicer with it.
You probably heard this crazy talk in a lobby, right? People take GTA too seriously. You'll see a bunch of threads here on how the racing sucks or how the shooting isn't right, etc. People don't realize GTA isn't a racing game, it isn't a 3rd person shooter. It's an open-world that manages to do all of these things to some extent. It's not f*cking Forza but at least in GTA if you get spun out you can catch up and shoot them in the head.
So no, spoilers don't affect anything. A good test would be a light car that tends to jump around a lot. Put a spoiler on it, try it for yourself.