Slightly wrong. Take Nasuverse for example. Someone only saw the Fate/Stay Night anime and came to liking the series.
They're fans but far from being the hardcore fans. People who read the original visual novel of Fate/Stay Night, played Tsukihime, read Notes, read the Fate/Zero Light Novel series, own several figures, fanbooks, artbooks and whatever related to the verse and being constantly up to date with the entire Nasuverse.
So yeah, if someone new to Fate/Stay Night would claim he's bigger Type-MOON/Nasuverse fan than me, I would just laugh.
But they aren't illegitimate fans, either. Let's say they're more "softcore" about it.
Now, here comes the next question:
Is it wrong to watch an anime series because it's so bad its just simply hilarious?