Simply adding 'phobic' to a thing doesn't make it wrong. If Islam really is a violent expansionist religion which degrades women and excuses pedophilia, it would mean we ought to be Islamophobic and anyone who isn't is delusional.
This is up for debate, same as Scientology or Christianity or Communism or Nazism or any other religion or political ideology (noting that Islam is both a religion and a political system).
EDIT: Recommended book on the subject is Shahi Hamid's "Islamic Exceptionalism".
Get dozens of book recommendations delivered straight to your inbox every Thursday.