What Makes a Book Controversial?

I have found that most books that are deemed controversial are usually about sex, religion, sorcery and/or magic, or contains graphic language. I understand that in school libraries this is an extremely important issue because some parents do not want their children reading about certain issues or exposed to certain topics. Media specialists must be aware of the books they select, yet be careful not to censor. There should also be a policy to handle challenged books to ensure the proper steps are taken to handle the situation.