[1]: http://www.amazon.com/The-Math-Gene-Mathematical-Thinking/dp...
You think of language as a basic skill because you have taken it for granted your whole life, but the same kinds of mental abilities used for language are what is needed for mathematics. There are heaps of scientific evidence supporting this. [1]
Again, you are putting mathematics on a pillar because it's foreign to you, not because of any inherent difficulty in the subject itself. You're "acknowledging" a myth as fact.
[1] http://www.amazon.com/The-Math-Gene-Mathematical-Thinking/dp...