It is, though. Einstein wrote an entire book explaining the theory of general relativity to people who didn't even understand the math:

http://www.amazon.com/Relativity-Special-General-Theory-Vol/...

Most ideas can be broken down into simpler parts. For instance, multiplication used to be a university level subject, but most literate people can do multiplication up to any digit (given the time, of course).

Supposed to be a good exposition, although not entirely easy; for modern editions you can buy:

http://www.amazon.com/Relativity-Special-General-Masterpiece...

http://www.amazon.com/Relativity-Special-General-Classic-Rep...