Robert Charles Wilson's Spin - http://www.amazon.com/Spin-Robert-Charles-Wilson/dp/07653482...
This is IMO the very best kind of sci-fi: a plausible, scientifically grounded story about interesting people experiencing some fascinating shit.
Wilson is a great writer, too; I hadn't heard of him previously, but have since read a bunch of his works.
(As an aside, there has never been a year in my life where the best book of the year for me was a nonfiction title. Am I weird?)
http://www.amazon.com/Spin-ebook/dp/B0016IXMWI/ref=kinw_dp_k...
Well that and Spin [2], which probably addresses your point - maybe they're out there and they're already messing with us?
Ultimately, whenever I think about Fermi's Paradox, I am reminded of xkcd [3]. Maybe we just can't tell they're right in front of us.
[1] https://www.amazon.com/Are-Legion-Bob-Bobiverse-Book-ebook/d... [2] https://www.amazon.com/Spin-Robert-Charles-Wilson/dp/0765348... [3] https://xkcd.com/638/