Get the two Souders (creator of YSlow, now works on web performance at Google) books, which are amazing. I'm not affiliated with Amazon.com, O'Reilly or Yahoo!; just like Patrick says, these guys know their shit:

High Performance Web Sites (O'Reilly, Sept. 2007): http://www.amazon.com/High-Performance-Web-Sites-Essential/d...

Even Faster Web Sites (O'Reilly, June 2009): http://www.amazon.com/Even-Faster-Web-Sites-Performance/dp/0...

Half of King's 'Website Optimization: Speed, Search Engine & Conversion Rate Secrets' (O'Reilly, July 2008) is devoted to performance, so it's also worth a look: http://www.amazon.com/Website-Optimization-Search-Conversion...