It's very hard and dense to read, but I don't think there's a better book out there about a networking stack implementation.
It might be too much implementation-focused for your taste though (it was for mine), in which case one of the other books more focused on protocols (already suggested in other comments) might be a better choice.
What really sets this book apart, in my opinion, is just how readable it is. I haven't run across anything with quite the same combination of style and depth for Linux.
You could read through something like The Linux Programming Interface [1] or Understanding the Linux Kernel [2] and with effort extract much of the same information, but without the pace/structure/commentary that I think this book does so well.
Thing is, conceptually much of what you'll read in this book will be applicable to Linux just fine and having read it will make it far easier to grok how/where the two differ.
1: http://www.amazon.com/dp/1593272200 2: http://www.amazon.com/dp/0596005652
I encountered many of these while reading through Understanding The Linux Kernel [0] and The Linux Programming Interface [1].
Both are great books which are primarily about the "how" of the kernel, but cover a lot of the "why" of the design and algorithms as well.
https://blogs.oracle.com/ksplice/entry/hello_from_a_libc_fre...
http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm...
http://timelessname.com/elfbin/
A wee bit heavy, but it's comprehensive. It deals with what happens when you run code, how the architecture of the computer works (by and large) including at the logic level:
http://www.amazon.co.uk/Computer-Systems-Programmers-Randal-...
If you want to go lower (and higher).. look at Understanding the Linux kernel for a good understanding of how an OS is put together, with specific examples i.e. Linux.
Code, by Petzold, deals with logic and computers from the ground up. It starts with relays and builds them up into gates and usable arithmetic blocks.
http://www.amazon.co.uk/Code-Language-Computer-Hardware-Soft...
http://www.amazon.co.uk/Understanding-Linux-Kernel-Daniel-Bo...
The physics is fairly simple, at least from a CRT or LED display perspective. Gets more tricky dealing with interconnecting microprocessors because a good chunk is vendor specific.
I think this kind of project is well suited to a guide on how to build a computer from the ground up, starting with logic gates, writing a real time OS and developing a scripting language that will run and compile on it. Then you can skip a lot of largely extraneous stuff and have a solid understanding of how the hardware works.
A good "gentle introduction" book is the Love book (440 pages): http://www.amazon.com/Linux-Kernel-Development-Robert-Love/d...
Learn C as necessary.
The FreeBSD kernel book is worth a look too: http://www.amazon.com/Design-Implementation-FreeBSD-Operatin...
https://www.amazon.com/Linux-Core-Kernel-Commentary-Knowledg...
Understanding the Linux Kernel, Third Edition 3rd Edition
https://www.amazon.com/Understanding-Linux-Kernel-Third-Dani...