I learned more about OS from this book than any academic material used at the university and gave me a good foundational handle on *NIX's.
https://www.amazon.com/Design-UNIX-Operating-System/dp/01320...
Which is probably less relevant today in terms of directly understanding the implementation. But an interesting and enlightening read. Things were much simpler and fundamental back in the 1980s. It's easier to understand that way. Then layer on top.
[0] http://www.amazon.com/The-Design-UNIX-Operating-System/dp/01...
1. http://www.quora.com/How-do-I-write-an-operating-system
2. http://www.quora.com/If-you-were-to-write-a-new-operating-sy...
3. http://www.cs.bham.ac.uk/~exr/lectures/opsys/10_11/lectures/...
4. http://www.amazon.com/The-Design-UNIX-Operating-System/dp/01...
5. http://www.amazon.com/The-Elements-Computing-Systems-Princip...
In no particular order:
1- http://www.amazon.com/C-Programming-Language-2nd-Edition/dp/...
2- http://www.amazon.com/The-Answer-Book-Solutions-Programming/...
3- http://www.amazon.com/The-Standard-Library-P-J-Plauger/dp/01...
4- http://www.amazon.com/C-Traps-Pitfalls-Andrew-Koenig/dp/0201...
5- http://www.amazon.com/Expert-Programming-Peter-van-Linden/dp...
6- http://www.amazon.com/Data-Structures-In-Noel-Kalicharan/dp/...
7- http://www.amazon.com/Data-Structures-Using-Aaron-Tenenbaum/...
8- http://www.amazon.com/Mastering-Algorithms-C-Kyle-Loudon/dp/...
9- http://www.amazon.com/Code-Complete-Practical-Handbook-Const...
10- http://www.amazon.com/Design-Patterns-Elements-Reusable-Obje...
11- http://www.amazon.com/The-Mythical-Man-Month-Engineering-Ann...
12- http://www.amazon.com/The-Programming-Language-4th-Edition/d...
13- http://www.amazon.com/The-Standard-Library-Tutorial-Referenc...
14- http://www.amazon.com/API-Design-C-Martin-Reddy/dp/012385003...
15- http://www.amazon.com/The-Linux-Programming-Interface-Handbo...
16- http://www.amazon.com/Computer-Systems-Programmers-Perspecti...
17- http://www.amazon.com/System-Programming-Unix-Adam-Hoover/dp...
18- http://www.amazon.com/Memory-Programming-Concept-Frantisek-F...
19- http://www.amazon.com/Memory-Management-Implementations-Prog...
20- http://www.amazon.com/UNIX-Filesystems-Evolution-Design-Impl...
21- http://www.amazon.com/PCI-System-Architecture-4th-Edition/dp...
22- http://www.amazon.com/Universal-Serial-System-Architecture-E...
23- http://www.amazon.com/Introduction-PCI-Express-Hardware-Deve...
24- http://www.amazon.com/Serial-Storage-Architecture-Applicatio...
25- http://www.amazon.com/SATA-Storage-Technology-Serial-ATA/dp/...
26- http://www.amazon.com/Beyond-BIOS-Developing-Extensible-Inte...
27- http://www.amazon.com/Professional-Assembly-Language-Program...
28- http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d...
29- http://www.amazon.com/Version-Control-Git-collaborative-deve...
30- http://www.amazon.com/Embedded-Software-Primer-David-Simon/d...
31- http://www.amazon.com/Programming-Embedded-Systems-C/dp/1565...
32- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...
33- http://www.amazon.com/Operating-System-Concepts-Abraham-Silb...
34- http://www.amazon.com/Performance-Preemptive-Multitasking-Mi...
35- http://www.amazon.com/Design-Operating-System-Prentice-Hall-...
36- http://www.amazon.com/Unix-Network-Programming-Sockets-Netwo...
37- http://www.amazon.com/TCP-Illustrated-Volume-Addison-Wesley-...
38- http://www.amazon.com/TCP-IP-Illustrated-Vol-Implementation/...
39- http://www.amazon.com/TCP-Illustrated-Vol-Transactions-Proto...
40- http://www.amazon.com/User-Interface-Design-Programmers-Spol...
41- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...
42- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...
43- http://www.amazon.com/Programming-POSIX-Threads-David-Butenh...
44- http://www.intel.com/p/en_US/embedded/hwsw/software/hd-gma#d...
45- http://www.intel.com/content/www/us/en/processors/architectu...
46- http://www.intel.com/p/en_US/embedded/hwsw/hardware/core-b75...
47- http://www.hdmi.org/index.aspx
48- http://en.wikipedia.org/wiki/Digital_Visual_Interface
49- http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-...
50- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...
51- http://www.amazon.com/Python-Programming-Introduction-Comput...
52- http://www.amazon.com/Practical-System-Design-Dominic-Giampa...
53- http://www.amazon.com/File-Systems-Structures-Thomas-Harbron...
54- ...well, I'll stop here.
Of course, the equivalent knowledge can be obtained by trial-and-error, which would take longer and might result in costly errors and imperfect design. The greater danger here is that a sole developer, without the feedback and interaction of even a small group of capable and experienced programmers could simply burn a lot of time repeating the mistakes made by those who have already trenched that territory.
If the goal is to write a small RTOS on a small but nicely-featured microcontroller, then the C books and the uC/OS book might be a good shove in the right direction. Things start getting complicated if you need to write such things as a full USB stack, PCIe subsystem, graphics drivers, etc.
[1] http://www.amazon.com/Computer-Systems-Programmers-Randal-Br...
[2] http://www.amazon.com/Design-Operating-System-Prentice-Softw...
To that end there is The Design of The Unix Operating System, Bach [1]. Or A Fast File System for Unix [2]. Or The Design and Implementation of a Log-Structured File System [3]. It seems to me that files solve for the problem of how to deliver an economic memory management system, and they do it by paging cheap disk space so that it's an effective proxy for fast-but-expensive memory. Also files, with their file pointers, can certainly be used as data structures, but this 'understanding' of what each section means is clearly variable, so is best left to specific applications and it's not good to build into the system as a default. Different applications will want different structures for optimal performance...
[1] https://www.amazon.com/Design-UNIX-Operating-System/dp/01320...
[2] https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf
[3] https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf