I remember studying this book in my undergrad which was pretty good and covered how programs look in assembly (calling conventions, call frames, data representation, assembly ...) along with whatever hardware architecture knowledge you require.

Also, to get better understanding of x86 ISA I followed the old i386 manual[2]. Its old but much more smaller. Perhaps once you feel comfortable, you can move to newer manuals. I never read newer ones (because I never actually needed assembly for any project, was just studying for fun).

[1] http://www.amazon.com/Computer-Systems-Programmers-Perspecti...

[2] http://css.csail.mit.edu/6.858/2015/readings/i386.pdf

I remember studying `Computer Systems: A Programmers Perspective'[1] in my undergrad which was pretty good and covered how programs look in assembly (calling conventions, call frames, data representation, assembly ...).

Also, to get better understanding of x86 ISA I followed the old i386 manual[2]. Its old but much more smaller. Perhaps once you feel comfortable, you can move to newer manuals. I never read newer ones (because I never actually needed assembly for any project, was just studying for fun).

[1] http://www.amazon.com/Computer-Systems-Programmers-Perspecti... [2] http://css.csail.mit.edu/6.858/2015/readings/i386.pdf

Computer Systems: A Programmer's Perspective, Bryant & O'Halloran. http://www.amazon.com/dp/0136108040/ http://csapp.cs.cmu.edu/public/pieces/preface.pdf

This is a unique blend of operating systems and hardware architecture, emphasising application programming over the system implementation approach in Hennessy & Patterson.

I highly recommend Computer Systems: A Programmer's Perspective

http://www.amazon.com/Computer-Systems-Programmers-Perspecti...

Oops!

http://www.amazon.com/Computer-Systems-Programmers-Perspecti...

Building a CPU in a circuit simulation framework[1] was one of my favorite, and most illuminating, projects I've done.

A great book to learn more: http://www.amazon.com/Computer-Systems-Programmers-Perspecti...

[1] http://tams-www.informatik.uni-hamburg.de/applets/hades/webd...

OK, if you don't have any real experience in low-level embedded coding (relevant to device drivers), RTOS or OS design in general, file systems, data structures, algorithms, interfaces, etc. And, if you have "hobby level" experience with Assembler, C and C++. And, if your intent is to write a desktop OS, from the ground up, without making use of existing technologies, drivers, file systems, memory management, POSIX, etc. Here's a list of books that could be considered required reading before you can really start to write specifications and code. Pick twenty of these and that might be a good start.

In no particular order:

1- http://www.amazon.com/C-Programming-Language-2nd-Edition/dp/...

2- http://www.amazon.com/The-Answer-Book-Solutions-Programming/...

3- http://www.amazon.com/The-Standard-Library-P-J-Plauger/dp/01...

4- http://www.amazon.com/C-Traps-Pitfalls-Andrew-Koenig/dp/0201...

5- http://www.amazon.com/Expert-Programming-Peter-van-Linden/dp...

6- http://www.amazon.com/Data-Structures-In-Noel-Kalicharan/dp/...

7- http://www.amazon.com/Data-Structures-Using-Aaron-Tenenbaum/...

8- http://www.amazon.com/Mastering-Algorithms-C-Kyle-Loudon/dp/...

9- http://www.amazon.com/Code-Complete-Practical-Handbook-Const...

10- http://www.amazon.com/Design-Patterns-Elements-Reusable-Obje...

11- http://www.amazon.com/The-Mythical-Man-Month-Engineering-Ann...

12- http://www.amazon.com/The-Programming-Language-4th-Edition/d...

13- http://www.amazon.com/The-Standard-Library-Tutorial-Referenc...

14- http://www.amazon.com/API-Design-C-Martin-Reddy/dp/012385003...

15- http://www.amazon.com/The-Linux-Programming-Interface-Handbo...

16- http://www.amazon.com/Computer-Systems-Programmers-Perspecti...

17- http://www.amazon.com/System-Programming-Unix-Adam-Hoover/dp...

18- http://www.amazon.com/Memory-Programming-Concept-Frantisek-F...

19- http://www.amazon.com/Memory-Management-Implementations-Prog...

20- http://www.amazon.com/UNIX-Filesystems-Evolution-Design-Impl...

21- http://www.amazon.com/PCI-System-Architecture-4th-Edition/dp...

22- http://www.amazon.com/Universal-Serial-System-Architecture-E...

23- http://www.amazon.com/Introduction-PCI-Express-Hardware-Deve...

24- http://www.amazon.com/Serial-Storage-Architecture-Applicatio...

25- http://www.amazon.com/SATA-Storage-Technology-Serial-ATA/dp/...

26- http://www.amazon.com/Beyond-BIOS-Developing-Extensible-Inte...

27- http://www.amazon.com/Professional-Assembly-Language-Program...

28- http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d...

29- http://www.amazon.com/Version-Control-Git-collaborative-deve...

30- http://www.amazon.com/Embedded-Software-Primer-David-Simon/d...

31- http://www.amazon.com/Programming-Embedded-Systems-C/dp/1565...

32- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...

33- http://www.amazon.com/Operating-System-Concepts-Abraham-Silb...

34- http://www.amazon.com/Performance-Preemptive-Multitasking-Mi...

35- http://www.amazon.com/Design-Operating-System-Prentice-Hall-...

36- http://www.amazon.com/Unix-Network-Programming-Sockets-Netwo...

37- http://www.amazon.com/TCP-Illustrated-Volume-Addison-Wesley-...

38- http://www.amazon.com/TCP-IP-Illustrated-Vol-Implementation/...

39- http://www.amazon.com/TCP-Illustrated-Vol-Transactions-Proto...

40- http://www.amazon.com/User-Interface-Design-Programmers-Spol...

41- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...

42- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...

43- http://www.amazon.com/Programming-POSIX-Threads-David-Butenh...

44- http://www.intel.com/p/en_US/embedded/hwsw/software/hd-gma#d...

45- http://www.intel.com/content/www/us/en/processors/architectu...

46- http://www.intel.com/p/en_US/embedded/hwsw/hardware/core-b75...

47- http://www.hdmi.org/index.aspx

48- http://en.wikipedia.org/wiki/Digital_Visual_Interface

49- http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-...

50- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...

51- http://www.amazon.com/Python-Programming-Introduction-Comput...

52- http://www.amazon.com/Practical-System-Design-Dominic-Giampa...

53- http://www.amazon.com/File-Systems-Structures-Thomas-Harbron...

54- ...well, I'll stop here.

Of course, the equivalent knowledge can be obtained by trial-and-error, which would take longer and might result in costly errors and imperfect design. The greater danger here is that a sole developer, without the feedback and interaction of even a small group of capable and experienced programmers could simply burn a lot of time repeating the mistakes made by those who have already trenched that territory.

If the goal is to write a small RTOS on a small but nicely-featured microcontroller, then the C books and the uC/OS book might be a good shove in the right direction. Things start getting complicated if you need to write such things as a full USB stack, PCIe subsystem, graphics drivers, etc.

I believe that

  Computer Systems: A Programmer's Perspective 

http://www.amazon.com/dp/0136108040

is superior to the widely mandated

  Computer Architecture: A Quantitative Approach 

for software engineers and programmers. The former has less hardcoded numbers than the latter and more timeless principles.

I have been keeping a list of books I used to augment my CS Masters Degree courses on various topics, here are the relevant ones I have found useful for the topics you have listed:

--Computer Organization--:

Computer Systems: A Programmer's Perspective http://www.amazon.com/Computer-Systems-Programmers-Randal-Br...

I liked this much better than Computer Organization and Design by Patterson and Hennessy which everyone has encountered at some point. The developer-centric view was very cool.

--Computer Security--:

Kernel Exploitation: Attacking the Core http://www.amazon.com/Guide-Kernel-Exploitation-Attacking-Co...

Most 'hacking' books are goofy. This one is very good and doubles nicely as a hackers operating systems text.

Web Application Hackers Handbook http://www.amazon.com/Web-Application-Hackers-Handbook-Disco...

Very nice overview for web concerns.

--Operating Systems-:

Operating System Design and Implementation http://www.amazon.com/Operating-Systems-Design-Implementatio...

I don't agree with Tanenbaum's views on micro vs. monolithic kernels but this book is a great mix of theory and implementation.

Linux Kernel Devleopment http://www.amazon.com/Linux-Kernel-Development-Robert-Love/d...

I used this to get a feel for the monolithic implementations of topics covered by Tanenbaum.

--Networking--:

TCP/IP Illustrated Series. More than you would ever want to know.