http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh...
tl;dr I think the main skill that is missing is being able to write a Python script that fits well within a shell script. The shell script often invokes tools written in other languages like C, Rust, JavaScript, etc. (most of which you didn't write)
Good book on this: https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...
Online for free: http://www.catb.org/esr/writings/taoup/html/
The best reference I've found on this is The Art of Unix Programming, which was published in 2003:
https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...
The book is also online for free:
Command Line Options: http://www.catb.org/~esr/writings/taoup/html/ch10s05.html
User Interface Design Patterns: http://www.catb.org/~esr/writings/taoup/html/ch11s06.html
Still it's nice to see this knowledge circulating!
Joel Spolsky's 2003 review of this book: https://www.joelonsoftware.com/2003/12/14/biculturalism/
The question isn't shell OR Python.
My shell scripts call Python scripts, many of white I wrote myself. That is working as intended.
They also call C programs, C++ programs and R programs. And put JavaScript programs in various places.
----
I guess this is a huge misconception about shell that I have to write a blog post about.
https://news.ycombinator.com/item?id=24083764
I'm often asked this about Oil [1]: Why do you want to write programs in shell?
That's not the idea of shell. The idea is that I write programs in Python, JavaScript, R, and C++ regularly, and about 10 different DSLs (SQL, HTML, etc.) And I work on systems written by others, consisting of even more languages.
I need a language to glue them together. A language to express build automation and describe deployed systems. Most big systems consist of more than one language.
Shell is the best language for that, but it's also old and crufty, with poor implementations.
When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".
-----
If you want some concrete examples, look in the Oil repo. There are dozens of shell scripts that invoke custom tools in Python, R, and C++.
https://github.com/oilshell/oil/tree/master/benchmarks
For example, to generate this report on how much we speed up Python by translating it to C++: https://www.oilshell.org/release/0.8.3/benchmarks.wwz/mycpp-...
The tables are manipulated by R code, and shell/Python/CommonMark generates the HTML.
-----
Another example is that the release page is essentially a huge shell script: https://www.oilshell.org/release/0.8.3/
This page and all the linked pages are generated by: https://github.com/oilshell/oil/blob/master/devtools/release...
------
If you don't understand why shell should call Python and C, I recommend reading The Art of Unix Programming
http://www.catb.org/~esr/writings/taoup/html/
https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...
My first reading of this book convinced me to move away from Windows. I usually read it at-least once a year.
A Pattern Language, Alexander and Ishikawa and Silverstein http://amzn.to/2s9aSSc
Advanced Programming in the Unix Environment , Stevens http://amzn.to/2qPOMjN
Algorithmics: the Spirit of Computing, Harel http://amzn.to/2rW5FNS
Applied Crytography, Wiley http://amzn.to/2rsULxS
Clean Code, Martin http://amzn.to/2sIOWtQ
Clean Coder, Martin http://amzn.to/2rWgbEP
Code Complete, McConnel http://amzn.to/2qSUIwE
Code: The Hidden Language of Computer Hardware and Software, Petzold http://amzn.to/2rWfR9d
Coders at Work, Seibel http://amzn.to/2qPCasZ
Compilers: Principles, Techniques, & Tools, Aho http://amzn.to/2rCSUVA
Computer Systems: A Programmer's Perspective, O'Hallaron and Bryant http://amzn.to/2qPY5jH
Data Flow Analysis: Theory and Practice, Khedker http://amzn.to/2qTnSvr
Dependency Injection in .NET, Seemann http://amzn.to/2rCz0tV
Domain Driven Design, Evans http://amzn.to/2sIGM4N
Fundamentals of Wireless Communication, Tse and Viswanath http://amzn.to/2rCTmTM
Genetic Programming: An Intrduction, Banzhaf http://amzn.to/2s9sdut
Head First Design Patterns, O'Reilly http://amzn.to/2rCISUB
Implementing Domain-Driven Design, Vernon http://amzn.to/2qQ2G5u
Intrduction to Algorithms, CLRS http://amzn.to/2qXmSBU
Introduction to General Systems Thinking, Weinberg http://amzn.to/2qTuGJw
Joy of Clojure, Fogus and Houser http://amzn.to/2qPL4qr
Let over Lambda, Hoyte http://amzn.to/2rWljcp
Operating Systems: Design and Implementation, Tanenbaum http://amzn.to/2rKudsw
Parsing Techniques, Grune and Jacobs http://amzn.to/2rKNXfn
Peopleware: Productive Projects and Teams, DeMarco and Lister http://amzn.to/2qTu86F
Programming Pearls, Bentley http://amzn.to/2sIRPe9
Software Process Design: Out of the Tar Pit, McGraw-Hill http://amzn.to/2rVX0v0
Software Runaways, Glass http://amzn.to/2qT2mHn
Sorting and Searching, Knuth http://amzn.to/2qQ4NWQ
Structure and Interpretation of Computer Programs, Abelson and Sussman http://amzn.to/2qTflsk
The Art of Unit Testing, Manning http://amzn.to/2rsERDu
The Art of Unix Programming, ESR http://amzn.to/2sIAXUZ
The Design of Design: Essays from a Computer Scientist, Brooks http://amzn.to/2rsPjev
The Effective Engineer, Lau http://amzn.to/2s9fY0X
The Elements of Style, Strunk and White http://amzn.to/2svB3Qz
The Healthy Programmer, Kutner http://amzn.to/2qQ2MtQ
The Linux Programming Interface, Kerrisk http://amzn.to/2rsF8Xi
The Mythical Man-Month, Brooks http://amzn.to/2rt0dAR
The Practice of Programming, Kernighan and Pike http://amzn.to/2qTje0C
The Pragmatic Programmer, Hunt and Thomas http://amzn.to/2s9dlvS
The Psychology of Computer Programming, Weinberg http://amzn.to/2rsPypy
Transaction Processing: Concepts and Techniques, Gray and Reuter http://amzn.to/
Types and Programming Languages, Pierce http://amzn.to/2qT2d6G
Understanding MySQL Internals, Pachev http://amzn.to/2svXuFo
Working Effectively with Legacy Code, Feathers http://amzn.to/2sIr09R
Zen of graphics programming, Abrash http://amzn.to/2rKIW6Q
[0] http://www.amazon.com/UNIX-Programming-Addison-Wesley-Profes...
3.0 out of 5 stars Autohagiography with some programming tips, December 24, 2003 By A Customer
The writing style of this book tends to hurt the reading experience, as Raymond trumpets his own minor achievments in the free software community. The work feels like it needed one more rewrite before being released to the public: some related sources Raymond hadn't yet read at the time of writing, and some of his advice gets repetitive.
The exposition itself is not up to par with The Elements of Programming Style. Raymond tries to give a list of programming rules or principles to follow, but it reads more like a list of slogans that should be taken as axioms. While The Elements of Programming Style itself had a list of rules, the rules were well woven with each other, well defended, and they were used as a means of conveying a larger story. In Raymond's case, he relies upon the slogans in absence of such a story.
Thus, the book ends up more like a list of random unrelated tips. Some very profound, like his writings on threads (which he acknowleges Mark M. Miller for his help). Others are very shallow and pointless in a book that supposes to call itself about "Art." Some of the pieces appear only to function to attack Windows, and sometimes the information about Windows is embarassingly inaccurate.
One final criticism is that Raymond does not understand object-oriented programming very well and misses the point in several cases. You just need to see the popularity of Python, Java, C# (Mono), OO Perl and C++ in the Linux world to see that Raymond is off base calling OO a failed experiment. In fact, with almost any matter of opinion in the book you can feel Raymond's bias and be hit in the face with misinformation or dull false dilemmas.
However, given this book's many flaws, I rate this 3 stars instead of 2 stars because it also has valuable information from the many contributors, some of them Gods in the Unix world. These contributors often even disagree with Raymond, or point out other interesting tidbits. For these tips alone, it is worth checking out this book, though I would not recommend you buy it.
To get the true Unix programming philosophy, I recommend Software Tools, by Kernighan and Plauger. It's somewhat dated, and I recommend the Ratfor version of it, but that single book has became very influencial as I grow as a Unix programmer.
[...]
One of the problems with this book is the overly partisan tone it takes - one gets the impression that absolutely nothing Microsoft has ever done is of value, but the other major desktop PC OSes (Apple, Linux) represent different forms of perfection. (At home, I run Mac OSX, RedHat Linux and Windows, and have a reasonable sense of their relative strengths and weaknesses.)
So, be warned: Art of Unix Programming paints a one sided picture. The author is a well-known figure in the open source community, one of its fiercest advocates, and one of Microsoft's most vocal critics, so it might seem to strange to wish for less anti-Microsoft spin from this source. After all, the Raymond brand certainly carries with it an obligatory expectation of Windows-bashing, doesn't it?
One of the only Windows design decision which Raymond doesn't condemn is the (now discontinued) .ini file format. Even the thorough-going support for object-orientation in Windows is given short-shrift: after explaining the many horrors of object-oriented programming (according to Raymond), Unix-programmers are praised as "tend[ing] to share an instinctive sense of these problems." This section ([...]) is particularly illustrative of the one-sided approach that Raymond takes.
[...]
His comments about the Windows registry were a bit distressing, though -- not because they're negative, which I consider fine. Rather, it was obvious he'd never used it (comments like "there's no API for it") and it was also clear that he hadn't even bothered to research why it existed and what problems it was intended to solve. The comments were typical of what I'd expect of a Slashdot troll, but not of a bright, respectable person like ESR. I've programmed on both platforms extensively and only comment on what I have first-hand experience and knowledge of; I'd expect him to do the no less, especially as an author.
It was also curious that several times he implied unit testing == XP == agile software development. For as tuned in as he seems to be to methodolgy work, missing the forest for a single leaf is a bit embarrassing.
[...]
Hate to be the one to burst the proverbial bubble..., March 13, 2004 By Bill Joy (Hamburg, DE)
...but ESR's theory and ideology is rediculously flawed with misappropriated valuations. This is, yet another advocacy campaign for gnu/Linux, mixed in with a UNIX context. Given the purpose of the book, it's a fair assessment to label the burdening bias as filler and firewood: filler for those who really just wanted an explanation of the single-purpose, POSIX illustrated, hows and whys of UNIX programming philosophy; and firewood for those who tend to buy into slanderous hype at the whim of suggestion.
Questions: What is the title of the book? What programmatic philosophies are portrayed? How many unices are open source? Of those, how many subscribe to the same opensource mentality as gnu/Linux? Answer: zilch. Question: Then what is the relevance of such topics to the objectives of the book, as depicted in the title? Answer: fudd-ala-mode.
Aside from these intertwined distractions, and a severe Napoleonic complex against anything new or different, ESR does adequately represent the real purpose of the book, and in those efforts place value in the read.
[...]
an interesting and often annoying read..., June 1, 2004 By A Customer
I suppose any book containing so many interesting quotes from so many UNIX luminaries cannot be overlooked. (I wonder if any of them would have co-authored this) It also happens to contain a great many topics that are well-worth writing about; My only wish is that someone less in awe with the contents of his own field of vision, and with greater depth and objectivity (not to mention humility) had the opportunity to write this book.
Quality of discussion is varied as expected; Raymond is not quite the UNIX expert he thinks he is. In places, Raymond's tone encourages one to throw the book at the nearest wall and go out just to get some fresh air; He is condescending, hectoring, lecturing, and sometimes just misleading. Alas, I will still recommend it as worth reading (check your local library) with a nice grain of salt; just enough friction for thought is provided in this edition.
[...]
programming book without any code, May 11, 2013 By davez (LA, CA, United States)
If you expect to see any code, you'll be disappointed. The author gives ample examples, but mostly preaching the Unix philosophy or religion, at many times repeating the similar points. 500 pages are a little long for a book of this kind. Having read other Unix books, I've heard many of the stories retold here, but still learned a few new things. It reenforces many notions of Unix.
This is a book that I would recommend to someone new to Unix, but it is not a book that I would keep. After 10 years, the computing landscape has changed with the rise of the mobile computing and cloud computing; the Unix has evolved to support smart phones and tablets, in which the user interface has become essential.
https://github.com/onetrueawk/awk - UNIX philosophy of small tools, DSLs, CS theory: state machines / regular expressions, Thompson algorithm ...
https://github.com/mirrors/emacs - Both a program and a VM for a programming language, hooks, before/after/around advices, modes, asynchronous processing with callbacks, ... Worth to think of challenges of designing interactive programs for extensibility.
https://github.com/rails/rails - Metaprogramming DSLs for creating powerful libraries, again a lesson in hooks (before_save etc.), advices (around_filter etc.), ...
https://github.com/git/git - The distributed paradigm, lots of CS theory again: hashing for ensuring consistency, DAGs everywhere, ... By the way, the sentence "yet the underlying git magic sometimes resulted in frustration with the students" is hilarious in the context of a "software architecture" course.
One of computer algebra systems - the idea of a http://en.wikipedia.org/wiki/Canonical_form
One of computer graphics engines - Linear algebra
...
There are loads of things one can learn from those projects by studying the source in some depth, but I can't think of any valuable things one could learn by just drawing pictures of the modules and connecting them with arrows. There are also several great books that explore real software design issues and not that kind of pretentious BS, they all come from acknowledged all-time master software "architects", yet all of them almost never find diagrams or "viewpoints" useful for saying the things they want to say, and they all walk you through real issues in real programs:
http://www.amazon.com/Programming-Addison-Wesley-Professiona...
http://www.amazon.com/Paradigms-Artificial-Intelligence-Prog...
http://www.amazon.com/Structure-Interpretation-Computer-Prog...
http://www.amazon.com/Unix-Programming-Environment-Prentice-...
http://www.amazon.com/Programming-Environment-Addison-Wesley...
To me, the kind of approach pictured in the post, seems like copying methods from electrical or civil engineering to appear more "serious", without giving due consideration to whether they really are helpful for anything for real-world software engineering or not. The "software engineering" class which taught those kind of diagram-drawing was about the only university class I did not ever get any use from, in fact I had enough industry experience by the point I took it that it just looked silly.
Hard copy:
https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe...
Soft copy:
http://catb.org/~esr/writings/taoup/
http://catb.org/~esr/writings/taoup/html/
The original Unix koans are found in Appendix D: http://catb.org/~esr/writings/taoup/html/unix_koans.html
Additional Unix koans were added (and removed) over time: http://catb.org/~esr/writings/unix-koans/