There is a really good book out there on one person startups called the $100 startup by Chris Guillebeau - it gives loads of examples of startups around the world, which were started by one person (http://amzn.to/1Kb63VH)
It was a little bit too self-help-y in tone for my taste, but it didn't feel scammy like "4 hour work week" (and the book mentioned in this OP sounds), and has lots of basic, practical advice.
The $100 Startup book addresses this kind of questions quite well. It's essentially a list of study cases of people that built side businesses (some of them becoming their main sources of income or even small very profitable companies) using their skills and interests. Might not give you silver bullet answer, but it's certainly inspirational and it does point out to certain patterns of how to do that... http://www.amazon.com/100-Startup-Reinvent-Living-Create/dp/...