I would trust your own instincts and internal compass: if you are learning new things and feel "flow" when you spend time on this side project, keep at it. It will have some kind of impact.
One good article worth reading is Peter Drucker’s "Managing Oneself" (available here http://www.amazon.com/Managing-Oneself-Harvard-Business-Clas... and a little Googling may turn up other versions as well) where he stresses the importance of learning how you learn.
One good book on this is "The Three Boxes of Life" by Richard Nelson Bolles http://www.amazon.com/Three-Boxes-Life-How-Them/dp/091366858... which covers life work planning: the three boxes are education, work, retirement (play).
In my own case I have always been a connector and cross-fertilizer, trying to apply insights from one field or area to another, sometimes where they hadn’t been considered before. There are many narrow specialists out there, you should absolutely feel free to follow your Muse as long as you are learning.
https://www.amazon.com/Managing-Oneself-Harvard-Business-Cla...
Drucker takes a firm stand on this issue. He advises to "put values first"