You should go for it. Check out things like csound [1], puredata [2], chuck [3], and supercollider [4]. "Computer Music" by Charles Dodge is a classic book on computer music [5] (oldie but goodie)

[1] http://www.csounds.com

[2] http://puredata.info

[3] http://chuck.cs.princeton.edu

[4] http://supercollider.sourceforge.net

[5] http://www.amazon.com/Computer-Music-Synthesis-Composition-P...