I first learned about calculus from, I think, this book, "Calculus the Easy Way" [1] (or, at least the first chapter or two). It does make use of some algebra, but you're not necessarily limited to the strict progression of pre-algebra -> algebra -> calculus.

[1] (amazon: https://www.amazon.com/Calculus-Easy-Way-Douglas-Downing/dp/...)