I think it's worth trying for a couple of weeks. If it doesn't start to click, then come back after a few years of imperative programming, at which time it won't be as easy as you'd expect (or so it is reported) since you have to learn the very different functional paradigm, but you'll appreciate it all the more having accumulated plenty of scars from side effect caused bugs.
That said, if you have something you want to build starting with a project and an appropriate language and ecosystem (including instructional material) is very good. I myself started out in a high school punched card IBN 1130 FORTRAN "IV" course, but soon started a project that really got me going.
I didn't try SICP until half a decade later; by then I was an experienced imperative language programmer (mostly C by that point) and fairly experienced in Maclisp and Lisp Machine Lisp. At which point it really made sense, both in terms of being able to understand the material and why it was really important.
