Yes, unless you have prior experiences, it is that hard. I agree with the others here who recommend you build an on-site tool/storage shed as practice. And no, it's not just about building foundations, floors, walls etc. You will need to consider how to rough-in for HVAC, plumbing, electrical, etc. as part of the build. Your wall-framing techniques have to take into consideration things like openings for doors, windows, and even how you will hang sheetrock.

Your biggest hurdle is probably going to be the permitting/inspection processes, assuming you want a legally habitual dwelling.