There was an accident with a Volvo where someone was showing off the pedestrian safety system, and hit a pedestrian. It turned out they hadn't purchased the pedestrian safety system.[1] Something is needed to prevent problems like that.
Then there's mode switching trouble. Classic problem with aircraft control systems. Tesla disengages the "autopilot" if the driver touches the brake. The trouble is that this also disables automatic braking, as the driver is assumed to now be in control. So tapping the brake without applying it fully in a hazardous situation causes a crash.
All these driver behavior problems with shared control authority are hard. Maybe harder than going for level 3 and letting the automation do it.