The way I perceived of what you said, the IMU is what determines where in space the drone will be in relation to you and in what direction it is headed.
The IMU basically makes sure that the center of gravity is leveled so the craft can achieve lift and sustain flight.