It might be normal. The location function, of being able to hold position is purely visual.
If your floor, or any surface under your Tello is too plain, it can drift. Try flying it over an area that has some visible contrast.
I once took mine on a walk through a park, and it performed very...