First it's a question of what you expect from RTH.
This RTH is based on the VPS data which is based mostly on dead-reckoning the images of the ground facing camera. Anything that confuses the VPS will cause low RTH accuracy.
If you fly on a relatively calm day in video mode over ground with easily recognizable patterns and too different altitude levels (trees, houses, etc) it should bring
Tello back near to you. If you fly for a few hundred meter it should be within maybe 10% of the travelled distance under such conditions. So after 500m flight a RTH to 50m within takeoff position is reasonable.
Closer is better but not to be expected. This is the level of RTH accuracy
Tello can do. If you expect better accuracy there is no way around a GPS drone.
Different conditions reduce the accuracy: Wind for example, flying in fast mode with higher speeds and higher attitude angles, flying over different ground levels (houses, trees), flying too low (below 1.5m) or too high (well above 15m), flying over dull fields or clean pavement, etc. Such conditions will reduce the RTH accuracy to some degree, but it should be OK otherwise.
If your
Tello can't achieve acceptable RTH accuracy in good conditions then you have a hardware problem with your drone. There is definitely no SW problem, at least not in recent firmwares. Could be vibrations (props, motor shafts, prop guards) that cause both the camera and the IMU to produce unreliable data. Or could just be a little dust in the pinhole of the camera, or a scratch.
You can try to calibrate both gravity and IMU, mount fresh OEM props, check the motor shafts, and use a fine brush to clean the camera pinhole. If that doesn't work I'm out of ideas