The Torque Enable setting toggles the supply of current to the motor on and off.
The difference in behavior you are reporting is because when the torque is off the motor is completely de-energized, and is not adding any resistance to backdriving. Once the torque is enabled the motor is actively adding backdriving resistance.
Using Current Based Control mode can allow you to adjust the amount of compliance when torque has been enabled, but this general behavior is intended operation and cannot be disabled.