It cost you something to let your patience grow, but let it grow.
Read: James 1:3-4
3 For you know that when your faith is tested, your endurance has a chance to grow. 4 So let it grow, for when your endurance is fully developed, you will be perfect and complete, needing nothing.