You can also use "мучить/казнить/изводить" instead of "терзать" to make it more colloquial.

I'm wondering whether "You needn't to give yourself a hard time!" or something aling the lines would work here. ;)