I just looked at my rep and I got hit with -70 because "User removed". Was someone banned? And I thought after about 5 mins up/down votes were cemented in and couldn't be changed unless an edit to the question/answer was made. Why do I get punished for someone else leaving?
When accounts are deleted, their votes go away with them. See What does “user was removed” mean and why did my reputation change because of it? on the main meta.
This isn't a user who was banned. There are only three reasons an account would be deleted:
You are not punished for that. The reputation you lost is reputation that you would not have gained if the user hadn't voted for you. You don't take a penalty: since the user is no longer there to grant you these votes, you can no longer claim to have earned them.