Should I be leaving my Ball Python to adapt to his terrarium before casually removing him?

It would be best to let him settle. How is his home set up? Does he have a cave or somewhere where he can hide, on the warm side and the cool side of his tank? The set up should have the heating pad and the heat lamp on the same side so there is a heat variance. Also you don't want to give him a live adult mouse because if he is stressed he won't eat and the mouse can bite and badly injure your ball python, I have seen it happen.

Besides that you let him get used to everything for at least a week before you start handling him a little bit every other day. He may not like it at first but he will tame down a bit. I have tamed a lot of aggressive snakes including an 8 foot Burmese python, who did bite me, and a 3 foot green anaconda which was very aggressive and wasn't eating.