Take your end mill chuck in a hand drill and tell me how much it pulls to the side. Also a cheap press with side to side play doesn't make a round hole.
There is a lot of sidways force from that cutter. Its a lot different than a drill bit.
Hand drill...*** u talking bout and smokin willis?
Now that's something completely different. It is not desgined for that. Same goes for using a tapered ream I'd only use that for this purpose in a good quality benchtop drill press or as the one I'm looking at buying a nice Radial arm floor standing press. Then with moderate pressure and correct speed things happen nicely.
Using a hand drill for an end mill well that's just W R O N G by any standard...there are tools for the job and hand drills are not one of them. However this procedure is not one for a hand drill really anyway to be accurate and square! That is unless you are using a MAG drill on a big plate bolted to the case! That's different ....
just saying!