Check out your drill press.
Does the 1/2" jacobs chuck come out? Mine has a #2 Morse taper on the quill that the chuck fits into. If yours does, get some bits with the morse tapers on them and skip the chuck altogether.
Next idea - If you do have the morse taper feature there should be a slot in your quill, (the shaft that goes up and down as you drill). With the slot you use a drift to knock out the chuck.
Now with that feature you may also look into four flute drills. They are hard to find and they aren't used to start holes. They make existing smaller holes bigger. The value of four flutes is that they make perfectly round holes. Beautifully round holes. They are like a ream and drill in one tool. You might find them at auctions of machine shop equipment. They would go cheap as most people don't know what to do with them. Try one and you'll love them.
The pix should make all this a whole lot easier to follow.
Hey- slowest possible speed when doing the big holes. Slow feed. Make chips. Lots of lube. CLAMP IT DOWN.
Use yourself as the standard,
seek to beat it daily,
Lincoln AC/DC tombstone
Hobart gas powered DC stick, 6 cyl flat head.
Miller 251 MIG
little Hypertherm plasma.