Did you not heed the advice above?
There's a lot of messages in this thread given by people with no experience of producing precision holes with reamers!
@pressbrake1 has told you exactly how it should be done.
A 4.0mm reamer needs a drilled hole of 3.88 to 3.92mm diameter.
Don't pilot the hole with a 3.5mm drill, when you stick the 3.9mm drill though after it's going to wander and not cut a round hole.
There's a lot of advice above, and I thought I HAD followed most of it.
I had read it as .25mm being 'nice' - and another of 10 thou (0.254mm?) being good. As such, I reasoned it was ok to use 3.8, as there's no risk of it cutting over 4.0.
The mistake I made which you suggested to avoid @Brad93 was I drilled the 3.5 on the mill, but thought it would be OK to drill the final 3.8 on the pillar drill, prior to hand reaming it. That may have led to the 3.8 not being completely centred.