Make it in two bits, the majority of the depth can be round, then just make (or buy?) A d shaped washer to go on the top. 1/4" thick would be plenty and easy to file.
You've lost me, between the photo and description. Unless my brain is just having a moment, your description and the photo seem to be contradicting each other. Does the spacer drop into that hole, or go from the other side?