I've always had an aversion for physical contact as well. That said, before 16 or so I was relatively normal-ish so I had a couple girlfriends, and some experience in non-serious relationships.
I've no idea where this complete disregard for human warmth comes from as I wasn't abused or whatever as a child. Fucking peculiar.