Can I give my bf genital HSV-1 during an outbreak if he already has HSV-1 orally?

My boyfriend gave me HSV-1 on my genitals. He has never had a cold sore since I've been with him, so we assume I got it from viral shedding during oral sex. He does not have HSV-1 on his genitals. We have been told by a doctor that having unprotected sex was fine, because he has built up immunities which make it almost impossible to contract it on his genitals. I was wondering if this fact stood true when I am having an outbreak.