Drinking and pregnancy all depends on what your goal is. If you want to minimize birth defects and health problems, then you don't drink while pregnant nor when you are trying (or not actively preventing) pregnancy. The risks are smaller in the first Trimester, but still existant. For that matter, I'm pretty sure that alcohol impairs sperm as well, so to minimize the bad chances he shouldn't drink during the trying phase either.
Otherwise, third trimester is definitely too late. Most doctors would probably recommend stopping by the second trimester.
It is all a sliding scale of risk. One drink might not lower your child's LSAT score, but one per day would forever doom him to T4 schools.