Some people will score you lower in interviews due to the facial hair. They may think your particular style looks ugly, scruffy, etc. It may also just be a sub-conscious idea that peoplw with facial hair are hiding something and less honest.
Very few people (if any at all) will score you lower for having a clean-shaven face. So play the odds.
Since you describe it as a small amount, it probably won't take long to grow back once interviews are done. You might be able to keep it over the summer job, as people won't be judging you based purely on a 30 minute slice of time.