Generally speaking, people join teach for america to improve their resumes, and get into better graduate schools. People who aren't trying to get into better graduate schools defer for a few years, and attend later on the same acceptance.
Because you rejected their initial acceptance rather than defer, they reasonably believe that you're shooting for a better school. Why accept you if you're just going to go to a better school? It brings their acceptance rate up, lowering their US News rank, which is unfortunately near the top of admission directors list of priorities.
If you really want to get in, make sure admissions know that you're serious about going there, and that they aren't just your "safety school" because you were accepted in the past.