Coming from a guys perspective, I would want you to tell me straight out ASAP. I'll be hurt, but it's worse when girls don't say anything and are wishy washy about what to do. You guys then sort of blow us off, then sort of not, then sort of do. It's like, come on already. Just say it. When you finally do tell him what you meant to say months agao, then he'll feel led on for all that time.
Just kick him to the curb. Throw out the fact that he said you are the only thing that is going good in his life. He'll just have to suck it up. That's life.
Also, he's down in his life right now. You don't want that baggage during school. This is important and should be a priority.
Plus, he's not the same year as you so who cares if you see him in the halls. He has no classes with you, and unless he has a key or something to your home/apt, I can't see the problem. If he stalks you, call the police. That's what my sister did, and she got a restraining order on an ex. He never showed up again.