And is it acceptable usage?
My thought process, followed by my answer:
1. Silence is good.
2. Eliminating unnecessary words increases silence, and therefore good.
3. Everyone will understand "I've never been!" to mean "I've never been there!" given the context of the exchange.
4. "There" is an unnecessary word in the sentence "I've never been there".
5. Eliminating the word "there" increases silence, and therefore good.