The aggressor, in the process of atoning for their atrocities, has no right to say that the recourse proposed by the victim is unreasonable.
We are the colonial aggressors, Indigenous people are the colonized victims.
Let’s say a man and a woman live in the same house, and the man hits the woman. If the man is truly seeking to atone for his crime, and the woman tells him to move out because even seeing his face is traumatic for her, would it be reasonable for the man to complain that he has nowhere else to go? To ask the woman where she thinks he should go? To try and guilt the woman into letting him stay? If he does any of those, is he truly sorry for what he did?
You’re right that most Indigenous people don’t want mass expulsion. We should be incredibly grateful for that and it’s a testament of their compassion and desire for equality among all people, even after all we did to them. What we shouldn’t do is tell them that they can’t tell us to leave or that we’d refuse to leave because we have a rightful claim to this land. Doing so is completely unproductive and will only serve to make us less deserving of staying.
IMO, simple fact that the majority of drunk drivers survive accidents that kill their victim instantly shows that no higher power is looking out to punish the wicked or protect the innocent.