Because to make a significant difference to that person's life you'd have to give them an astounding amount of money (and probably other forms of support), and personally speaking I'm still desperately saving just for a chance at having a place of my own myself. Nobody claimed to be 'virtuous', but caring is a good first step. I donate some of my time and a small amount per month to a local multicultural group. Small things help. Caring helps. Why is this suddenly about whether you're happy to give away half your savings to a person you don't know? Society as a whole could make things better by sharing wealth just a little more equitably. But some people seem to think that billionaires are just fine and not at all a symptom of a sick society.
Sorry but I get really frustrated by this lazy kind of 'well why don't you do it?' response. It just feels like you're trying to convince yourself that you're right not to care (to be clear, I have no idea whether you do or not), and implying that anyone who does, and hopes for something better, is just a hypocrite.