Being big is also a problem, but I did in fact mean heat.
"Not melting itself" is not the threshold we need to meet for a phone. We should probably consider 5 thermal watts the limit. If we have a 30% efficient betavoltaic panel, then we can produce 1.5-2 watts of device power within that limit. If we have a 5% efficient panel, then we're only getting a quarter of a watt. A quarter of a watt can stop an idle phone from draining, but you'd only recover about 1% of a normal phone battery per hour.
By the time we're considering external devices, I think "battery bank with a solar panel" beats betavoltaic on the technical merits even when we're ignoring difficulty and radiation. Even a tiny panel built into the back of a phone generally wins.