Lots of people that aren't working do pay taxes though. Social Security benefits, for example, are taxable in a lot of cases.

Using 2015 as a reference, of the people that work in the US (represented by 141,204,625 tax returns), the top 50% paid 97.17% of all federal income taxes. The statement that about half of Americans don't pay federal income taxes is only for those that are filing taxes (and therefore probably working), so if you want to include the non-working, it's more like 75% of Americans don't pay income taxes. Here is the data from the IRS, look at the last table cell at the bottom all the way on the right. https://www.irs.gov/pub/irs-soi/15in03etr.xls