National central banks in the modern era have two functions: manage the money supply and make sure that the economy is running well. It's true that Libra would be the only direct impact, but then you have people who have debts in dollars and revenues in Libra, and if they can't pay back their dollar debts then their lenders are now short of dollars and can't pay back their obligations, and now we have a financial crisis.
Examples of this include 1997 in Asia: https://en.wikipedia.org/wiki/1997_Asian_financial_crisis