Sovereign nations do indebt themselves through banks and financial markets in general, most often through the issuance of bonds, which can be defined as highly tradeable loans.
This is why sovereign debt is given a rating by credit agencies and why interest rates on debt freely vary based on the perceived risk by creditors. This also means that it can become difficult for a country to borrow at all (hence organizations like the IMF sometimes stepping in)
There is no magic.
However you spend money, inflation is created by an excess supply, not least when money is printed to cover debts (an effective devaluation). This is actually what happened in Zimbabwe.