The problem is hitting the damn things. They're small, can turn on a dime, don't reflect radar very well, and run cold enough to give IR tracking some issues. Army manuals like [1] typically call for a coordinated fire of small arms across a large volume of airspace to hit a drone, but advise commanders that hit probabilities are low. Their most urgent advice is to keep spotters looking for enemy drones, and to use concealment and camouflage as much as possible to avoid enemy drone surveillance.
[1] https://fas.org/irp/doddir/army/atp3-01-8.pdf, search for "UAS" (Unmanned Aerial System)