When a GC-managed language uses reference counting as implementation algorith, the compiler might be able to optimize the reference counting to only occur when it is unavoidable or too costly to reason about, in a similar vein to bounds checking.
When using library types for reference counting, there is no way for the compiler to implement such optimizations, unless the types are somehow tagged with compiler intrisics so that they could apply the same kind of optimizations.