The kids build their connection to you in weird ways, smell, touch etc. Avoiding that 100% until they speak could have weird effects in their attachment (said without any actual proof).
It's of course your choice, but just saying that that choice could end up biting you or the kid in the end.
No it's not fun hanging around kids all the time, but it takes a lot of not fun moments to get to the good parts.
