I think part of the problem is that universities have lots of people who do one job and that job is not everyday. For instance, where I'm at we have two people in charge of summer enrollment. That seems to be it. They are way way overworked for about two weeks at beginning of the summer. I have no idea what they do the other 50 weeks of the year. I think their boss is happy as long as they deal with summer courses.
“I have no idea what these other people I don’t work with do, so it must be nothing” is a really naive and insulting thing to say. They probably don’t know what you do either, would it be fair to say you do nothing of value?