Having been a kid who moved around a fair bit, and was in the "gifted" (cor, I hate that word) programs when they were available, and in the regular classes when they weren't, I decry this false dilemma.
The truth of the matter is, I learned more and was more intellectually stimulated when I was in the regular classes. Because they let me burn through the homework more quickly, leaving more time in the evening to pursue my own projects and pursuits.
The gifted programs were certainly more difficult. But they largely achieved that by assigning more homework, and it was invariably crap work - at best, something that was much more interesting to the teacher than it was to me, and just as often pure busywork that was made artificially difficult for the sake of "challenging" me.
In some sense, I even found them insulting, and I found my parents' need to put me in them insulting, because the clear implication was that the only achievements of mine that mattered were ones that could be written down and certified by the school. Spending my time learning more musical instruments wasn't an option, because school would only give me a report card grade on one instrument at a time. Spending my time learning an additional language - perhaps a more challenging one than anything my school offered - wasn't an option, because school would only give me a report card grade in Spanish. Spending my time learning linear algebra so I could do 3D graphics programming (this was before OpenGL was really a thing, so you had to DIY) didn't count, because that wasn't on the list of kinds of math that had been blessed by the school's curriculum. It was absolute BS; from my perspective my personal interests were being subtly curtailed because they couldn't be pursued in a way that let my parents feel like they were Good Students™ vicariously through me.
And the worst of it is, I'm pretty sure parents and school systems are so busy stumbling over themselves to create these elitist selective enrollment programs that they trap both the kids who do get into them and everyone else into a local minimum. The very best classes I experienced in school weren't the "advanced" ones, they were the regular (on paper) ones that followed more of an inverted classroom model, where the lecture time in the classroom was kept to a minimum, so that more time could be reserved for working together on homework. And the very best of those was a math class, where those of us who had a relatively easier time following the textbooks and lectures could put that advantage to good use by then trying to explain the new concepts to our classmates who were having a harder time. It was better for me intellectually, because, unlike what a lot of gifted programs seem to believe (or at least did a few decades ago), depth of knowledge is at least as important as breadth of knowledge, and there's no better way to ensure you've mastered a subject than teaching it to others. And it was better from a mental health perspective, too, because it steered clear of turning school into this awful competitive tar pit where the students who are having an easier time are seen as making everyone else seem comparatively inferior due to an unearned natural talent.