This is my mindset as well. I often need to work with text that isn't code. Maybe it's a SQL dump someone sent me, some bespoke configuration format, a collection of log files, etc. Knowing how to manipulate and navigate text is extremely helpful as a general skill. IDEs degrade to notepad.exe quality when presented with unknown formats.
Personally, my recent work has been: short term freelance work, new code bases, a variety of programming languages including ones I haven't previously used, problem identification and solving, and mature code bases where you'd only write a couple lines of code a day.
In that context an IDE optimizes a tiny part of the work, while getting in the way for everything else.
I'll add that when I worked on a greenfield Java project as part of a large team, I absolutely used an IDE (IntelliJ at the time) and couldn't imagine working differently.