People used to memorize a few phone numbers, likely less than 10, and used notebooks made specifically for writing down phone numbers to keep track of the rest.
Phone numbers of the people you called the most (the 10 you memorized) were overwhelmingly likely to be local numbers, so you were only memorizing (3 number chunk) + (4 number chunk). Password rules are all over the place. Memorizing numbers, letters, whole words, the capitalization of those letters and words, and special characters, that are far longer than ye olde timey phone numbers, is orders of magnitude more difficult.
I have over 100 passwords in my password manager. They are all unique, so if any one is compromised, it is contained. My password manager is protected by strong 2FA, so someone would have to physically interact with my property to gain access. In the real world, there is no scenario where memorizing all your passwords is more secure.
