There were black peoples, or at least Middle Eastern, all over Medieval Europe. The Arthurian cycle has moorish knights. When Eleanor of Aquitaine married the English king, part of her dowry was a North African city. Are we to believe none of them travelled to England? Berber pirates raided Iceland for slaves so there were whites people in Africa, why not black people in Europe?
Some people travelled a lot. The Saxons sent emissaries to what’s now Sri Lanka, while an Armenian Christian monk born in Beijing met the pope and Edward I. In Middle Earth the Numenorians had extensive colonies in the far south of the continent. This obsession with racial segregation is absurd and utterly ahistorical.