If I disable that your project builds in 30 seconds (no cache!) on my pretty old and not very fast i5-8350U, which is roughly what I'd expect for a project of this site. It will be even faster with cache, obviously.
I didn't look too closely, but I'm reasonably sure that can fixed so the total build time will be 30-40 seconds without to much effort; it looks like it's just generating the same HTML every time, so just caching it on the first generation seems like the obvious way.
And sure, Gojekyll is loads faster and more forgiving of these kind of inefficiencies, but "a few minutes" still seems pretty slow, especially considering even plain Jekyll can do it significantly faster.