Mozilla and later Firefox were browsers that were easily extensible in every way. Given experience with the XUL/XPCOM/JS stack, you can still pretty much do anything you want without too much difficulty. However, three things have happened since then:

1) No one outside of Mozilla adopted XUL/XPCOM/JS as an application platform. There are probably many reasons for this, but the main one, based on my recollection of early Mozilla, was that XUL was too slow and didn't look native, and JS was likewise too slow and not considered to be a real programming language. That has changed significantly since Mozilla's introduction, as both the layout and JS engines have improved substantially, but that was the picture at its introduction. In time, JS was revitalized, but so was HTML. HTML has now achieved some success as an application platform (e.g. Atom and Spotify), but XUL remains basically unused outside of Mozilla.

2) Because no one adopted XUL/XPCOM/JS, Mozilla stopped looking at it as an application platform and started looking at it as a legacy technology that they at one time used to build a browser. As a result, it began to stagnate, and interest in documenting things so that other people outside of Mozilla could use them waned. And because XUL/XPCOM/JS are now niche technologies, no one knows them and there's no value in learning them outside of being able to write Firefox extensions, so Firefox does not seem so easily extensible to outsiders.

3) Chrome happened, and Mozilla saw what Chrome gained by not building a fully extensible platform and wants that for itself. On the flip side, I'm not sure that Mozilla fully recognizes what it currently gains from having an extensible platform.