From a technical perspective, how would one structure a program like this? Where the main developers also use the same 'engine' to develop the product itself, while also exposing the same engine to the third parties?

The first such case I have seen was Maya (the 3d software) when it first came out use a custom scripting language called MEL(Maya Embedded Language) to develop the entire user interface and corresponding actual entity manipulations. Third party plugin developers also use the same MEL.

I thought wow, and that is still impressive today.

I'm in agreement, but I think you still need to build tooling around it.[1] At the very minimum:

1. The code editor needs to be part of the system - pointing the customer at VS Code is a dealbreaker.

2. Debugger needs to be provided: you can't ask the customer to hit F12 and learn how to use the browser's devtools.

3. Documentation is still needed: At least one folder of samples/examples, and at least a single exhaustive reference of all the functions provided by system.

Finally, I wouldn't go with Javascript: Non-developers are less-likely to understand something as complicated as Javascript.

You're using a language that has subtle and hard to find bugs if variables are declared in one of three ways, when a different way is more appropriate. Same with the two different ways to define functions. Same with the difference between all the forms of a for loop. Same with the distinction between a prototyped object and a class. Same for the usage of instance variables vs class variables.

Add typescript support to the mix and you may as well not even bother looking for customers.

[1] I just created a system that uses a vanilla webcomponent for client-side includes of HTML snippets, using only HTML. It's not possible to now set breakpoints in any `script` element that is included using this webcomponent, e.g. `<my-include remote-src=./common-stuff.html>` where `common-stuff.html` has script tags.