Web Stable Diffusion (opens in new tab)

hm. It seems it's been in firefox, pref'd off by default, for a few years now.. https://hacks.mozilla.org/2020/04/experimental-webgpu-in-fir...

Webgpu will ship this year, so it will be more widely available pretty soon

There is an origin trial. They should enable that, then it would work in Chrome stable today. It's only supported on Windows and Mac right now though, I think.

WebGPU comes out April 26th - then 65% of the worlds web browsers will have access to it.

Yes! Something like ControlNet is great for this. I use this[1] API on Replicate specifically because it offers the various methods (depth maps, edge detection, etc.)

Replicate sometimes give you free use (I forget if this model does), but if you pay then an image output will cost you about one cent.

Give it your image, write a prompt (something like "a modern living room"), choose your ControlNet model from the drop-down, and submit.

If you choose depth map, for example, it will generate its best guess depth map for your image and use that to steer the Stable Diffusion output. It's fascinating, and a lot of fun to play with.

[1] https://replicate.com/jagilley/controlnet

Edit: I'd love to see what you produce, if you do use it.

ControlNet maybe? https://github.com/lllyasviel/ControlNet

At huggingface: https://huggingface.co/spaces/hysts/ControlNet

Play around with the different models, you might get better results with some vs others.

The existing tooling around Stable Diffusion very often features either a style transfer tool or even have a an input image to be given together with the text prompt.

For example: https://nmkd.itch.io/t2i-gui

You can also try importing your CAD model into Blender, assigning some materials to the different objects and getting renders there, though tuning the render may take more time than tuning a prompt.

Thanks for your interest! Most of the existing stable diffusion demos rely on a server behind to run the image generation. It means you need to host your own GPU server to support these workloads. It is hard to have the demo run purely on web browser, because stable diffusion usually has heavy computation and memory consumption.

The web stable diffusion directly puts stable diffusion model in your browser, and it runs directly through client GPU on users’ laptop. This means there is no queueing time for the server’s response. It also means more opportunities for client server co-optimizations, since essentially the “client” and “server” are both the single laptop. The web stable diffusion is also friendly for personalization and privacy. Since everything runs only on the client side and no interaction with server is needed, you can imagine to have your own style stable diffusion deployed and demonstrated on the web without sharing the model to anyone else, and you can also run with personalized model input (e.g., the text prompt in this case) without letting others know.

Thanks for your interest again! And we are happy to hear your feedback on your experiences and the functionalities you would like us to add in the future.

Usually Stable Diffusion (and most machine learning models) run on a server with the web front end (eg A1111's SD web ui) just providing a user interface. Even if you download it, you need to run the server on your computer. When you make a request to it you are using the CPU/GPU on the server.

The linked version runs a version of the stable diffusion model in the webbrowser, so it uses your own CPU (and in the case GPU) via the APIs provided by the web browser. This specific implementation uses an API called WebGPU which isn't yet widely supported.

Yep it surprisingly works on my AMDGPU too, even if it’s designed only for M1/M2

Yes of course. Optimizing and building the model to the format acceptable by ONNX web runtime will getting this in. On the other hand, we also need to enhance our own runtime (for example for better memory pool management) in the future.

Same on Windows 11, 5800x3d with Vega64, PortableApps Chrome Canary.

17s DPM (20 steps), 41s PNDM (50 steps)

Ouch, torchlib doesn't have a webgpu target though

(not the author) This is taking the model and running it entirely in the browser via web assembly and using webGPU, meaning the browser is executing compiled code directly, inside its own process. This is different to the 'web ui' implementations like Auto1111 which are just a website front-end for a python script that is running a server and running the model in a background process on your computer. It's a very different type of implementation.