Code should be easy to get started with for anyone familiar with the JS ecosystem.
Questions for the HN community: I spent 20-25% of the entire time just setting up the project and fighting issues with the JavaScript ecosystem. This experience has made me consider learning Cpp, Metal, and XCode. Has anyone made the transition from WebGL/TS to Cpp or done it the other way around? What was your experience with it? And what about debugging? That's a big issue with WebGL.
As for now, I'm thinking about picking up WebGPU next because it has an up-and-coming debugger made by Brendan Duncan[0] that looks promising.
Edit: Anyone know why MRI data is sent via CDs and not the web? I started working on this project specifically because some people close to me got MRI scans and received CDs of their results. I know that some MRI data can be huge so downloading over the web doesn't make sense, but anything under 64MB should be sent over the web, no? For reference, I believe most MRI data can be under 250MB[1].
[0] https://github.com/brendan-duncan/webgpu_inspector
[1] https://www.researchgate.net/figure/Typical-image-dimensions...
I recently tried Metal a bit. Some features are only available for visionOS (the headset) and not iOS or macOS and vice versa.
Here are some questions that came up:
- Do I use Next.js or Vite.js or CRA (is that still used these days)
- With Vite.js do I pick React or go with TypeScript template?
- But what if I don't want to write GLSL files in raw text, and use GLSL file extension?
- Oh now you want to import GLSL files directly, well you can't do that
- Oh, I needed to install a Vite plugin but then TS throws errors saying it can't read files that end with .glsl
- Wait is my TS server running? No, I thought this always starts up in VS Code
- Okay so I can't even find the button to restart TS server, so now I need to... okay figured that out
- That still didn't fix it the TS errors
- After some more research, ah so you can import text files like this a.glsl?raw extensionsounds exactly like every other dev pushing JS libraries. that's not necessarily a bad thing, but it does get messy on the bleeding edge of things
E.g. Vite is handy for bootstrapping and hot reloading (don't really need it for build anymore given remote imports, etc). Do you really need typescript? If it saves you time in the long-term, sure add it in (it requires a build step). If you are building a big UI then OK - add a CSS / JS framework. But best bet is always to keep complexity down to a minimum. Last thing you want to be doing is debugging / working around peculiarities of other people's code.
I tend to demo in a single HTML file (you can even embed your shaders in the HTLML!), which can then be easily shared / hosted.
On the demo. Not sure if you referenced it, but I remember being wowed by a similar (official) demo back when Three was first rolling out WebGL2 support. The first commit I could find was 2018, but I'm sure it was actually earlier than that - might even have been a pre-release demo (I'm certain cross browser support wasn't there when I tried it).
https://threejs.org/examples/webgl2_materials_texture3d.html
They used the "NRRD" format, which I haven't heard of (but sounds pretty close to your 1x1x1 raw files ... x-array style thing). Did you look at this approach? Have things moved on since?
Edit: maybe the real tutorial here is around pre-processing of common volume data formats for use with e.g. three / webgl2
What you have produced is wonderful, but has nothing to do with the web. That’s like comparing apples and oranges.
The web was built for images. All the glsl you have is rendering to that image. Cpp or whatever, your problem domain is pixel shader. A program, run on a gpu, for a single pixel and ray marching through a dataset on also on a gpu. It’s a hardware problem why debugging is so hard. Unfortunately, this is the reality of graphics programming in general.
Vite, React, Typescript, etc etc are all faff exterior to what you are doing.
If you want debugging tools and a good graphics programming experience, and a simple view with some interactivity something like Unitt, Godot, TouchDesigner or if you want lower level, OpenFrameworks, Processing, etc. would be leaner.
The medical industry moves slowly. Medical data is often covered by HIPAA, which is why it is often not available easily on the web. Using CDs is just an artifact of their slow movement through technology.
Fair point about the health tech industry.
You are really asking, “why does medical imaging not have budgets for software?” It faces all the same monetization challenges as all business to business software, that tends to, unfortunately, disfavor operationally cheap and performant solutions.
I really like how you organized your project. As someone who's developed with Three.js for years both imperatively and in a highly-declarative way using R3F I'm always interested to see different approaches to architecture/organization.
The most important reason for my code structure is that I just like writing GLSL shaders! It's annoying that I need to set up Vite, TypeScript, Tailwind, React, etc. all so that I can write some shaders. I know I can write shaders in ShaderToy, but then I won't be able to upload custom 3D texture files or add simple user controls.
Maybe check out https://cables.gl
That's what the frontend hype canal wants you to think!
Your frontend JS could just `fetch` the GLSL files from the backend instead of trying to compile them into your build process.
There are tradeoffs to this of course, but dealing with the complexity death star of JS tooling can be opted out of.
It just really irked me that people got CDs instead of being fully digital. The last time I used a CD (besides my PS4) was 10+ years ago. I don't even have a CD player anymore. I'm sure Gen Alpha will look at CDs the way I look at floppy disks.
Working in WebGL/JS is nice since you can deploy it everywhere, but it can be really hard for graphics programming as you've found because there are very few tools for doing real GPU/graphics debugging for WebGL. The only one I know of is [1], and I've had limited success with it.
WebGPU is a great next step, it provides a modern GPU API (so if you want to learn Metal, DX12, Vulkan, they're more familiar), and modern GPU functionality like storage buffers and compute shaders, not to mention lower overhead and better performance. The WebGPU inspector [2] also looks to provide a GPU profiler/debugger for web that aims to be on par with native options. I just tried it out on a small project I have and it looks really useful. Another benefit of WebGPU is that it maps more clearly to Metal/DX12/Vulkan, so you can use native tools to profile it through Chrome [3].
I think it would be worth learning C++ and a native graphics API, you'll get access to the much more powerful graphics debugging & profiling features provided by native tools (PIX, RenderDoc, Nvidia Nsight, Xcode, etc.) and functionality beyond what even WebGPU exposes.
Personally, I have come "full circle": I started with C++ and OpenGL, then DX12/Vulkan/Metal, then started doing more WebGL/WebGPU and JS/TS to "run everywhere", and now I'm back writing C++ but using WebGL/WebGPU and compiling to WebAssembly to still run everywhere (and native for tools).
With WebGPU, you could program in C++ (or Rust) and compile to both native (for access to debuggers and tools), and Wasm (for wide deployment on the web). This is one of the aspects of WebGPU that is most exciting to me. There's a great tutorial on developing WebGPU w/ C++ [4], and a one on using it from JS/TS [5].
[0] https://www.willusher.io/webgl/2019/01/13/volume-rendering-w...
[1] https://spector.babylonjs.com/
[2] https://github.com/brendan-duncan/webgpu_inspector
[3] https://toji.dev/webgpu-profiling/pix
The funny thing is that I was getting more confident about using JS + WebGL/WebGPU ecosystem for graphics programming after having read your posts. Very interesting to hear that you've come full circle back to Cpp + WebGL/WebGPU + WebAssembly. I'll look more closely to assess options as I head down this journey. Thank you for your tips and advice!
Edit: Perhaps you'd find my "What is WebGPU" video on YouTube interesting. I'd love to get it fact-checked by someone who's been doing WebGl/WebGPU way longer than most people! I only got into this field ~2 years ago.
There are some tradeoffs w/ WebAssembly as well (not sharing the same memory as JS/TS is the biggest one) and debugging can be a bit tough as well though now there's a good VSCode plugin for it [0]. Another part of the reason I also moved back to C++ -> Wasm was for the performance improvement from Wasm vs. JS/TS, but the cross compilation to native/web was the main motivator.
[0] https://marketplace.visualstudio.com/items?itemName=ms-vscod...
I was provided MRI DICOM files a month ago via the radiology office's used SaaS platform, Sectra PatientPortal.
After the MRI session I was given a login code. Back home, I could use that code on the Sectra PatientPortal.
It provides a very basic DICOM image web viewer (contrast/dynamic range modification and slice scrolling, four windows), albeit 2D only, and download of the DICOM files as well as PNG exports.
I am looking for a web-based or Linux-based FLOSS DICOM viewer, but couldn't find any professional program so far.
Have you looked into Slicer3D[0] which is a multi-platform desktop app or Open Health Image Foundations dicom viewer[1] which is web-based? Perhaps one of these will help.
Also I love your videos. The internet is a smaller place than it seems.
Related - a while I made a process to convert DICOM / MRI files to voxels for printing at the native resolution of 3D printers [0]. It means you can preserve all the fine detail of the scans, rather than converting to a mesh (segmentation).
On the CD question - it's probably because there is little incentive to build a secure / cross platform solution for patients to access their scans. The CD model is very outdated, but does work, and there is no need for HIPAA compliance even though a CD isn't very secure.
Oh wow, your project looks really cool too! I'll check it out.
Interesting. I did not know that CDs did not need to get HIPAA compliance. I suppose web apps would certainly need that. I wonder what will happen once Gen Alpha starts needing to get MRI scans.
At first I thought you‘ve been using ImGUI as your interface looks similar.
Maybe that‘s a good starting point for your CPP career.
I've heard good things about ImGUI so I'll have to check that out if I ever go down the Cpp route.
Here is my summary of it[1] and the original paper[2] I found this in
[1] https://azeemba.com/posts/homomorphic-encryption-with-images...
The volume rendering engine I have been working on uses a histogram for the value distribution, and on top of it, one can draw lines that indicate the opacity. Additionally, one can set colors to the control points, which are then linearly interpolated for the given ranges.