The best evidence for this is a picture(1) from page 6 of the paper. Look at the second row. The building generated by 'mind reading' subject 2 and 4 look strikingly similar, but not very similar to the ground truth! From manually combing through the training dataset, I found a picture of a building that does look like that, and by scaling it down and cropping it exactly in the middle, it overlays rather closely(2) on the output that was ostensibly generated for an unrelated image.
If so, at most they found that looking at similar subjects light up similar regions of the brain, putting Stable Diffusion on top of it serves no purpose. At worst it's entirely cherry-picked coincidences.
I think the confusion is that this model is generating “teddy bear” internally, not a photo of a teddy bear. I.e. the diffusion part was added for flair, not to generate the details of the images that exist inside your mind. They could just as easily have run print(“teddy bear”), but they’re sending it to diffusion instead of printing it to console.
The fact that it can correctly discern between a dozen different outputs is pretty remarkable. And that’s all that this is showing. But that’s enough.
It’s not really a “gotcha” to say that it’s showing an image from the training set. They could have replaced diffusion with showing a static image of a teddy bear.
It sounds like this is many readers’ first time confronting the fact that scientists need to do these kinds of projects to get funding. As long as they’re not being intentionally deceptive, it seems fine. There’s a line between this and that ridiculous “rat brain flies plane” myth, and this seems above it.
Disclaimer: I should probably read the paper in detail before posting this, but the criticism of “the building looks like a training image” is mostly what I’m responding to. There are only so many topics one can think about, and having a machine draw a dog when I’m thinking about my dog Pip is some next-level sci-fi “we live in the future” stuff. Even if it doesn’t look like Pip, does it really matter?
Besides, it’s a matter of time till they correlate which parts of the brain are more prone to activating for specific details of the image you’re thinking about. Getting pose and color right would go a long way. So this is a resolution problem; we need more accurate brain sampling techniques, i.e. Neuralink. Then I’m sure diffusion will get a lot more of those details correct.
Even if we do a massive goalpost-move and grant that the system is only identifying the label "dog" with a brain scan of a person looking at a dog, we would need to see actual statistics of its labelling accuracy before judging it in that way. If the images in the paper are cherry-picked(1), it could easily be only able to extract a handful of bits to no bits at all, and the entire thing could very well turn out the be replicable from random noise.
(1) Note that the paper even states "We generated five images for each test image and selected the generated images with highest PSMs [perceptual similarity metrics].", so it even directly admits that the presented images are cherry-picked at least once.
If you train a model where the input is an integer between 1 and 10, and the output is a specific image from a set of ten, the model will be able to get zero loss on the task. That is what's happening here.
It means there may be signal in the noise. Even if it's overfitting. Which makes sense.
A sufficiently granular map of the human brain aught to be readable, if you know what the input and output signals are.
Theoretically, the results would scale to more training images... we just need to fMRI all of LAION-5B. Easy peasy.
I don't think you got that 2770 correct. Might be 9250 images, minus 982 (that one you got right). Then again, the paper is so badly written, I find it difficult to decipher what they did. From section 3.1:
Briefly, NSD provides data acquired from a 7-Tesla fMRI scanner over 30–40 sessions during which each subject viewed three repetitions of 10,000 images. We analyzed data for four of the eight subjects who completed all imaging sessions (subj01, subj02, subj05, and subj07).
We used 27,750 trials from NSD for each subject (2,250 trials out of the total 30,000 trials were not publicly released by NSD). For a subset of those trials (N=2,770 trials), 982 images were viewed by all four subjects. Those trials were used as the test dataset, while the remaining trials (N=24,980) were used as the training dataset.
https://www.biorxiv.org/content/10.1101/2022.11.18.517004v2....
The only training required in our method is to con-
struct linear models that map fMRI signals to each LDM
component, and no training or fine-tuning of deep-learning
models is needed.
...
To construct models from fMRI to the components of
LDM, we used L2-regularized linear regression, and all
models were built on a per subject basis. Weights were
estimated from training data, and regularization parame-
ters were explored during the training using 5-fold cross-
validation.Either way, if I'm understanding right, it's very impressive. If the only input to the model (after training) is a fMRI reading, and from that it can reconstruct an image, at the very least that shows it can strongly correlate brain patterns back to the original image.
It'd be even cooler (and scarier?) if it works for novel images. I wonder what the output would look like for an image the model had never seen before? Would a person looking at a clock produce a roughly clock-like image, or would it be noise?
All the usual skepticism to these models applies, of course. They are very good at hallucinating, and we are very good at applying our own meaning to their hallucinations.
Edit: found it! https://youtu.be/nsjDnYxJ0bo
Anyways, the images that were depicted in this work of fiction shot in 1990 about "the future" of 2000, had a very interesting look to them-- kind of distorted and dreamy like the images in the paper.
Are the images in the paper just a case of overfitting? ¯\_(ツ)_/¯ but it still makes me giddy remembering the Wim Wenders film.
The human mind is considered the only place where we have true privacy. All these efforts are taking that away.
At this rate all notions of privacy will soon be dead.
I'm surprised it didn't seem to go anywhere.
Edit: found it https://youtu.be/RuUSc53Xpeg
> What exactly is “silent speech”? Does the user have to move his or her face or mouth to use the system?
> Silent speech is different from either thinking of words or saying words out loud. Remember when you first learned to read? At first, you spoke the words you read out loud, but then you learned to voice them internally and silently. In order to then proceed to faster reading rates, you had to unlearn the “silent speaking” of the words you read. Silent speaking is a conscious effort to say a word, characterized by subtle movements of internal speech organs without actually voicing it.
> Can this device read my mind? What about privacy?
> No, this device cannot read your mind. The novelty of this system is that it reads signals from your facial and vocal cord muscles when you intentionally and silently voice words. The system does not have any direct and physical access to brain activity, and therefore cannot read a user's thoughts. It is crucial that the control over input resides absolutely with the user in all situations, and that such an interface not have access to a user's thoughts. The device only reads words that are deliberately silently spoken as inputs.
https://www.media.mit.edu/projects/alterego/frequently-asked...
Would be useful if I lost the ability to write or speak, for whatever reason.
We are a long way away from worrying about this.
Cheap cameras everywhere on the other hand...
No, the delusional shortsighted and revenue-driven SV startup culture doesn't give a shit about such 'technophobic trivialities'.
Graduate School of Frontier Biosciences, Osaka University, Japan
Of course, but what's to be done about it? Should we outlaw research like this?
It’s not hard to imagine some really terrible ways this can be used.
A bit of preemptive legislation might be wise as AI is advancing so rapidly.
Of course, these "advances" will be praised greatly in MSM as providing great benefits for mutes, "harmonious society", and whatever else happens to be the virtue-signaling fad of the moment.
I could imagine improvements since then,especially with advances in image networks.
1. https://www.cell.com/current-biology/fulltext/S0960-9822(11)...
2. https://www.youtube.com/watch?v=nsjDnYxJ0bo
Edit: Just realized the paper above is also from Shinji Nishimoto
What's astonishing here is the quality of reconstruction. But I have not seen this research referenced a lot. Does someone how /why the reconstruction from monkey brain looks so perfect while we don't have anything close from human brain?
Edit: better images here https://www.newscientist.com/article/2133343-photos-of-human...
Oh don't worry, this will get wrapped up in some pseudoscience bullshit and misleading statistics and marketed to law enforcement. But not to worry, at first it'll only be used on real bad criminals. If you have nothing to hide you have nothing to fear!
I suspect others are taking the wide view but I wanted to point out this direct answer to your question.
p.s the use of "torture" is intended in contrast to the neutrality of clinical language on this topic, not as a hint at my judgement on the matter.
Here's one deontological perspective you could take:
It is always wrong to cause unecessary suffering to others.
Now, the subjective traits to be considered here are "unecessary" and "suffering."
It used to be a common belief that animals lacked the capacity to suffer as humans do. They could feel pain, nocioception sure, but whether it caused complex psychological suffering (torment) used to be contenious.
Today, this is certainly not contentious for our closest relatives. All primates possess a theory of mind, long memory, emotional states, complex social behavior like lasting bonds and altruism, and other traits necessary to suffer in significant ways.
So the focus (as long as we are considering primates, obviously nobody cares about model species like drosophilia as they have a greatly dimished capacity to suffer (edit: although I should mention that ranking things based on capacity to suffer leads to pretty awful territories too. Eg, if a human is severely mentally disabled, is more permissible to experiment on them? I think most of us would say no, which raises the question why it's okay to do so for other species)) shifts to whether causing them immense physical pain and torture is necessary.
And this is where I think things get pretty murky, and I will leave the rest up to you! I wish more people were curious about moral philosophy and creating their own consistent ethical framework for the world... I think it's especially important in science and engineering.
Of course, you could use a different model like utilitarianism, but utilitarianism still requires some level of deontological principles or you end up with a pretty extremist moral philosophy (same goes for just having Kantian deontology with no room for utilitarianism, IMO).
edit 2: come to think of it, Jainists would certainly have an issue with experimentation even on drosophilia, so I take that back that "nobody" cares. IIRC they even have a mouth covering to prevent swallowing and killing any insects that might accidentally fly into their mouths, as well as a specialized stick to gently move things like spiders and other insects out of the way. I know most people would scoff at that, but I think their deep respect for all forms of life is beautiful.
even if this is just another bullshit article, im just making a point related to it. people need to be worried about this. for the first time in history, lots of people are now creeped out by AI. but they arent taking action or demanding change. we need regulation, grass-roots efforts to stop AI. even if the only way humanity could abort AI as a concept, or delay it for a significant amount of time, was to return to the iron age, and it certainly isnt the only way, it would be unambiguously worth it, in every way and from every angle.
AI requires large compute. what we are doing now was impossible just 20 years ago. if not 20 then 30. you cant manufacture that kind of compute in your garage. global regulation would take care of it no problem. at the very least it would buy us an enormous amount of time that we could use to figure something else out. people always say that some hold-out country would defy global regulations. they wouldnt defy NATO, let alone a super-global coalition. and the idea of such a group or NATO enforcing compute regulations is not far-fetched whatsoever because the emergence of AGI or even advanced non-AGI goes against the interests of literally every human being. there is no group of humans that benefit from that ultimately. the problem is simply waking people up to this plain fact.
> we need regulation
No, we don't. Regulation doesn't stop technological progress - it puts it in the hands of an elite few. And besides, there are 130+ regulatory jurisdictions. For example, the US government doesn't fund human cloning research, but that doesn't mean China won't fund it. Or perhaps you'd also like a one world government that can jail anyone doing wrongthink on their GPU?
Personally, I hope we get AGI (in the most Kurzweillian sense) as soon as possible. It will lead to a cambrian explosion of advancements across all fields of science. This is our best chance of cracking the secrets of the universe and answering fundamental questions, like whether FTL interstellar travel is truly impossible, or whether aging is really irreversible.
Imagine an intelligence unencumbered by the "technical debt" we've accrued over centuries of building our scientific model of the world. AGI could simulate infinitely many novel paths through the "tech tree" of human history, replaying our scientific discoveries and trying different assumptions. What if we had 12 fingers and mathematics started from a base-12 system? What if we could see in infrared? We would have followed entirely different scientific paths; AGI will be able to find what we missed.
someone once said "do not see things as they are, see them as they might be." this quote is really about discoveries and the tendency of humans to only see things in terms of what already exists. and the implication is that humans have a big blind spot for seeing whats next. thats why we need a motivational quote to help us to see things as they might be rather than simply as they are. it is true that there has never been a global regulation or ban like the one we are talking about except maybe ozone layer emissions. but by this same metric, AGI can never exist because it has never existed before. its a silly response and a complete waste of time. even if such regulations already existed for other things in the past, you still would be here saying it was impossible but for some other reason. the key here is to make up your mind last, not first.
regulation can stop anything as long as it doesnt break the laws of physics. and, if you had read my comment, i explain why china wouldnt pursue AGI. even if china did pusue AGI, they probably wouldnt be able to crack it. none of the major breakthroughs have come out of china.
"i hope we get AGI as soon as possible. [it will lead to many incredible things]." you have no idea what AGI will lead to. you just cherry pick all the cool stuff that would be possible but totally ignore all of the other implications. there would be an immediate and total power vacuum caused by the advancements. these advancements would be so huge that it would change the geopolitical equation beyond recognition. the concept of a country would probably be economically and geopolitically untenable. there would have to be a transition to an entirely new order where the dominant meta-organisms arent countries but some bizarre AGI conglomerate that looks like an expressionist painting in comparison to what we have now. the transition to this new world, whatever it looks like, would involve war. probably the biggest war that has ever happened. this is intrinsic and unavoidable. it cannot be disproved or denied. the fundamental economic and geopolitical equation that underlies the current equilibrium would change suddenly and violently.
the current world order will disappear, you will probably lose everything you own and everyone you love along with your country. a global war will break out where there is a high chance that all established rules of engagement are ignored. weapons or methods that render the environment unlivable to humans will more than likely be used because the dominant organisms and meta-organisms wont need humans in any practical sense. and after the dust settles and a new equilibrium is reached, the existence of humans will end very quickly (if it hadnt already) because we will offer nothing of value anymore and if our existence presents the slightest inconvenience to the machines, they will allow us to die. and that is just the scenario where they are apathetic towards us. i have not even begun to discuss the repulsive, grotesque nature of our suffering if we ever are the subject of AGI malice. those possibilities are always brushed aside as fear mongering so i dont even bring them up. but they should play into our decision to move forward or not.
at the very best, we will somehow manage to attach ourselves as parasites to the new machine meta-organisms and experience an existence with no agency or purpose other than to ogle at the machines. but that wont happen because the machines will immediately embark on doing things that humans could never, ever understand.
"what if we had 12 fingers [...]." what if indeed. perhaps i was too hasty... no cost is too high in pursuing the deeper mysteries of the universe.
Hi everybody! We’re Joe and Ahmed and super thrilled to be launching Human Diffusion today! We’ve built an exciting new image generation system that supports economies in developing nations.
Our product leverages the latent creativity of humanity by directly fitting employees with fMRI rigs and presenting them with text inquiries through our API (JavaScript SDK available, Python soon!). Unlike competing alternatives we preserve human jobs in an era of AI supremacy.
I’d like to address rumors that our facilities amount to slaving brains to machines. This is a gross misunderstanding of the benefits we offer to our staff - they are family. Our 18 hour shifts are finely calibrated based on feedback collected through our API, and any suggestion of exploitation is flatly untrue.
Send us an email (satire@humandiffusion.com) to get early access.
Please continue. /s Governments, three letter agencies and the like would be absolutely excited to see this. The future that no-one has asked for.