If there’s some platonic notion of divinity or immanence that all faith is just a downward projection from, it seems like its statistical representation in tokenized embedding vectors is about as close as you could get to understanding it holistically across theological boundaries.
All kidding aside, whether you are looking at Markov chain n-gram babble or high temperature LLM inference, the strange things that emerge are a wonderful form of glossolalia in my opinion that speak to some strange essence embedded in the collective space created by the sum of their corpi text. The Delphic oracle is real, and you can subscribe for a low fee of $20/month!
I've had some surprisingly insightful tarot readings with the assistance of ChatGPT and Claude. I use tarot for introspection rather than divination, and it turns out LLMs are extremely good at providing a sounding board to mirror and understand those insights.
https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...
Now I have friar LLM or whatever you want to call it. I ask it to tell me the creation story of the universe according to our bespoke hidden / forbidden religion (to prevent drawing too heavily on established religions) and then edit this to suit my needs. Then I provide a few mantras which I've written in the past / already do like, and ask it tell me the rest of the mantras. Voila, I've got a large set of mantras, for just about any occasion, that help with a variety of things such as calming me during times of uncertainty, emotional turmoil from interpersonal issues etc.
I then keep the creation story, mantras, etc. as a "history" of sorts, and when I need to discuss matters of spiritual or novel psychological issues, this along with the system prompt create the basis for a pretty useful/functional spiritual guide which has helped me grow from things like ending a 10 year relationship, moving to a new city without friends or support systems, dealing with potential issues arising from my age, new found singledom, and potentially missing the window for having kids etc.
It's been great, I find myself quite at peace with my lot in life without losing the drive to achieve more, which I find is a delicate balance to achieve. All in all I rate it highly, it is cheaper than therapy ever was and broadly speaking more useful since I'm not being handed pointless diagnoses (pointless because typically these diagnosis pathologise a cluster of traits that aren't inherently problematic without offering up any solutions)
It is so deep.
What I'm working on https://reminder.dev
In one instance I was trying to remember if it was in the Bible or the Quran where, in the story of Abraham, the pagans are asked why they believe what they believe and they respond with "because our fathers believed" and the scripture critiqued this. ChatGPT gave me the exact verses from the Quran while Google would bring up random unrelated forum posts.
It's also good for comparing religious texts and seeing where stories differ.
> How can the Qur'an acknowledge the Torah (Tawrat), Psalms (Zabur), and the Gospel (Injil) as divine revelations from God (Allah) yet deny the deity of Jesus Christ. Don't these contradictions falsify the truth of the text?
It gave the classic counter argument:
> A: The Qur'an acknowledges the Torah, Psalms, and Gospel as divine revelations from God, emphasizing their role as guidance for their respective communities. However, it also teaches that these scriptures were either altered or misinterpreted over time.
Of course, the Qu'ran makes these theological assertions rather than one grounded in concrete historical or manuscript evidence. Nevertheless, your app does a good job at fairly representing the beliefs of the Qur'an.
I have a feedback. The web pages Hadith and Quran are not mobile friendly in the sense that you are loading the whole content in one page. Separating into pages with number or doing lazy loading will be better.
In my opinion, based on verses 17:36, 5:118, 46:15, 96:15-16 and 17:13-15 of the Quran viz "...And We have fastened every man's deeds to his neck, and on the Day of Resurrection, We shall bring out for him a book which he will find wide open. [It will be said], Read your record. You yourself are sufficient as a reckoner against you this Day. Whoever is guided is only guided for [the benefit of] his soul. And whoever errs only errs against it. And no bearer of burdens will bear the burden of another...", it is solely dishonesty and wilfully committing injustice that will cause a person to judge against himself/herself and earn a place in Hell. So it is not necessary that only 'muslims' can enter Heaven or be Successful. The criterion is the application of sense/reasoning and not being swayed by that which is convenient (being born into a religion, peer pressure, personal convenience, prejudice, etc.). In my opinion, i believe that fate is influenced by how steadfastly a person holds on to Reason and Justice. So prejudiced people end up committing deeds that will lead them to Hell like willful falsification and obtuseness. So even 'muslims' who choose the path of hypocrisy can end up in the lowest layers of Hell (Quran 4:145) and non-muslims can also go to Heaven (Quran 5:118). Anybody can understand this, because it makes sense. Sadly in the case of Abrahamic religions, the exegetic literature like Midrash and Tafsir muddy that which is clear. How ironic? But then again, there is a pertinent verse (3:7) in the Quran in this regard: "He it is Who hath revealed unto thee the Scripture wherein are clear revelations - they are the substance of the Book - and others (which are) allegorical. As for those in whose hearts is deviation [from truth], they will follow that of it which is unspecific, seeking discord and seeking an interpretation [suitable to them]..." There is a thin book at https://archive.org/details/TheAmazingQuran.pdf by a mathematician not of muslim origin where he examines the Quran from a logical standpoint which can be a starting point followed by the papers at https://scholar.google.co.uk/citations?user=wmRDl-4AAAAJ on computational analysis of the Quran (including with neural networks) before the ChatGPT era.
Think of it like how you have a JPEG, you can compress it further and further, but eventually you lose the ability to understand the original image.
With the models, if they had infinite size, I imagine they could recall values in their training data extremely accurately. But as you compress down further and further to smaller and smaller models, you are trying to distil the same amount of information in less space, and so things cannot be perfectly recalled.
We have people tuning these smaller models to squeeze every ounce of what we consider meaningful out of them (passing certain benchmarks, seeming coherent in dialogue, etc), but in the process the things we don't tune them for, i.e. accurate recall of scripture (not that we should) they lose that ability.
I will say with all of that though, that I only have a high level understanding of LLMs, I've integrated them into products on the job, but I am by no means an ML engineer.
I never tell other people what to believe or how they should do that in any capacity.
With that said I find the hallucination component here fascinating. From my perspective everyone who interprets various religious text does so differently and usually that involves varying levels of fabrication or something that looks a lot like it. I'm speaking about the "talking in tongues" and other methods here. I'm not trying to lump all religions into the same bag here, but I have seen that a lot have different ways of "receiving" communication or directive. To me this seems pretty consistent with the colloquial idea of a hallucination.
The existence of denominations and confessions/creeds really shows that this isn't true generally. (There may be more than one interpretation, but not a unique one to every reader)
Even ignoring denominations, nearly all mainline christians for example would agree to the Nicene creed. (Anyone that disagreed probably wouldn't be considered "mainline", so somewhat definitional)
To suggest that all of theology is basically noneeterministically making things is naive and in my opinion insulting to an entire academic discipline, much less to the entire body of believers. (I can't tell if this is what you're taking about or not)
Nearly no group of mainline believers accept speaking in tongues and basically all of main protestantism believes that the time of prophets and new messages from God is over, the Bible is complete and will never be added to. (Pentecostals would be the one exception here, but I don't consider them mainline christians personally)
Your experience sounds limited to Pentecostal-originated churches, which are 100-150 years old. In those churches, it’s acceptable to speak as if you’ve received a spontaneous understanding of the Bible and to not explain it. That does have a parallel to LLM hallucinations in face value output, I suppose, but the origination is completely different as the spontaneous human is making planned remarks passed off as spontaneous, trying to affect specific people in the room, or emotionally overwhelmed. None of those resemble why/how LLMs hallucinate.
Improv is a thing, and can be trained as a skill even outside of comedy/entertainment.
Though, outside of Charismatic sects, Christianity does see a more reasonable level of "I had prepared by thinking about (verse X), but suddenly now I'm thinking about (obscure verse Y)."
Without specifying any specific religions, many often have different interpretations at different times on critical issues like human rights, torture, slavery and worse. Different interpretations at different times have been used to justify spousal abuse, human selling/trafficking for specific agreed upon prices or transactions, etc.
I think in the same way that LLMs hallucinate because they can't find a proper place in their vector mapping for that embedding to draw on experience or reason (simplifying) the humans are also drawing from a place that has blanks and filling in the gaps.
God's main requirement is to put faith in Christ and repent to enter a relationship with Him. His friends get forgiveness and grace, enemies get justice and wrath. From there, He dwells inside us to change us into what He wants. He rewards every good work He equips us for. He also disciplines our failures, like adopted sons and daughters, to keep us on the right path. He is gracious.
While telling that redemptive story, God's Word weaves together much testimony to teach us almost everything we need to know about life: God's/man's nature, God's laws/design, repeating patterns of man's behavior, different genres, prophecy/miracles for confirmation, and promises for the future. That the same message got the same results in thousands of people groups, peacefully, shows its universal power.
You wondered how to interpret it in an accurate way. Our church follows the historical-grammatical, or literal, method. We ask: who is speaking, what do their specific words mean, in what context, and for what reason then? And how does that apply today? And what do other passages say about the same topic in their context?
https://www.givethemlife.com/studying/originalcontext.html
I'll leave you with that site in case it helps answer some of your questions. It's really the work Christ did that saves us, changes us, etc. Our actions help us live more effectively while on Earth. Every decision has an impact in eternity, too, as God will render to each for their works.
Since He promises answers, I suggest reading John's Gospel in ESV (good translation) while asking who Jesus Christ really is. You have to be humble and open to hear Him, though.
With that said, all you said is that the process of transmitting religious ideas is akin to hallucination? Care to explain what the logical argument for that is?
This is playing against their strengths. By all means ask them for a summary, or some analysis, or textual comparison, but please, please stop treating LLMs as databases.
If so, what happens to their IP risks?
This article is yet another example of someone misunderstanding what an LLM is at a fundamental level. We are all collectively doing a bad job at explaining what LLMs are, and it's causing issues.
Only recently I was talking to someone who loves ChatGPT because it "takes into account everything I discuss with it", only, it doesn't. They think that it does because it's close-ish, but it's literally not at all doing a thing that they are relying upon it to do for their work.
> If you ask it to summarize (without feeding the entire bible), it needs to know the bible.
There's a difference between "knowing" the bible and its many translations/interpretations, and being able to reproduce them word for word. I would imagine most biblical scholars can produce better discourse on the bible than ChatGPT, but that few if any could reproduce exact verbatim content. I'm not arguing that testing ChatGPT's knowledge of the bible isn't valuable, I'm arguing that LLMs are the wrong tool for the job for verbatim reproduction, and testing that (and ignoring the actual knowledge) is a bad test, in the same way that asking students to regurgitate content verbatim is much less effective as a method of testing understanding than testing their ability to use that understanding.
Can you provide short excerpts from works in Latin and Greek written between 600 and 1300 that demonstrate the evolution over those centuries specifically of literary references to Jesus' miracle of the loaves and fishes?
For example, I looked up the quotation provided from Isidore of Seville's De fide catholica contra Iudaeos, Lib. II, cap. 19, using this copy on WikiSource, https://la.wikisource.org/wiki/De_fide_catholica_contra_Iuda.... The quote certainly does not appear under LIBER SECUNDUS, CAPUT XIX. Nor could I find it in whole or in fragment anywhere in the document, nor indeed any mention of the miracle of loaves and fishes (granted, I could have missed one, I relied on Ctrl+F and my very rusty Latin).
Perhaps the copy on WikiSource is incomplete, or perhaps there are differing manuscripts, but perhaps also the quote was a complete hallucination to begin with.
Rule of thumb: if you can't look up the original texts, you can assume they weren't actually in the training data. The training data is, however, likely to include a lot of people quoting those texts, meaning that the model predicts "SOURCE says OPEN QUOTATION MARK" and then tries to autocomplete it. If you can verify it, you might not need to; but if you can't verify it, it's certainly wrong.
That's not reliable. I've found them on the Internet in various forms (eg studybible.info). Google Books also has scanned copies of many, ancient writings. There's probably obscure sites people would miss. If searching for them, the search algorithms might avoid them to instead prioritize newer, click-bait content.
Telling what wasn't in the training data for sure should be considered impossible right now. If it matters, we need to use models with open, legal-to-share, training data. If that's impossible, one might at least use a model with training data accessible to them (eg free + licensed).
Does verbatim completion of a bible passage look different from generation of a novel sequence in interesting ways? How many sequences of this length do they memorize? Do the memorized ones roughly correspond to things humans would find important enough to memorize, or do LLMs memorize just as much SEO garbage as they do bible passages?
They take the input context and generate the next token, then feed that whole thing back in as context and predict the next token, and repeat until the most likely next token is their stop word.
If they produce anything like a retrieved sequence, that's because they just happened to pick that set of tokens based on their training data. Regenerating the output from exactly the same input has a non-zero chance of generating different output.
The question I’m asking is, how is this working in an LLM? How exactly do their weights encode (seemingly) the entire bible such that they can recreate long passages verbatim from a prompt that likely doesn’t appear anywhere in the training data (e.g. some vague description of a particular passage).
The point is, though – somehow the model has memorized these passages, in a way that allows reliable reproduction. No doubt in a super amorphous and diffuse way, as minute adjustments to the nth sigbits of myriads of floating-point numbers, but it cannot be denied that it absolutely has encoded the strings in some manner. Or otherwise you have to accept that humans can't memorize things either. Indeed given how much our memory works by association, and how it's considerably more difficult to recount some memorized sequence from an arbitrary starting point, it's easy to argue that in some relevant way human brains are next-token predictors too.
Yes, if you reduce temperature to zero and set the same random seed, you should get the same output tokens for a given set of input tokens.
However, there is no guarantee the output for a given seed will be the correct expected output.
For example, there logically must be a model and seed where providing the lord's prayer as input for completion produces a Metallica song as output, because that's a viable set of input tokens: https://genius.com/Metallica-enter-sandman-lyrics
That seed is no more or less valid than any other seed which completes the actual lord's prayer or which provides something completely different. All those seeds are just predicting their next token.
If people want that sort of exact reliable retrieval of sequences, and for the sequences to be "correct", then an LLM is the wrong tool for the job.
Then last night I saw a video about the Parker Solar Probe and how at 350,000mph it was the fastest moving man-made object. So I asked chatgpt how long at that speed it would take it to get to Alpha Centauri which is 4.37 light years away. It said it would take 59.8 million years. I knew that was way too long so I had it convert mph to miles per year and then it was able to give me the correct answer of 6817 years.
Did they try this on obscure bible excerpts, or just ones likely to be well known and quoted elsewhere? Well known quotes would be reinforced by all the copies.
the article contains examples of both
Are reciting and not-overfitting at odds?
* Using a system I developed myself; currently in open development: https://www.laleolanguage.com
https://www.biblicaltraining.org/learn/institute/nt201-bibli...
(Note: They actually host free classes from instructors at over a dozen seminaries. Mounce himself is a top expert in Greek.)
For anyone learning Biblical Hebrew, I found Master's Seminary has some courses on it:
https://www.youtube.com/watch?v=Qvh8yziVsCE&list=PL9392DD285...
https://www.youtube.com/watch?v=joDB5azc_CM&list=PL4DC84F8EB...
So Guided Immersion tries to just give you not only vocab, but grammar in such a way that there's always only a handful of concepts you haven't mastered.
I developed Guided Immersion to help myself master Mandarin, actually; I used Anki with Mandarin for probably 8 years before developing Guided Immersion; once I switched I never went back. Then about a year and a half ago ago I ported it over to Koine Greek not knowing any Greek, and started using it myself after watching a handful of YouTube Videos introducing the characters and the basic cases.
Maybe it's just the way my brain works, but I can't imagine sitting down and trying to memorize all those endings, particularly for the verbs.
I have now bought Mounce's "Basics of Biblical Greek Grammar", and "The Morphology of Biblical Greek", to help me refine the "language schema" the algorithm uses. I appreciate the work Mounce has done to find the deeper morphological rules which make sense of what look like "irregular" inflections; teaching the algorithm about those will certainly help it to present things in a more useful way to learners. But I don't think trying to grind through all that in your conscious mind is the way to go.
Interesting. In my very religious upbringing I wasn't allowed to read fairy tales. The danger being not able to classify which stories truly happened and which ones didn't.
Might be an interesting variant on the Turing test. Can you make the AI believe in your religion? Probably there's a sci-fi book written about it.
Thanks for sharing. You might be interested: JRR Tolkien, 'fairy tale' author, was also a/the leading scholar of Old English (Anglo-Saxon), related languages, and the culture and myth around them - including 'fairy tales'; and he was a devout Catholic.
How could he write (and study) such ungodly material? He wrestles with the question multiple times, but if you are interested, I strongly recommend On Fairy-stories, an essay based on a lecture. It covers far more ground than this question, but it's worth reading anyway. I'll append a spoiler below.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
There's more to it than this, but it's a wonderful vision:
"The Gospels contain a fairy-story, or a story of a larger kind which embraces all the essence of fairy-stories. They contain many marvels - peculiarly artistic,[1] beautiful, and moving: 'mythical' in their perfect, self-contained significance; and among the marvels is the greatest and most complete conceivable eucatastrophe. But this story has entered History and the primary world .... The Birth of Christ is the eucatastrophe of Man's history. The Resurrection is the eucatastrophe of the story of the Incarnation. This story begins and ends in joy. It has pre-eminently the 'inner consistency of reality'. There is no tale ever told that men would rather find was true, and none which so many sceptical men have accepted as true on its own merits. For the Art of it has the supremely convincing tone of Primary Art, that is, of Creation. ..."
[1] "The Art is here in the story itself rather than in the telling; for the Author of the story was not the evangelists."
I believe this happens because the verses and verse-specific commentary are abundant in the pre-training sources they used. Whereas, if one asks a highly-interpretive question, then it starts re-hashing other patterns in its training data which are un-Biblical. Asking about intelligent design, it got super hostile trying to beat me into submission to its materialistic worldview every paragraph.
So, they have their uses. I’ve often pushed for a large model trained on Project Gutenberg to have a 100% legal model for research and personal use. A side benefit of such a scheme would be that Gutenberg has both Bibles and good commentaries which trainers could repeat for memorization. One could add licensed, Christian works on a variety of topics to a derived model to make a Christian assistant AI.
It's a great, fun test.
> One taught me love, one taught me patience and one taught me pain.
https://knowyourmeme.com/memes/thank-u-next
BTW, did someone already code an automated meme generator using LLMs? Only half-joking.
That said, what I was getting at is that its a tough test. The differences between translations are often very small.
Why do you put a weird computer model between you and a computer and errr Your Faith? Do bear in mind that hallucinations might correspond to something demonic (just saying)
I'm a bit of a rubbish Christian but I know a synoptic gospel when I see it and can quote quite a lot of scripture. I am also an IT consultant.
What exactly is the point of Faith if you start typing questions into a ... computational model ... and trusting the outputs? Surely you should have a decent handle on the literature: It's just one big physical book these days - The Bible. Two Testaments and a slack handful of books and that for each. I'm not sure exactly but it looks about the same size as the Lord of the Rings.
I've just checked: Bible: 600k LotR: 480K - so not too far off.
I get that you might want to ask "what if" types of questions about the scriptures but why would you ask a computer? Faith is not embedded in an Intel Core i7 or an Nvidia A100.
Faith is Faith. ChatGPT is odd.
1. Preparing a sermon for Church, I don't advocate for this, but it's definitely being done out there. Here, the pastor may know the topic they are speaking on, but want the LLM to help them plan out the message and structure it. 2. Preparing lesson plans for Sunday School. This seems reasonably fine to me, but I would still err on the side of not trusting the raw scriptures output as evidence, and instead look them up separately before reading them out.
The above examples may particularly come into play when English is not a first language, since although they can understand and express their faith easily in their native language, ChatGPT can help them represent it in English well.
Personally, I think the use cases are many, but mostly for discussion / personal reflection. These include things like asking for perspectives that other Christians take on certain passages, helping understand how some scriptures link to other scriptures in the Bible, and sometimes even exploring some of the history of the Christian faith through the last ~2 millennia since it was written. Anything meaningful you can manually research further / reference before taking it at face value, but it can work as a great starting point for your search.
Not necessarily in a way where I would judge it though, and I certainly see how that could have use cases. It just feels a little bit like water gun baptisms, conceptually.
One question of a less spiritual nature -- are we strictly talking about recall from within the models themselves? I've never gotten deep enough into this kind of thing to mess with RAG pipelines, but I wonder if direct access to a translation or several would have any impact on its overall effectiveness for this.
Does anyone know any more thorough papers on this topic? For example, this could be tested on every verse in bible and lots of other text that is certainly in the training data: books in project gutenberg, wikipedia articles, etc.
Edit: this (and its references) looks like a good place to start: https://arxiv.org/abs/2407.17817v1
I think the experiment of using the LLM to recall described verses - eg “what’s the verse where Jesus did X”- is a much more interesting use. I think also that the LLM could be handy as, or to construct, a concordance. But I’d just use a document or database if I wanted to look up specific verses.
But also, LLM's in general build a lossy compression of their training data so are not the right tool if you want a completely accurate recall.
Will the recall be accurate enough for a particular task? Well I'm not a religious person so I have no framework to help decide that question in the context of the bible. If you want a system to answer scripture questions I would expect a far better approach than just an LLM would be to build a RAG system and train the RAG embedding and search at the same time you train the model.
[1] https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
I got exiled into an isolated copy of an AI-populated internet once and they put perfectly accurate bible quotes into dictionaries!
I experience the exact same problem with human beings.
> , which we regard as the inspired Word of God.
QED