Tell HN: ChatGPT Can Decode Base64

8 pointsSupply54112y ago26 comments

Some engine must hooking into the LLM to answer questions like these. Are there any details about this process?

https://chat.openai.com/share/696def43-be52-4574-ad18-4f8ecf8557b0

EDIT: An even longer, more convincing test https://chat.openai.com/share/417169ed-8256-46cb-a3ed-81ab06a1d70b

8 pointsSupply54112y ago26 comments

Some engine must hooking into the LLM to answer questions like these. Are there any details about this process?

https://chat.openai.com/share/696def43-be52-4574-ad18-4f8ecf8557b0

EDIT: An even longer, more convincing test https://chat.openai.com/share/417169ed-8256-46cb-a3ed-81ab06a1d70b

26 comments

14 comments · 7 top-level

usgroup2y ago· 3 in thread

Just reading the wikipedia page on Base64 (https://en.wikipedia.org/wiki/Base64). Algorithmically it seems to be a 8-bit byte to character mapping. So, if you were to base encode a character string it'd be {characters} -> {bytes} ->[base64] -> {characters}.

I'd presume this prompt would work fine:

Here is a mapping of numbers to letter given as a pairs: (A,1) (B,7) (C,23) (D,14). Use the mapping to convert the following string to numeric representation: ACCDAB.

I'd also presume that the same template could be used to instruct ChatGPT to do Base64 (with some additions for padding).

Yet, I suspect the capability given in the above prompt is not surprising to anyone. So, why is Base64 surprising? What am I missing?

Supply5411OP2y ago

>Algorithmically it seems to be a 8-bit byte to character mapping

Base64 is 6-bits, so there is no mapping of a single character to another single character. Changing 1 character in the input can change 1 or 2 characters in the output because each of the 6-bit "characters" are up against each other. So there is no mapping of input character to output character(s), it depends on context.

I'm aware that regular NL translations are a similar problem, but with base64, it's a much more exact and unforgiving problem, and GPT-4 seems to perform at a very high level.

usgroup2y ago

Ok I see, but that doesn't strike me as different in kind, is it? I.e. bytes to 6-bit sequences for which there is a mapping table. Now you have a string of 6 symbol sequences. Then divide the string into 8 symbol sequences and map each to a character (again via mapping table). Ignoring padding rules, its two mappings, one followed by the other?

cpach2y ago

One can indeed argue that it’s not surprising – depending on your prior understanding of how Base64 works.

But for a person where both Base64 and LLMs are black boxes, then it can indeed be surprising.

To me, the most surprising thing about ChatGPT and other LLMs are the quality (despite all their flaws) and ther general applicability to a broad range of diverse tasks.

Tiberium2y ago· 2 in thread

That's been widely known for quite some time, there's no engine, it just happens "magically" when a LLM gets big enough (and they sure have a lot of that stuff in their training datasets). GPT-4 is generally very good at this, while Turbo will often make mistakes if you force it to decode Base64 that's some nonsense (like random characters).

Supply5411OP2y ago

Sorry but I have to call bs. How do you explain this? It decodes a huge (guaranteed unique) string perfectly:

https://chat.openai.com/share/417169ed-8256-46cb-a3ed-81ab06...

ipaddr2y ago

Too generic to be used as a test. Try changing the comments to include gjdckfkdkxkdkskx34717Q 22371 re-encode and report back

1 more reply

muzani2y ago· 2 in thread

It's quite inaccurate. It makes frequent mistakes especially in unknown words like names, typos, and swear words. And at some length, it hallucinates the rest of the content.

So it seems to be able to "feel" the answer out there but it's not directly translating it. That's far more fascinating than if it were decoding while hooked to something else.

Supply5411OP2y ago

I'm getting perfect results with very long strings. Can you share an example that fails for you?

muzani2y ago

Base64:

VGVsbCBFYS1uYXNpcjogTmFubmkgc2VuZHMgdGhlIGZvbGxvd2luZyBtZXNzYWdlOiAgV2hlbiB5b3UgY2FtZSwgeW91IHNhaWQgdG8gbWUgYXMgZm9sbG93cyA6ICJJIHdpbGwgZ2l2ZSBHaW1pbC1TaW4gKHdoZW4gaGUgY29tZXMpIGZpbmUgcXVhbGl0eSBjb3BwZXIgaW5nb3RzLiIgWW91IGxlZnQgdGhlbiBidXQgeW91IGRpZCBub3QgZG8gd2hhdCB5b3UgcHJvbWlzZWQgbWUuIFlvdSBwdXQgaW5nb3RzIHdoaWNoIHdlcmUgbm90IGdvb2QgYmVmb3JlIG15IG1lc3NlbmdlciAoU2l0LVNpbikgYW5kIHNhaWQ6ICJJZiB5b3Ugd2FudCB0byB0YWtlIHRoZW0sIHRha2UgdGhlbTsgaWYgeW91IGRvIG5vdCB3YW50IHRvIHRha2UgdGhlbSwgZ28gYXdheSEiICBXaGF0IGRvIHlvdSB0YWtlIG1lIGZvciwgdGhhdCB5b3UgdHJlYXQgc29tZWJvZHkgbGlrZSBtZSB3aXRoIHN1Y2ggY29udGVtcHQ/IEkgaGF2ZSBzZW50IGFzIG1lc3NlbmdlcnMgZ2VudGxlbWVuIGxpa2Ugb3Vyc2VsdmVzIHRvIGNvbGxlY3QgdGhlIGJhZyB3aXRoIG15IG1vbmV5IChkZXBvc2l0ZWQgd2l0aCB5b3UpIGJ1dCB5b3UgaGF2ZSB0cmVhdGVkIG1lIHdpdGggY29udGVtcHQgYnkgc2VuZGluZyB0aGVtIGJhY2sgdG8gbWUgZW1wdHktaGFuZGVkIHNldmVyYWwgdGltZXMsIGFuZCB0aGF0IHRocm91Z2ggZW5lbXkgdGVycml0b3J5LiBJcyB0aGVyZSBhbnlvbmUgYW1vbmcgdGhlIG1lcmNoYW50cyB3aG8gdHJhZGUgd2l0aCBUZWxtdW4gd2hvIGhhcyB0cmVhdGVkIG1lIGluIHRoaXMgd2F5PyBZb3UgYWxvbmUgdHJlYXQgbXkgbWVzc2VuZ2VyIHdpdGggY29udGVtcHQhIE9uIGFjY291bnQgb2YgdGhhdCBvbmUgKHRyaWZsaW5nKSBtaW5hIG9mIHNpbHZlciB3aGljaCBJIG93ZSg/KSB5b3UsIHlvdSBmZWVsIGZyZWUgdG8gc3BlYWsgaW4gc3VjaCBhIHdheSwgd2hpbGUgSSBoYXZlIGdpdmVuIHRvIHRoZSBwYWxhY2Ugb24geW91ciBiZWhhbGYgMSwwODAgcG91bmRzIG9mIGNvcHBlciwgYW5kIMWgdW1pLWFidW0gaGFzIGxpa2V3aXNlIGdpdmVuIDEsMDgwIHBvdW5kcyBvZiBjb3BwZXIsIGFwYXJ0IGZyb20gd2hhdCB3ZSBib3RoIGhhdmUgaGFkIHdyaXR0ZW4gb24gYSBzZWFsZWQgdGFibGV0IHRvIGJlIGtlcHQgaW4gdGhlIHRlbXBsZSBvZiBTaGFtYXNoLiAgSG93IGhhdmUgeW91IHRyZWF0ZWQgbWUgZm9yIHRoYXQgY29wcGVyPyBZb3UgaGF2ZSB3aXRoaGVsZCBteSBtb25leSBiYWcgZnJvbSBtZSBpbiBlbmVteSB0ZXJyaXRvcnk7IGl0IGlzIG5vdyB1cCB0byB5b3UgdG8gcmVzdG9yZSAobXkgbW9uZXkpIHRvIG1lIGluIGZ1bGwuICBUYWtlIGNvZ25pemFuY2UgdGhhdCAoZnJvbSBub3cgb24pIEkgd2lsbCBub3QgYWNjZXB0IGhlcmUgYW55IGNvcHBlciBmcm9tIHlvdSB0aGF0IGlzIG5vdCBvZiBmaW5lIHF1YWxpdHkuIEkgc2hhbGwgKGZyb20gbm93IG9uKSBzZWxlY3QgYW5kIHRha2UgdGhlIGluZ290cyBpbmRpdmlkdWFsbHkgaW4gbXkgb3duIHlhcmQsIGFuZCBJIHNoYWxsIGV4ZXJjaXNlIGFnYWluc3QgeW91IG15IHJpZ2h0IG9mIHJlamVjdGlvbiBiZWNhdXNlIHlvdSBoYXZlIHRyZWF0ZWQgbWUgd2l0aCBjb250ZW1wdC4g

ChatGPT's translation (3.5):

The decoded message is: "Tell EA-namir: Nanni sends the following message: When you came, you said to me as follows: 'I will give Gimp-Sin (when he comes) five shekels of silver.' You took your garments, even the garments of your back, just as you had promised, to give the messenger of Gimp-Sin (when he came) five shekels of silver, and then send my messenger. They grasped me, took me away and made me enter for the presence of Bel-Adda (the king)."

Errors: "EA-namir" instead of "Ea-nasir", "Gimp-Sin" instead of "Gimil-Sin", "five shekels of silver" instead of "fine quality copper ingots".

After that it's just hallucination.

2 more replies

pillefitz2y ago

Would be interesting to feed it into some open source LLM that for sure does not use an external decoder and determine whether it still gets it (approximately) right.

lfconsult2y ago

Does ChatGPT integrates basics known encoders/decoders aside from LLM (if not, I thought that it was the case)?

Base64 is about encoding not hashing... So decoding it is... natural. The point here is that it's integrated inside ChatGPT? It seems to me that it's an "obvious" "feature" kind of.

al2o3cr2y ago

Some previous work related to this:

https://tildes.net/~tech/15kn/rot13_base64_on_gpt4_reliable_...

thesimpleone2y ago

I would be shocked if it couldn’t decode base64.

j / k navigate · click thread line to collapse

26 comments

14 comments · 7 top-level

usgroup2y ago· 3 in thread

I'd presume this prompt would work fine:

Here is a mapping of numbers to letter given as a pairs: (A,1) (B,7) (C,23) (D,14). Use the mapping to convert the following string to numeric representation: ACCDAB.

I'd also presume that the same template could be used to instruct ChatGPT to do Base64 (with some additions for padding).

Yet, I suspect the capability given in the above prompt is not surprising to anyone. So, why is Base64 surprising? What am I missing?

Supply5411OP2y ago

>Algorithmically it seems to be a 8-bit byte to character mapping

I'm aware that regular NL translations are a similar problem, but with base64, it's a much more exact and unforgiving problem, and GPT-4 seems to perform at a very high level.

usgroup2y ago

cpach2y ago

One can indeed argue that it’s not surprising – depending on your prior understanding of how Base64 works.

But for a person where both Base64 and LLMs are black boxes, then it can indeed be surprising.

To me, the most surprising thing about ChatGPT and other LLMs are the quality (despite all their flaws) and ther general applicability to a broad range of diverse tasks.

Tiberium2y ago· 2 in thread

Supply5411OP2y ago

Sorry but I have to call bs. How do you explain this? It decodes a huge (guaranteed unique) string perfectly:

https://chat.openai.com/share/417169ed-8256-46cb-a3ed-81ab06...

ipaddr2y ago

Too generic to be used as a test. Try changing the comments to include gjdckfkdkxkdkskx34717Q 22371 re-encode and report back

1 more reply

muzani2y ago· 2 in thread

It's quite inaccurate. It makes frequent mistakes especially in unknown words like names, typos, and swear words. And at some length, it hallucinates the rest of the content.

So it seems to be able to "feel" the answer out there but it's not directly translating it. That's far more fascinating than if it were decoding while hooked to something else.

Supply5411OP2y ago

I'm getting perfect results with very long strings. Can you share an example that fails for you?

muzani2y ago

Base64:

ChatGPT's translation (3.5):

Errors: "EA-namir" instead of "Ea-nasir", "Gimp-Sin" instead of "Gimil-Sin", "five shekels of silver" instead of "fine quality copper ingots".

After that it's just hallucination.

2 more replies

pillefitz2y ago

Would be interesting to feed it into some open source LLM that for sure does not use an external decoder and determine whether it still gets it (approximately) right.

lfconsult2y ago

Does ChatGPT integrates basics known encoders/decoders aside from LLM (if not, I thought that it was the case)?

Base64 is about encoding not hashing... So decoding it is... natural. The point here is that it's integrated inside ChatGPT? It seems to me that it's an "obvious" "feature" kind of.

al2o3cr2y ago

Some previous work related to this:

https://tildes.net/~tech/15kn/rot13_base64_on_gpt4_reliable_...

thesimpleone2y ago

I would be shocked if it couldn’t decode base64.

j / k navigate · click thread line to collapse