Additionally, the premise of sending gradients of models trained on private data while retaining privacy seems problematic. While you likely can't reverse it to calculate the batch's contents, it is leaky.
Further, gradient calculations are not a good proof of work. A good proof of work is difficult to calculate and easy to verify (i.e. an extremely low hash value).
The core premise, using blockchains to store terabytes of models and datasets, doesn't make any sense whatsoever.
The problems highlighted in this article however are valid, and it would be great to see something like IPFS for datasets and models.
Well perhaps one way is you could have another LLM take a look at the data you are submitting and have it predict p(useful|not useful) , and create an incentive for users to generate authentic data
This must be quite an insult to the inventor of fully decentralised AI. In 2012 Prof. Jelasity invented gradient mining with SGD, https://arxiv.org/pdf/1109.1396 Works reasonably well, you probably also need a ledger like Trustchain for accountability and Sybil protection. Consensus is too costly. This problem is unsolved and actively worked on academia for decades.
you can randomly ask sometimes 4 different nodes to calculate the same gradient, and see who is cheeting.
That's a specific Proof-of-Work system known as Hashcash [1]. There are many others, such as Cuckoo Cycle, in which the solution is a fixed-length cycle in a random bipartite graph [2].
[1] https://en.wikipedia.org/wiki/Hashcash
[2] http://cryptorials.io/beyond-hashcash-proof-work-theres-mini...
Edit: The number of AI safety sessions I’ve joined where the speakers have no real AI experience talking about potentially bad futures, based on zero CS experience and little ‘evidence’ beyond existing sci-fi books and anecdotes, have left me very jaded on the subject as a ‘discipline’.
I believe it comes down to three groups:
AI researchers/organisations wanting to make their work sound very important/scary
Humanities researchers wanting STEM funding
AI organisations trying to bring in legislation to slow down competition
(Alas if you wanted to avoid "killer robots" you're already too late, see that whole episode where a certain large military power decided that target selection of cellphones to kill due to "metadata" could be achieved by a system not unkind to the facebook algorithm, with the only safeguard some wet-behind-the-ears kid who's piloting the drone from half way around the world and a "computer says die" attitude).
This article manages to say that a consensus algorithm is the answer to problems with AI without using the word "consensus" a single time. Probably because it doesn't even consider that's what blockchains are.
The article talks about using "proof of gradient" to do inference instead of crunching hashes. But this is nonsense, because inference takes inputs and produces a deterministic output. There's no mining. Checking the work takes the same resources as doing the work. Proof of work output can be checked with essentially a single hash.
As much as this would be wonderful in a universe where it's possible, it's simply not possible. The author throws out a bunch of buzz words for things that sound similar in AI and crypto and tries to make them sound like they are interchangeable. They're not.
We do live in that universe, under some currently believed assumptions. An NP-complete problem is an example of something where checking a solution is (thought to be) easier than finding one.
Zero-knowledge proofs make it such that checking that a computation (such as inference) has been done correctly is easier than doing the computation (even keeping some parts private). A great reference is here: https://people.cs.georgetown.edu/jthaler/ProofsArgsAndZK.pdf
And if the crypto-economy is doing well enough that node-runners are economically incentivized to continue running, then is the rogue AI even that harmful?
Double spending isn't one of the problems in AI. This article is absolute nonsense
> a cryptographically secure, decentralized ledger is the only solution to making AI safer
The decentralized AI that is working is just sharing models. Splitting the inference across the net for cooperative execution is possible.
The thing that really needs to be solved is distributed training. We need to be able to train base models in a manner more like Folding@Home so we’re not captive to organizations with big expensive GPU farms to make any innovation.
And that is safer because if everyone has access to the model, then people can use it in unsafe way. And it would be much more difficult to enforce the laws, when you have to control what everyone in the world does, than just one or few entities (companies, organizations, governments, whatever...). Unless it talks about decentralizing it in such way that different nodes store different parts of the model, instead of all nodes having the entire model (in that case, it's fine).
> High Cost and Resource Requirements
For training and local inferencing use, quantization may help. Problem becomes local via quantization vs. remote full tensor use. Solution may involve distributed inferencing. Techniques like model distillation can help create smaller, more efficient models for inferencing.
> Data Privacy
For training, some private datasets may be needed. For local inferencing use, determining what needs to be inferenced locally vs. what needs to be run remotely may be useful. Problem becomes privacy scope mapped onto a marketplace to mitigate high cost and resource requirements. Techniques like model explainability (versioning) and robustness testing can help build trust in AI systems.
Complying with data privacy regulations and ensuring that AI systems adhere to legal and ethical standards can be a challenge, especially in international contexts.
> Incentives
Instead of assuming the solution when considering the problem, we assume there is an incentive to either simply train a model or use one. Problem becomes financial rewards, data access agreements, or even altruistic motivations.
> Stale Data and Reproducibility
Both the code and datasets for training the model need to be updated. Inferencing needs RAG, so the augmented reference data needs to be updated as well. Anything updated might need some type of revision control, especially if that data (or code) results in poor output. Labeling data and knowledge transfer are other problems that needs revision control.
> Interoperability
We can assume a marketplace for a ML train/inference platform is needed. We have HuggingFace, for example. The problem here is likely based on the tendency for datasets to be private, such as in the case of Llama 2. Models contain the "essence" of the dataset, but we still need RAG to ground the responses.
The use of the Lightning Network combined with a proposed 402 response code is an interesting concept for addressing some of these challenges: https://github.com/lightninglabs/aperture
It could provide a decentralized and efficient way to facilitate payments for dataset access, training, and inferencing to incentivize data sharing and model usage.
Actually it is. But in perverse way it isnt.
"AI" is currently being developed and commercialized in what is essentially an old-fashioned way, with a whiff of theft of intellectual property of the training material never too far.
Decentralized AI is a distinct destination but the technologies that might realize it need not have anything remotely related to blockchain (a solution that was conceived for a different - perceived - problem, the centralization of money creation)
Probably too late, but Elsevier could fix its bad reputation if they required relevant academic papers to submit reproducible code and subjected them to automated testing.
