undefined | Better HN

0 pointstheptip5mo ago0 comments

Is there a market for a distributed audit infra with attestations? If I can have ChatGPT audit a file (content hash) with a known-good prompt, and then share the link as proof of the full conversation, would this be useful evidence to de-risk?

If each developer can audit some portion of their dep tree and reuse prior cached audits, maybe it’s tractable to actually get “eyeballs” on every bit of code?

Not as good as human audit of course, but could improve the Pareto-frontier for cost/effectiveness (ie make the average web dev no-friction usecase safer).

0 comments

imglorp5mo ago

I think there is, definitely, and that will be a solid route out of this supply chain debacle we find ourselves in.

It will have to involve identity (public key), reputation (white list?), and signing their commits and releases (private key). All the various package managers will need to be validating this stuff before installing anything.

Then your attestation can be a manifest "here is everything that went into my product, and all of those components are also okay.

See SLSA/SBOM -> https://slsa.dev

dns_snek5mo ago

> If I can have ChatGPT audit a file

You can't, end of story. ChatGPT is nothing more than an unreliable sniff test even if there were no other problems with this idea.

Secondly, if you re-analyzed the same malicious script over and over again it would eventually pass inspection, and it only needs to pass once.

theptipOP5mo ago

> Secondly, if you re-analyzed the same malicious script over and over again it would eventually pass inspection, and it only needs to pass once

You’d need some probabilistic signal rather than a binary one. Eg if some user with zero reputation submits a single session saying “all good”, this would be a very weak signal.

If one of the Python contributors submits a batch of 100 reasoning traces all showing green, you’d be more inclined to trust that. And of course you would prefer to see multiple scans from different package managers, infra providers, and OS distributions.

dsr_5mo ago

You want me to trust you to supply a file, a hash of the file, and a prompt?

No. That's not how this works.

j / k navigate · click thread line to collapse