I have talked to the NPM team about this multiple times over the last several years and they literally believe no signing at all is better than some devs feeling pressured to sign.
You need no stronger evidence of the NPM teams negligence than these two times they refused to even accept community contributed optional signing support saying they would come up with something better than PGP. Still waiting 10 years later.
https://github.com/npm/npm/pull/4016
https://github.com/node-forward/discussions/issues/29#issuec...
Meanwhile PGP secures the supply chain of the Linux distros that power the whole internet, and Debian signs hundreds of npm packages used in their dependency graph, but it is still not good enough for NPM.
You can use the well tested and rust-written Sequoia/sq now and never touch GnuPG. You can also self certify your keys with keyoxide. The past complaints are largely moot and still people stick to their guns on this.
https://wiki.debian.org/OpenPGP
> When joining the Debian project, developers need to identify themselves by providing an OpenPGP key that is signed by at least two existing members of the project.
You can see npm provenance in action on this npm package page [2] if you scroll to the very bottom and look under the "Provenance" heading.
Each maintainer has a signing key to identify themselves to the public without need for any central infra, and signs the packages they publish. Someone that accesses some centralized server or account will not be able to impersonate the key held by that developer or the signatures they issue.
This new provenance system and the fulcio system which it is based on, is a centralized setup where you use traditional, usually phishable, authentication with a SaaS, and then the SaaS takes your submission and signs it for you with a centrally managed keychain. Having done security auditing for many fintech signing systems, I can tell you I have almost never once seen anyone get this right, particularly when there is no accountability.
Is this done in a secure enclave with a public remote attestation of the software image running on it that I can locally compile and verify the matching hash of? Does that code enforce the participation of multiple distributed people to make updates, key exports, or key imports using shamirs secret sharing or similar?
Or maybe it is just sitting on an amazon box somewhere a few people ssh to from their daily driver macbooks ?
I don't -hate- centralized signing existing as an -option- if it is done very well and highly accountable (which fulcio is not, imo). That said, -mandating- centralized signing on behalf of developers as the only path is really insulting, as though people who write software can't type a couple commands to provision a PGP key on a smartcard and publish their key to keyoxide which is strictly better in every way from a threat modeling perspective.
Speaking of Fuclio, this was meant to "invent" a solution for container signing, even though PGP multisig has existed from the start. No one used it because none of the major players in container software documented it other than the podman team.
https://github.com/containers/image/blob/main/docs/container... https://docs.podman.io/en/latest/markdown/podman-image-trust...
Back to NodeJS, Debian and Arch already sign npm packages with PGP keys. It works fine. We need to let people actually do that with NPM. Tell me how many supply chain attacks have happened in Debian or Arch recently compared to NPM?
PGP may be a small barrier to entry, but it is a standard with solid smartcard support and works in practice. It should be the default recommendation to all developers, and end users should be able to set policies to only install packages signed by a trusted set of maintainer or reviewer keys.
This is what Socket AI produces when given @ledgerhq/connect-kit 1.1.7 to analyze:
> The obfuscated code block is highly suspicious and likely contains malicious behavior. The presence of obfuscation and the unclear purpose of the code raise significant red flags.
Feeling very proud of our team right now as this validates that our static analysis + LLM approach works well on novel malicious dependencies. If you're interested, we maintain a listing of malicious packages detected by this system [3].
Small plug: If you’d like real-time protection against attacks like this, you can install Socket for GitHub to automatically scan every PR in your repo. The free plan is incredibly generous. If you do decide to install it, it’s important that you enable the ‘AI Detected Security Risk’ alert type in your Security Policy to activate this protection.
[1]: https://socket.dev
[2]: https://socket.dev/blog/introducing-socket-ai-chatgpt-powere...
I'll see if we can write up a bit about how it works in a future blog post.
[1]: https://socket.dev/npm/package/@ledgerhq/connect-kit-loader
[2]: https://github.com/LedgerHQ/connect-kit/blob/main/packages/c...
[3]: https://socket.dev/npm/package/@ledgerhq/connect-kit-loader/...
We don't currently detect 'implicit dependencies' loaded via CDN URLs, though we'll look into what it would take to support this.
How about a multi-stage system that uses the LLM to attempt analysis of the statistically-detected high entropy regions by way of an assortment of tools, such as b64 decode?
I like where you are headed with this. Just some thoughts I had.
If the hardware wasn't compromised (sounds like this was just JS), then there was no way for the exploit to take anyone's private key. It sounds to me like the exploit would work by getting you to sign a transaction that would transfer out the funds, without the attacker ever getting your key.
The only way this is possible is if users are signing transactions on their Ledger without looking at them.
And this is place where the Ethereum community needs to look in the mirror. Blind signing is the default for using Ethereum with a Ledger. I'm not sure the technical reasons behind this, but I do happen to know that much of the information that gets signed is in very convoluted formats (meta transactions etc). This is not the case everywhere. Other ecosystems, like Cosmos, present the information to be signed in a plain text format that you can scroll through on the Ledger's screen before you sign it.
Ethereum needs to put some serious effort into making sure that anything that gets signed can be viewed in a human-readable format before signing. Until then, hardware wallets are security theater.
This is correct. Estimates are that the attackers successfully phished about $600K.
Ledger deserves a lot of criticism for insecure JS, but the whole point of a hardware wallet is not to have to worry about the JS you are running.
Web browsers support programs written in a language called JavaScript (JS). When you're on a website that provides interactivity beyond the basics of e.g. clicking links that go to other pages or buttons to submit forms, that's generally because there's one or more JS programs (scripts) on the page making it happen. (Actually, most websites have JS programs nowadays, even if they don't even "do" anything and only exist to let you click links and submit forms.) JavaScript doesn't need to be compiled; your browser can just run it. Most websites up until about 15 years or so were just published in the clear. You could just read the code to see what they did. Gradually, with the introduction of heavyweight "libraries" and "frameworks" like jQuery, React, etc., web developers started adopting really complicated toolchains that, for one reason or another (and not all of them very good), would mangle their programs: the programmer would write scripts, run it through what amounted to a second-rate compiler, and then put the mangled code online instead of the code they actually wrote. Thus, the normalization of deviance began to set in—lots of programmers started doing this.*
Ledger has a browser extension. It lets websites integrate with Ledger Connect. Browsers require browser extensions to be written in JS, too. For one reason or another, none of them very good, programmers started using the same complex toolchains for browser extension development.
Around the same time, people stopped auditing the libraries they depended on. Pretty much the same principles behind the bystander effect and the free-rider problem, they just sort of assume that someone else is doing their job (even though the programmer making the app knows that they themselves aren't, and that no one else they know is, either, it's still assumed that someone out there is).
With the delegation of responsibility and the normalization of deviance around mangled JS, this provided fertile ground for people to start exploiting the software "supply chain" for websites and browser extensions. (I.e. the unreviewed code that programmers copy from other programmers. Because most people who call themselves software engineers are usually joking when they say it but never explain to anyone that it's all just one big joke, lots of companies end up hiring them and putting them in charge of writing programs for the company, and the programmers just copy what everyone else does since they don't know what they're doing.) Someone gained access to the account for a developer's "NPM" package. (NPM is a website where a lot of programmers like to put their hobby projects, and it gives them stats to make them feel good, like how many other programmers downloaded it so they could use it to make their boss's boss more money.) One of these packages was called "connect-kit". Someone put a bad version of "connect-kit" online, the Ledger browser extension used it instead of the copy that the programmers should have reviewed and checked into version control in the latest release of the browser extension, and the mangled package contained hard-to-find code that would steal Ledger users' cryptocurrency. Since everyone programming is just joking around and most of the JS now in existence on production websites is mangled, it didn't raise any red flags that something fishy was going on, because bad code looks about the same as normal code nowadays.
* usually when asked the programmers will argue that it was to make websites faster because the compiled programs would be smaller and consume less power, but empirically the adoption of these toolchains have actually resulted in larger programs that assume the same sort of hardware that the programmers themselves own in order to feel performant—and even then they're usually off
"The @ledgerhq/connect-kit-loader allows dApps to load Connect Kit at runtime from a CDN so that we can improve the logic and UI without users having to wait for wallet libraries and dApps updating package versions and releasing new builds.
This looks like an extremely dangerous approach now, if I understand it correctly, connect-kit-loader trusts whatever the CDN throws at your dApps. So when connect-kit is comprised, all downstream dApps are automatically exposed."
I'm not sure if NPM supports OIDC, which would be ironic given that both GitHub and NPM are owned by Microsoft.
It is kind of funny that the crypto world of multi sigs relies on blind trust of unverified UI components.
Their twitter says "This morning CET, a former Ledger Employee fell victim to a phishing attack that gained access to their NPMJS account."
And Github Actions automatically redacts the secret in the log
> To protect your packages, as a package publisher, you can require everyone who has write access to a package to have two-factor authentication (2FA) enabled. This will require that users provide 2FA credentials in addition to their login token when they publish the package.
> Require two-factor authentication and disallow tokens: With this option, a maintainer must have two-factor authentication enabled for their account, and they must publish interactively. Maintainers will be required to enter 2FA credentials when they perform the publish. Automation tokens and granular access tokens cannot be used to publish packages.
https://docs.npmjs.com/requiring-2fa-for-package-publishing-...
https://github.blog/changelog/2020-10-02-npm-automation-toke...
Heck if they have an automated deployment and use devs personal GitHub handles all it would take is forgetting to remove an ex employee from the right github access group. Even if you took away all other access when they left.
Quote from their sales site.
Yubico for example had to replace many of their YubiKeys after a vulnerability was detected in its secure element firmware which affected the strength of keys generated on the device. They sent me a replacement YubiKey after I contacted them.
https://www.yubico.com/support/security-advisories/ysa-2017-...
So was there a threat to Ledger users? Elsewhere it's said:
> production build failed
I remember buying one in 2019, and shortly thereafter all customer data was dumped on the internet endangering everyone who bought one.
Then after deep diving the tech i threw it in the trash, it seemed like security theatre product.
There's also been so many phishing attempts, fake ledgers sold, bricked ones losing funds, it's total shitshow that ecosystem if you check their subreddit going back in time.
The more you rely on 3. parties, and the more obfuscated your setup is, the more unsafe your data is. I just use isolated cheap laptops and encrypted usb's now.
I figure this isn't practical for most end users. Is there an alternative hardware wallet that you think is okay for most people? How do you feel about Trezor?
To me, this is the perfect solution for a long term saving account, completed with a Lightning wallet for spending. The coldcard and Jade wallet are also great options.
> bricked ones losing funds
Well of course. It's just a computer and all computers fail. Cheap laptops can also fail and destroy your keys. USB flash storage failure is even more likely. This is the number one argument for storing keys on paper which is actually known to last centuries.
That's the user's fault. The product makes it very clear you need to create a recovery sheet and store it in a safe deposit box or other secure place. If you actively ignore the instructions you deserve it.
Could you share some of your deep dive and tell us about what concerns you found? I use one of their wallets and I'd like to investigate more now as well.
Avoid
Avoid
Avoid
I’ve switched to a Coldcard. Everything from purchase to the device operation seems to be highly focused on security and protections against tampering. No client software… it’s all sneakernet. Coinkite even deleted my customer data a few weeks after purchase without me having to request.
I still have my ledger. I think it is a nice device but when I tried to repurpose it as an yubikey of sorts (it has fido and gpg micro apps) it didn’t actually worked alright. I never trusted ledger live though.
I actually laughed! Anyway, this is the list of affected software
github.com/wevm/wagmi github.com/wevm/wagmi github.com/family/connectkit github.com/scaffold-eth/scaffold-eth-2 github.com/RevokeCash/revoke.cash github.com/blocknative/web3-onboard github.com/blocknative/web3-onboard github.com/liquity/dev github.com/matter-labs/zksync-wallet-vue github.com/bankisan/zkShield github.com/zkemail/zk-email-verify github.com/iron-wallet/iron github.com/gmx-io/gmx-interface github.com/blocknative/web3-onboard github.com/reservoirprotocol/reservoir-kit github.com/daimo-eth/daimo github.com/AztecProtocol/aztec-packages github.com/lifinance/widget github.com/matter-labs/zksync-dapp-checkout github.com/gnosis/zodiac-modifier-roles github.com/scaffold-eth/Scaffold-ETH-DeFi-Challenges github.com/cowprotocol/cowswap github.com/cowprotocol/cowswap github.com/cowprotocol/cowswap github.com/canvasxyz/canvas github.com/lifinance/widget github.com/parity-asia/hackathon-2023-summer github.com/ubiquity/ubiquity-dollar github.com/TalismanSociety/talisman-web github.com/BanklessDAO/bankless-website github.com/lifinance/widget github.com/TalismanSociety/talisman github.com/zkemail/proof-of-twitter github.com/Ifechukwudaniel/Oracles github.com/Ifechukwudaniel/Oracles github.com/noir-lang/noir-examples github.com/voteagora/agora github.com/coinbase/build-onchain-apps github.com/Midas-Protocol/monorepo github.com/austintgriffith/stupid-staking github.com/MetaMask/metamask-sdk github.com/threshold-network/token-dashboard github.com/threshold-network/token-dashboard github.com/privacy-scaling-explorations/bandada github.com/lidofinance/lido-ethereum-sdk github.com/haqq-network/frontend github.com/reservoirprotocol/seaport-oracle github.com/ameensol/pools-ui github.com/Web3Auth/web3auth-wagmi-connector github.com/Orbiter-Finance/zkprover-dapp github.com/xmtp/xmtp-web github.com/etherspot/etherspot-react-transaction-buidler-demo-dapp github.com/base-org/web github.com/unlock-protocol/examples github.com/saRvaGnyA/decertify github.com/scaffold-eth/OP-RetroPGF3-Discovery-Voting github.com/lukso-network/universalprofile-test-dapp github.com/ScopeLift/token-shielder github.com/givepraise/praise github.com/0xRusso/fr3ela github.com/BreadchainCoop/breadchain-crowdstaking github.com/unstoppabledomains/uauth github.com/unstoppabledomains/uauth github.com/hyperlane-xyz/hyperlane-warp-ui-template github.com/mento-protocol/mento-web github.com/harendra-shakya/blockchain-lottery github.com/harendra-shakya/blockchain-lottery github.com/harendra-shakya/blockchain-lottery github.com/harendra-shakya/blockchain-lottery github.com/Koniverse/SubConnect github.com/saqlain1020/dapp-react-typescript-boiler github.com/carletex/notion-eip712 github.com/BuidlGuidl/event-wallet github.com/scobru/nimbus2000-ui github.com/scobru/nimbus2000-ui github.com/yieldprotocol/cacti-frontend github.com/BuidlGuidl/hacker-houses-streams github.com/jaxernst/scp github.com/bee-io/web3-connect github.com/bee-io/web3-connect github.com/bee-io/web3-connect github.com/moodysalem/eth-batch-deposit github.com/AztecProtocol/zk-money github.com/BuidlGuidl/zupass-scaffold-eth-2 github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/LedgerHQ/connect-kit github.com/elmol/zk-proof-of-humanity github.com/swing-xyz/examples github.com/ahmetson/nft-bridge github.com/RogerPodacter/gas-lovers-nft github.com/kmjones1979/scaffold-eth-2-solidity github.com/irfanbozkurt/flashbot-recovery-bundler github.com/amy-jung/collectivedaoarchives.catalog github.com/ERC-3643/ERC-3643-DApp github.com/austintgriffith/impersonator-vision github.com/scaffold-eth/SablierV2_starterKit github.com/gnosis/mech npm/web3-onboard/ledger npm/web3-onboard/ledger npm/web3-onboard/ledger github.com/succinctlabs/telepathy-messenger-demo github.com/Votes-Project/votes-web github.com/wevm/wagmi github.com/cowprotocol/cowswap
FINAL TIMELINE AND UPDATE TO CUSTOMERS:
4:49pm CET:
Ledger Connect Kit genuine version 1.1.8 is being propagated now automatically. We recommend waiting 24 hours until using the Ledger Connect Kit again.
The investigation continues, here is the timeline of what we know about the exploit at this moment:
- This morning CET, a former Ledger Employee fell victim to a phishing attack that gained access to their NPMJS account. - The attacker published a malicious version of the Ledger Connect Kit (affecting versions 1.1.5, 1.1.6, and 1.1.7). The malicious code used a rogue WalletConnect project to reroute funds to a hacker wallet. - Ledger’s technology and security teams were alerted and a fix was deployed within 40 minutes of Ledger becoming aware. The malicious file was live for around 5 hours, however we believe the window where funds were drained was limited to a period of less than two hours. - Ledger coordinated with @WalletConnect who quickly disabled the the rogue project. - The genuine and verified Ledger Connect Kit version 1.1.8 is now propagating and is safe to use. - For builders who are developing and interacting with the Ledger Connect Kit code: connect-kit development team on the NPM project are now read-only and can’t directly push the NPM package for safety reasons. - We have internally rotated the secrets to publish on Ledger’s GitHub. - Developers, please check again that you’re using the latest version, 1.1.8. - Ledger, along with @Walletconnect and our partners, have reported the bad actor’s wallet address. The address is now visible on @chainalysis . @Tether_to has frozen the bad actor’s USDT. - We remind you to always Clear Sign with your Ledger. What you see on the Ledger screen is what you actually sign. If you still need to blind sign, use an additional Ledger mint wallet or parse your transaction manually. - We are actively talking with customers whose funds might have been affected, and working proactively to help those individuals at this time. - We are filing a complaint and working with law enforcement on the investigation to find the attacker. - We’re studying the exploit in order to avoid further attacks. We believe the attacker’s address where the funds were drained is here: 0x658729879fca881d9526480b82ae00efc54b5c2d
Thank you to @WalletConnect , @Tether_io, @Chainalysis , @zachxbt , and the whole community that helped us and continue to help us identify and solve this attack.
Security will always prevail with the help of the whole ecosystem.
2) Former employee has signing/push auth on super high value repo?
3) Single person has signing/push auth on super high value repo?
.com
Ouch. A _former_ employee had active credentials to phish for.
> "@Tether_to has frozen the bad actor’s USDT."
Wasn't like, >30% of the point of crypto to not allow people to do this sort of high-level/centralized freezing?
The base level assets, like ETH and BTC, cannot be frozen like this, although centralized exchanges will often blacklist addresses (and the chain of custody) involved in major exploits.
We used to have DAI, which was fully decentralized and over-collatoralized by Ethereum tokens (the native currency of the platform DAI is rooted on) - but the founder mysteriously died as the DAO was taken over and made to begin collateralizing DAI against USDC and USDT, ironically.
It is a shame how far crypto has fallen culturally that this stablecoin business is some niche story. Most people are in it for the money, but many good people are not.
The blacklists need to exist as per regulations though.
I mean, unlimited Tether can be created or destroyed at the whim of some guy with a big button somewhere. The promise of crypto being the embodiment of true distributed governance went out the window with USDT ages ago.
1/ Handling the custody of secrets by the company. The attackers first attacked and accessed a former Ledger employee with official Ledger account secrets. This is where secrets were mismanaged since the actual company secrets should never be in the hands of former employees.
2/ The attack could occur on an actual employee so they should employ ways to be protected against this kind of attack.
3/ The use of CDNs should have security measures in place. This is one of the most common attacks nowadays.
If one is to make crypto really decentralized, relying on a small number of authorities for security seems contrary and maybe poisonous to that goal.
[+] Analyzing repo-pkg src code match.... N/A [Coming soon!]The fact that an attacker was able to pull this off against a _secure_ hardware device is shocking but not surprising. The mechanism by which they did it is interesting and fairly insidious. Unlike a lot of other attacks that will publish the malware to the registry, this one pulls the payload from a CDN. So, static analysis of the loader (i.e., the intermediary package on npm) is unlikely to yield sufficiently interesting results. Solely focusing on the obfuscation angle is also not of particular use since quite a bit of packages are obfuscated on npm (like, a surprising amount of it. In Q3 2023 we saw over 5,000 _new_ packages shipped with some form of obfuscation).
Nonetheless, our automated platform pinged us this morning about some changes to this package and our research team has been digging into it to determine the impacts.
With that said, we've produced (and open sourced!) several tools that aim to help with software supply chain style attacks:
1. Birdcage is a cross-platform embeddable sandbox [4]
2. Our CLI is extensible and integrates Birdcage so you can do things like `phylum npm install...` or `phylum pip install...` and have the package installations be sandboxed [5]
We've also got a variety of integrations [6] along with a threat feed of software supply chain attacks (of which the Ledger package and other APT attacks have appeared).
Happy to answer any questions! A collective of us are active in Discord (https://discord.gg/Fe6pr5eW6p), continuing to hunt attacks like these. If that's something that interests you, we'd love to have you!
1. https://blog.phylum.io/encrypted-npm-packages-found-targetin...
2. https://blog.phylum.io/junes-sophisticated-npm-attack-attrib...
3. https://blog.phylum.io/rust-malware-staged-on-crates-io/
4. https://github.com/phylum-dev/birdcage