To be honest, we usually bill out by time rather than code base size.
To determine costs, we: * look at the application size * estimate how long it will take us to get good coverage * take in all the other factors (source provided vs blackbox) * then we give an estimate based on how long we think it will take
I must say that I honestly believe that what we provide is totally worth the money. We work really hard to provide a clear assessment of the problems with recommendations on how to fix them.
We aren't an "automated tool" shop, we actually look at the app and understand how it works and then see how to break it.
But to actually answer your question, having NO IDEA what your application actually is: (Keep in mind I am not the actual guy who does the bids, I do assessments and training. No promises, please don't fire me, etc.)
Prices vary drastically by project, and it would really depend on what we were looking at.
A good firm will help you do this, gratis, if you're serious about funding the work. We do it "on spec" for most of our clients, even though that work sometimes ending up helping a competitor deliver the project.
It's fine if firms ask you for lines-of-code counts, but if that's the only question they ask, I'd consider that a red flag.
I'd turn away any client who asked me to do appsec work, because I don't think I'd produce work of a sufficient quality to justify the sort of rates I charge, but I do think I'm probably good enough to roughly scope appsec projects on technologies I understand well. Example: Appointment Reminder is an architecturally simple Rails application. I think an audit of the marketing site, application, and architecture would reasonably require probably 1 to 2 billable weeks depending on how much I asked you to plumb e.g. line-by-line HIPAA requirements, and that would probably run in the $4k to $12k region based on my understanding of prevailing rates for appsec work. (I'm sure that if I had $25k budgeted for that audit many firms could find a way to get me my money's worth for every penny of that budget, by the way.)
If someone offers you a $4k week, make sure they know they're cutting you a deal.
They were extremely easy to work with, and very fast about getting stuff to us and verifying when it was fixed, and I felt like we definitely got more than our money's worth.
A+, would recommend, will hire again.
1) app-agnostic bugs, such as XSS/CSRF and other blatant issues
2) app-specific bugs such as access bypass, goto-fails, other obvious bugs like eval(params[:serialized]), security measures switched off, mass assignment :)
3) complex bug chains. Usually I end up with account hijacking or similar severity bugs by chaining few of unrelated and barely exploitable bugs, such as redirects, cookie encodings etc. This requires at least a week (which is $12k if you work with me).
4) infinity. Checking some unpopular ruby gems project uses. Checking popular ones. Checking rails codebase to be sure methods don't have "magic" arguments. Nobody goes that far usually, because attackers will have to do 2-4x more work to get same bugs you may find.
TL;DR, for quick & budget auditing a website like npm $3,200 and one day of work is enough, for any medium sized website people should take 1+ week.
Do people now really believe that there are definitely no security vulnerabilities related to the npm registry? Or any other type of registry, or website or application for that matter?
People have completely unrealistic expectations about security. Every time you had a significant amount of new functionality, or even a very insignificant amount, you could have introduced a new security vulnerability.
Basically this other company Lift or whatever could have two full time engineers doing security audits for npm from here until npm is done, and you could still have some other hacker who was thinking differently come up with a vulnerability in some new feature that they missed.
Its great to have the attitude that you are going to make a serious effort, but totally unrealistic to think that you are going to do one security audit and then there will be no more vulnerabilities. And also really makes no sense to make a big deal about it or give them grief or for them to even feel bad. If you think that way then you don't understand security.
You see all of these large projects having security issues not because they are full of negligent or sloppy engineers, but because security takes a lot of resources and is very difficult. The security firms will certainly suggest that engineers are negligent, of course. That ensures that they will continue to get new clients. But the reality is even with a lot of resources dedicated to just security, projects can easily have new vulnerabilities.
So that's great that they are getting regular security audits now. They are ahead of the curve.
EDIT: I notice I have been buried without anyone bothering to even respond. If you disagree, say why you disagree.
While this is true, there are certain classes of software for which vulnerabilities are a bigger deal and thus should be treated much more seriously, which includes much better up-front analysis of potential attacks and decent disclosure systems for when an attack (yes, almost inevitably) occurs.
Software distribution systems are pretty high up on the list for which exploits are A Big Deal because of how an exploit within them can very quickly spiral out of control (eg. attacker injects malicious code into very widely used module that is distributed from your repository -- now you're responsible for a lot more security problems than just the original problem on your site).
IMO this response is a good one; not over the top at all but gets the point across that they take this situation very seriously, which they should.
(FWIW, I upvoted you, so just because I partly disagree don't lump me in with those 'burying' you; I think the question you raise is entirely valid, but I do think this was a serious situation and the response was warranted).
It's just the sad state of the 'industry'. As soon as some armchair warrior finds something remotely wrong with whatever, they'll go nuts on you and you need this sort of touchy-feely PR nonsense to placate the comic book shop types. Hats off to these guys for being level-headed enough to be able to play the game this way - I couldn't do it any more, I'd go bonkers over overhead like this.
For those who are especially serious about their careers, reputations, budgets and power, incidents like this involving the technologies they hyped and pushed through can be disastrous. Now they're seen as being very wrong about something very important, and this in turn makes them extremely angry.
They know that their competitors within the business will use this incident against them. Their next initiative will surely face comment like, "Why should we listen to you after the npm disaster?", and they will face a much harder battle if their decision involves any controversy or doubt at all.
There are the inevitable almost impossible to fully plan for security vulnerabilities (e.g. in code outside your control) and then there are vulnerabilities due to extremely poor process at a fundamental level (like sql and html injection).
They knew they needed to have an audit and have intrusion detection when they started, four years ago.
Don't worry. Some of us agree with you. Sadly I'm one so I can't explain as to why you got buried.
TL;DR always use a templating engine that makes you think about XSS and don't allow unsanitized user-provided HTML through raw.
It's sad that you even need to say this, both for the fact that Javascript sandboxing is so terrible and the fact that developers aren't aware of the hazards of just blindly taking user-provided HTML.
[1] https://github.com/chjj/marked/commit/904c71b7713979b01d5bc5...
[2] https://github.com/npm/npm-www/commit/a1ed923870609b578fcde4...
edit: On closer inspection it looks like it may have been a problem with the html sanitizer[3] it used as opposed to the marked `sanitize` option (which is not used at all). I guess my conscience is clear here at least.
[3] https://github.com/npm/npm-www/blob/master/models/package.js...
We have no reason to assume they haven't. The 0-day market is extremely lucrative, and they trade exploits rarely known to the wider community.
http://blog.npmjs.org/post/78165272245/more-help-with-self-s...
> * We fixed it on February 17th
the fix scares the shit out of me:
https://github.com/isaacs/st/commit/5a0c1886737a20d78ae00b61...
Properly escape all relevant html entities
Avoid problems with files named things like '<img>' and so on.
- var name = f.replace(/"/g, '"')
+ var name = f
+ .replace(/"/g, '"')
+ .replace(/>/g, '<')
+ .replace(/</g, '>')
+ .replace(/'/g, ''')Jesus tapdancing christ that is seriously scary to see in something that's allegedly "all totally secure now, for really reals". More so the fact that such simple sanitization was missing for so long.
Well, I guess I'll put off learning node a bit longer then.
The more serious fix occurred here: https://github.com/isaacs/st/commit/6b54ce2d2fb912eadd31e2c2...
And here: https://github.com/isaacs/st/commit/6d6100eec8b19e2774a6f2bb...
With some icing on the cake here: https://github.com/isaacs/st/commit/8b2f212f64b762e351f311f4...
Also, as a general rule:
ANY SECURITY PATCH THAT IS A REGEX IS NOT A SECURITY PATCHLet's not pretend that security incident post-mortems are anything new; they aren't. There's not really even anything special about this one.
I don't really see much point in applauding these guys for doing what's perhaps the minimum we should expect from them in this situation.
.replace(/>/g, '<')
.replace(/</g, '>')
.replace(/>/g, '>')
.replace(/</g, '<')Has there been any talk within the node community about auditing node modules themselves? Maybe start with the most popular? I could see this being popular with enterprise development, etc.
I want to say that Strong Loop made noise about doing something like this, but I haven't seen much on it of late.
We are accepting contributions from the community to build the tools that get the job done efficiently and to audit modules, disclosing vulnerabilities in a responsible manner.
I wonder if the npm, Inc. team told ^Lift about the disclosed vulns before ^Lift's own audit completed. I can imagine being tempted to see whether they'd discover it themselves, to gain more evidence on how comprehensive the audit was.
As part of the audit, ^Lift audited a lot of the third-party modules we use, and notified the authors of those packages separately (I'm not aware of the details, but I don't think there was anything major).
The issue that post refers to (it is a little unclear, because we ourselves were a little unclear what had gone wrong at that point) is when we broke older clients by moving to a non-GlobalSign cert. We cleared that up here: http://blog.npmjs.org/post/78165272245/more-help-with-self-s...
We had already planned to move to a new cert before the security disclosure, and hadn't anticipated the size of the problem with a non-GlobalSign cert, so although the two happened pretty much simultaneously, they weren't really related.
Thanks for doing!
Great job guys and very responsible choices.