* The people who are good at QA usually have the skills to work as a developer. QA is generally not as desirable as development to a young developer, so why do it?
* Many QA people are almost completely useless and exist just to generate paperwork so that the company can cover its ass. Warning signs for bad QA people seem to be certifications and fear of anything that does not strictly adhere to classic waterfall development. The bulk of all defects are found by a small number of people on my team.
* Most developers are horrible at testing and writing unit tests. We have thousands of unit tests and the main thing that they catch is that someone did not update the test after refactoring code.
Sadly, those people are not as common as they should be. There's honestly a huge market opportunity for high quality testers and it's not something that requires years of training. Heck, maybe we can solve the unemployment problem by just convincing companies to care more about quality software! ... probably not, though.
Even though most orgs do not officially treat the positions as an entry level (QA), and a mid-level (junior dev) - I think the general unspoken understanding is that devs are higher on the food chain. The two positions need to be established as more distinct and less like rungs in the same ladder.
(I put QA in quotes because their testing structure does not seem to be at all similar to most companies.)
I wish this were less true, but it's totally spot on. I have definitely been guilty of this sin...
"We need a test framework" are terrifying words to me. It means that I (the dev) will have no one at my back. It means that tests will be written by junior QA engineers (the senior ones are off writing and maintaining the framework, right?). It means that I am responsible for my own tests, because /no one/ else besides the customer will be testing my code.
That's generally all right; I think I do a good job. But not everyone does; some devs think they can toss code over the fence to a group that will tell them if it worked or not.
Everyone is responsible for quality; Devs must do unit tests, but they have blind spots and that is what QA is for, namely higher-level assurance that things are working as intended.
If they failed, these SDETs will move to PM role.
I'm not saying all SDETs are like this, but join MS and you'll see quite a few of them.
Google probably have some too but lately I noticed that their approaches are different: instead of building a large automation frameworks, they build tools to help developers be more productive and produce less bugs even before the automation testing happens.
Everyone should always code and test their code like that, right?
That MMO that is getting away with using customers as QA, and only fixing bugs that NEED fixing? I bet they're making a nice profit. I'm sure the customers are annoyed, but in my experience, gamers won't quit playing over bugs. Bad customer service, price hikes, etc... Sure. But not bugs. Even if they stick around for years.
However, business applications are a different story. Businesses can't afford to be stopped from doing what they need to. And a QA person makes all the difference in the world there.
For those that don't understand why, a programmer can't find their own bugs because they've already done their utmost to find bugs in the code. They've already stretched themselves to the limit before they sent the code to QA. You can train, coddle, beat, or otherwise influence them and only get a few more bugs found by programmers. It's much better for everyone's sanity and wallets if you just pay for proper QA.
You've sure got that right. Our application conducts real-time purchasing events (reverse auctions) where bidders are ranked based on a couple of different configurable algorithms. An error in calculation means incorrect ranking, which means the entire outcome is invalidated. Considering that it can take weeks to put together a purchasing event of this type means that there is a lot riding on that single series of calculations.
Simply releasing and finding bugs in production would be financial suicide. QA isn't even a question in our business. Everything is tested thoroughly when it goes out the door.
Having worked at many companies that didn't believe in QA at all, I think having a team of testers is beneficial not only for software quality, but also filtering potential bug reports from the field. When a client reports a potential, QA does the initial leg work in duplicating the issue, then passes it off to engineering.
I think a lot of bugs made in day-to-day coding could be found given such methods. It would be best to integrate something like that into the compiler (and IDE) as it has the necessary information and structures. I believe the clang team is working on this (http://clang-analyzer.llvm.org/).
I guess it'd require defining "higher level" pre- and post conditions and assertions and such in the code...
Not a substitute for "proper QA", but I think sometimes that "software engineering" is simply dabbling in the dirt as a engineering discipline, a computer could probably run the function you are editing right now 1000000x per second and/or verify the hell out of it while you are thinking about the next letter to type :p
I've got into a habit of defining pre and post-call checks on just about everything. It's knocked down my regression and defect rate quite considerably. Assumptions are dangerous - so always check them.
Microsoft are actually leading this if you ask me with PEX:
Try it here: http://www.pexforfun.com/
i would never say that "we don't need no stinking QA", but i do believe that teams who live by "think and analyse" need QA a lot less than teams who don't.
[1] http://www.dustingetz.com/linus-think-and-analyze-motherfuck...
I have had my own problems with incompetent QA "engineers," but I do think an excellent QA team helps create an excellent product.
Greg Kroah-Hartman has given a few talks on this, and from the dev perspective a lot of it boils down to "make small, discrete changes, compile, and just f*cking ship it". Testing happens among users.
But, because this is free software, there are a substantial number of systematic testers among the users, whether this is commercial distros (Red Hat, Suse, Ubuntu), noncommercial ones (Debian), organizations with a substantial interest in the quality of Linux code (IBM, Intel, AMD, from which we've got some pretty extensive test suites), proprietary software developers (who hate it when a kernel change breaks their code), independent researchers (I know there's a lot of software quality research based on Linux), and even some performance bashing by Linux haters ... which results in better Linux code.
And there are a lot of users who end up discovering bugs as well.
But the coverage is pretty good. Even if it's not explicitly rolled into the dev process itself.
Similar situations apply to other projects, though Linux for its size, scope, and user base tends to do better than many.
I'd say that if you have a T&A team it would be great to measure and publish your defect rate. Of course, if you just have better than average coders, then naturally your defect rate can be low.
But even a low defect rate does not preclude having a formal QA process. Even the best coders in the world can ship the wrong product when the specifications are wrong.
I am not claiming that a bug count of zero is possible with most software, but testing quality in at the end is the wrong approach. Nothing affects quality as much as the skill level of the developers.
So much of our thinking about roles and teams and efforts and project management is still in last century's approach to monolithic one-shot releases.
On the other hand, when you are building your MVP, progress is not built features, but validated learning. So QA tasks takes more of an approach of building measuring tools (which have to be done by the people building the MVP).
Even on a later stage startup, QA takes a whole new role, specially when you start doing continuous deployment and start developing immune systems. These can get quite complicated (think of Netflix Chaos Monkey[1]), and having a dedicated team can be beneficial.
What we don't need is a 20th century waterfallish QA. But ensuring the quality of a product[2] has never been more important.
[1] http://www.codinghorror.com/blog/2011/04/working-with-the-ch...
[2] http://www.ashmaurya.com/2011/06/your-product-is-not-the-pro...
[1] http://jessicamah.com/the-ceos-job-part-2
[2] http://blog.eladgil.com/2010/12/5-myths-to-building-awesome-...
The company had literally fired the whole testing department previously for saying the product wasn't ready. With those annoying testers out of the way, out the door it went. And promptly had multiple emergency point releases the following week to fix showstoppers cropping up at the customer sites.
That got them to create a testing department again - but it was still without honor in the company. And firing the whole group again was often discussed, as they were _still_ delaying releases by finding bugs!
I departed. They got eaten in a merger several months later.
...and again in 2010: http://www.joelonsoftware.com/items/2010/01/26.html
...(with more than a couple of other mentions in between). I've worked at ISO 900x shops that worked to the letter of the law, so to speak, without actually understanding what software QA was all about. A really good tester is "chaotic neutral", and the best we can come up with as testers of our own work is "lawful evil".
That is parts like coding and deployment were necessary because they actually were important and helped move business ideas to the customer. QA to them was a redundant step. This coming from a shop that avoided automated testing because it was too expensive.
I was disappoint. QA to me is insurance against being human (which we all are). The ended up not getting rid of QA but the fact that they seriously considered it was shocking.
IMO QA is something that should be included throughout the task lifecycle. Sure, after a task is marked as complete there should be some sort of usability testing, but that isn't the sole job of QA. They should be working before tasks are even started by defining the exit criteria and writing integration tests. Giving QA its own step is wasteful because its effectively pigeonholing the job.
QA is nice, but it's a finite resource that you should save for things that actually matter, not stupid things that a computer can fix for you in 30 seconds.
Can't believe I'm defending Java's type system but I'm pretty sure you can only call bar on a null Foo (Foo f = null) if you cast it to a Bar first. Otherwise it'll fail to type check. What Java does allow you to do is call foo on a null Foo and in that case you get a null pointer exception at runtime.
When you're doing QA on a product, do you worry about border cases? I found quite a few bugs related to time syncing and inadequate safeguards in a project I worked on - but they were difficult to reproduce without a macro.
I'm still not sure whether people were happy or angry with me, for reporting all those bugs.
One company I worked at had both buffer overflow and timing problems and didn't dedicate the time to fixing either properly so easily half the bugs we found were the same giant ones (and thus always ignored even if crash-level). This slowed us incredibly as you can imagine. The timing bugs could cause really anything and the buffer overflows prevented us from stuffing commands properly which kept us from ever reliably reproducing the timing bugs.
Eventually one of the lead devs fought through the backlog and implemented scripting into the product itself and we were able to get a much better handle on things through more intensive and repeatable testing.
These days I expect the developer has written specs for their code at the unit level. How else do they know what they claim to be delivering? So the job of QA (or, as I'd prefer - just another developer playing QA) focuses more on testing the stuff the weird stuff, edge cases, etc.
Automate everything. Writing a system to parse screenshots to decide if your windows drew properly is cheaper, and sooner than you think, than manual testing. Especially if you count the cost of bugs you miss through inattention.
And by "real engineer", I mean one whose "engineer" is preceded by one of: agricultural, chemical, civil, electrical, environmental, industrial, mechanical, nuclear, or petroleum, and who is in possession of a state-issued license.
QA needs to be done, and it needs to be funded. Everything you put into will return in less development time (overall, not necessarily in the short term), and more satisfied users.
I work at a company where they have a 5 year old product, and the mentality at the time was "Yes I know it doesn't make sense but if the user does X, then it could cause the server to crash, so let's do Y." This grew to a series of matrices that has now turned the act of adding 1 single feature into a 3 pages worth of "If A then B, otherwise C". This has made the QA effort in terms of testing incredibly complex, and even developers get confused as to what is supported and what isn't.
For example, there is an instance where copying a directory takes a different code path than copying a single file. Why? Because they wanted to "special case" this situation. It makes the functionality and behavior utterly unpredictable.
The takeaway that comes with experience is don't spend too much time trying to prevent your users from doing stupid things. If they want to do stupid things and the system crashes, it's okay to say "Don't do it, otherwise it will crash." This keeps your feature set, your code, and your QA effort much cleaner and more maintainable over the long haul.
Users and bosses thinks they have done more than their duty when they said you that "it doesn't work", "nothing work" without more details. You have to run after them to get more context and detailed information.
I love QA.
One of the major problems can be keeping the newer people & graduates interested in the testing & QA process. There can be a lot of 'grinding' through test cases and I guess it takes a certain mentality to get through it on some days.
I've been a QA, QA Developer, Developer, and Integrator during my internships.
QA is often the less honoured role of all. This creates a stigma that QA is some sort of crappy role to be in.
Being second class citizen requires more support than ever to work at the top day to day. Unfortunately, most people are not built that way so quality degrades because QA was treated like crap.
This leads to less advancement in the QA community because it's not a good role to be in. You only see a few books and a few noted experts in the QA community that continue to push better practices.
At the end of the day, it's a loss for all of us.
I wish there's none or minimum differences between QA and Developer as I agree with one of the comments here that if you do that, there will be more jobs created and IT professionals can experience different roles within our world.
My opinion is that QA should have a wide range of skills from load/performance testings (that means knowing the tools and statistics), automation testings (that means if DEV created hard-dependency issue that is not easily testable, QA should step in and teach DEV the patterns and best practices to avoid such situation), QA should understand business practices to beat up stupid Business Analysts.
More importantly, QA should keep track deficiencies and ineffectiveness during the development and potentially offer solution (or can be done through group discussions).
Unfortunately, that's not how the world works. Telling someone that he's not efficient or pointing out where the most bugs occurred when the team lead is the one whose responsible for the feature isn't desirable.
Not to mention that certain type of companies don't necessarily get the most ROI from having QA (if your typical Web 2.0 social networking tagging machine had a few issues, users probably complain but everyone will move on).
The only reason they needed QA is the software was a giant toilet-clogging turd. There were no formal unit or integration test cases so it was used to catch regressions against massive test plans knocked up in Excel, Word and god-knows what else.
