System Testing

General Testing Books particularly recommended books are stated below:

Black Box

Boris Beizer, Black Box Testing, Wiley, 1995

Rick D. Craig and Stefan P. Jaskiel, Systematic Software Testing, Artech House Publishers, 2002

* Cem Kaner, Jack Faulk, Hung Quoc Nguyen, Testing Computer Software, 2nd Ed., International Thompson Computer Press, 1993

Cem Kaner and James Bach, Black Box Software Testing (Professional Seminar), 2002

Cem Kaner, James Bach, and Bret Pettichord, Lessons Learned in Software Testing: A Context Driven Approach, Wiley, 2002

Scott Loveland, Geoffrey Miller, Richard Prewitt, Jr, Michael Shannon, Software Testing Techniques: Finding the Defects that Matter, Charles River Media, 2005

Glenford Myers, The Art of Software Testing, Wiley, 1979

Glenford Myers, Tom Badgett, Todd M. Thomas, and Corey Sandler, The Art of Software Testing, 2nd Ed, Wiley, 2004

Software Program Managers Network, The Little Book of Testing, Volume I, Overview and Best Practices, Software Program Managers Network, 1998

Software Program Managers Network, The Little Book of Testing, Volume II, Implementation Techniques, Software Program Managers Network, 1998

* James A Whittaker, How to Break Software, Addison-Wesley, 2003

White Box

* Boris Beizer, Software Testing Techniques, 2nd Ed., Van Nostrand Reinhold, 1990. Old, but the ultimate reference. Binder quotes Beizer.

Bill Hetzel, The Complete Guide to Software Testing, 2nd Ed., Wiley, 1988

Test Case Design

Rex Black, Pragmatic Software Testing: Becoming and Effective and Efficient Software Professional, Wiley, 2007
Covers much the same test design strategies as the Copeland book.

* Lee Copeland, A Practitioner’s Guide to Software Test Design, Artech House Publishers, 2003.
This is an excellent book on test case design, covering both white box and black box techniques.

Cem Kaner, "What is a Good Test Case?", Software Testing and Quality Engineering September/October 2003

Object Oriented

* Robert Binder, Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison Wesley, 2000. This book is good for both white box and black box techniques applied to object oriented systems.

Defect Analysis

Data on number of defects in a typical system, percentage of time spent on testing, rate of finding defects, etc.

General Metrics

Robert D Austin, Measuring and Managing Performance in Organizations, Dorset House Publishing, 1996

Barry Boehm, Software Engineering Economics, Prentice Hall, 1981

Barry Boehm, "Industrial Software Metrics Top 10 List", IEEE Software, Vol 4, No 9, September, 1987, pages 84-85

S D Conte, H E Dunsmore, V Y Shen, Software Engineering Metrics and Models, Benjamin/Cummings, 1986

Robert B. Grady, "Successfully Applying Software Metrics", IEEE Computer September 1994

Robert B. Grady, "An Economic Release Decision Model: Insights into Software Project Management.", Proceedings of the Applications of Software Measurement Conference, 227-239, 1999. Orange Park, FL: Software Quality Engineering.

Douglas Hoffman, "The Darker Side of Metrics", 2000

Capers Jones, Programming Productivity, McGraw-Hill, 1986

Capers Jones, Applied Software Measurement, 3rd Ed., McGraw-Hill, 2008.

Capers Jones, Software Quality, Analysis and Guidelines for Success, Thompson, 1997.

Capers Jones, "Software Engineering: The State of the Art in 2005", 2005

Capers Jones, "The Impact of Poor Quality and Canceled Projects on the Software Labor Shortage", 2005

Jones, Capers, "Conflict and Litigation Between Software Clients and Developers", March 1, 2007. Published in IT Metrics and Productivity Journal, August 7, 2007.

Cem Kaner, "Rethinking Software Metrics", Software Testing and Quality Engineering March/April 2000

Brian Marick, "How Many Bugs do Regression Tests Find?"

Lawrence H. Putnam and Ware Myers, Measures for Excellence, Yourdon Press, 1992

Shull, et al, 2002. "What We Have Learned About Fighting Defects," Proceedings, Metrics 2002. IEEE; pp. 249-258.

Shull, Forrest and Roseanne Tesoriero, "What We Have Learned About Fighting Defects, Results of the METRICS02 workshop" 2002

United Kingdom Software Metrics Association, "Quality Standards Defect Measurement Manual, Release 1.a", October 1000

Wagner, Stefan, "A Literature Survey of the Quality Economics of Defect-Detection Techniques," Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering ISESE '06, 2006, pp 194-203.

Vil - does code metrics in the .NET environment

Testing Specific

Susan S. Brilliant, John C. Knight, Nancy G. Leveson, "Analysis of Faults in an N-version Software Experiment"", IEEE Transactions on Software Engineering, V16, N 2, February, 1990 A analysis on why it makes sense to test at the boundaries.

Barry Boehm and Victor R. Basili, "Software Defect Reduction Top 10 List", IEEE Computer, January 2001

Cem Kaner, "Measurement of the Extent of Testing", Pacific Northwest Software Quality Conference 2000. The paper is at pages 108-144 and the slides at pages 145-172 in the proceedings document

Brian Marick and Steve Stuckenborg, The Test Manager at the Project Status Meeting", 1997
This paper is very good at suggesting how a testing manager can present numbers which show the project manager the status of testing.

Erik Simmons, "When Will We Be Done Testing? Software Defect Arrival Modeling Using the Weibull Distribution", 2000. The paper is at pages 194-210 and the slides at pages 211-243 in the proceedings document

Reviews and Inspections

Some books call reviews 'static testing'. In any case, reviews and testing work well together. They tend to find different types of defects, and reviews tend to find defects more cheaply than waiting for testing.

Inspections

Jeff Carver, "Improving Software Inspections by Using Reading Techniques" in A Quantitative Approach to Software Management and Engineering

Evert, Christof and Philip De Neve, "Surviving Global Software Development", IEEE Software, March/April 2001, published in IT Metrics and Productivity Journal, June 5, 2007. They broke projects up into chunks that could be developed relatively independently at one location. They found that collocated teams achieved an efficiency improvement during initial validation activities of 50%, i.e. for the same amount of defects in design and code, those teams sitting in one location needed less than half the time for defect detection.

Ebert, Christof, Casimiro Hernandez Parro, Roland Suttels, Harald Kolaaczyk, "Improving Validation Activities in a Global Software Development", IEEE, published in IT Metrics and Productivity Journal, June 5. 2007. Collocated teams found 25 defects/1000 logical source statements, non-collacated found 13. Collocated teams spent .08 hours/defect, non-collated .33 hours/defect. Thus collocated teams spend less than half the time for defect detection. And almost twice as many defects are detected. This saved greater than 10% of the project cost.

Michael Fagan, "Advances in Software Inspections", IEEE Transactions on Software Engineering, Vol 12, No 7, July, 1986

Michael Fagan, "Design and Code Inspections to Reduce Errors in Program Development", IBM Systems Journal, Vol 15, No 3, 1976

Tom Gilb and Dorothy Graham,, Software Inspection, Addison-Wesley, 1993

R R Larson, "Test Plan and Test Case Inspection Specification", IBM Corp., Tech. Report TR21.585, April 4, 1975

Karl Weigers, Peer Reviews in Software, A Practical Guide, Addison-Wesley, 2002

David Wheeler, Bill Brykczynski, Reginald Meeson,, Software Inspection - An Industry Best Practice, IEEE Computer Society Press, 1996

General Reviews

Boehm, Barry and Richard Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Boston,Mass.: Addison Wesley, 2004 This is a book on agile processes, but it has statistics about the savings from doing reviews rather than relying on testing

Barry Boehm, "Software Defect Reduction Top 10 List", IEEE Computer, January 2001

Jack Ganssle, "Faster, Better Code", The Embedded Report, Miller Freeman, August, 1998
In the Break Points section

Defect-Cost Increase

Michael Fagan, Design and Code Inspections to Reduce Errors in Program Development, IBM Systems Journal, Vol 15, No 3, 1976

Humphrey, Watts S., Terry R. Snyder, and Ronald R. Willis 'layout-grid-mode: line'>. Software Process Improvement at Hughes Aircraft. IEEE Software 8, no. 4 (July): 11-23, 1991.

Leffingwell, Dean. Calculating the Return on Investment from More Effective Requirements Management, American Programmer, 10(4):13-16, 1997.

Willis, Ron R., et al,. Hughes Aircraft’s Widespread Deployment of a Continuously Improving Software Process, Software Engineering Institute/Carnegie Mellon University, CMU/SEI-98-TR-006, May 1998.

Grady, Robert B. An Economic Release Decision Model: Insights into Software Project Management. In Proceedings of the Applications of Software Measurement Conference, 227-239, 1999. Orange Park, FL: Software Quality Engineering.

Shull, et al.What We Have Learned About Fighting Defects, Proceedings, Metrics 2002. IEEE; pp. 249-258, 2002.

Boehm, Barry and Richard Turner. Balancing Agility and Discipline: A Guide for the Perplexed, Boston, Mass.:Addison Wesley, 2004.

Testability of programs

Cem Kaner, James Bach, and Bret Pettichord, Lessons Learned in Software Testing: A Context Driven Approach, Wiley, 2002 has good coverage of testability, see pp 123-124 and 149-150.

DiMaggio, Len, "Looking Under the Hood", STQE magazine, January-February 2000, pages 24-31.

Johnson, Karen, "Mining Gold from Server Logs", STQE Magazine, January-February 2001, pages 62-67.

Marick, Brian, " Using Ring Buffer Logging to Help Find Bugs".

Formal Methods

Formal Methods provide mathematical proofs of program correctness, using special languages, like Z (pronounced 'zed')

Ross, Philip E., "The Exterminators ", IEEE Spectrum, September 2005

Quality Risk Analysis

Boris Beizer, Software Testing Techniques, 2nd Ed., Van Nostrand Reinhold, 1990. See Chapter 1 for the origins of risk-based testing.

Rex Black, Pragmatic Software Testing: Becoming an Effective and Efficient TestProfessional, Wiley, 2007. Part II, Chapters 5-7 are on risk-based testing.

Rex Black, Managing the Testing Process, 2nd Edition, New York :John Wiley and Sons, 2001. See Chapter 1.

Rex Black, Critical Testing Processes, Reading, MA Addison Wesley, 2004. See Chapter 2.

R.D. Craig and S.P. Jaskiel, Systematic Software Testing, Norwood, MA Artech House, 2002. See Chapter 2.

Bill Hetzel, The Complete Guide to Software Testing, 2nd Ed., Wiley, 1988. See Chapter 2 for the origins of risk based testing.

Peter Neumann, Computer-Related Risks, New York Addison-Wesley, 1995

M. Pol, R. T. Teunissen and E. van Veenendaal, Software Testing: A Guide to theTMap Approach, Addison Wesley, 2002. See Chapter 11.

Failure Mode and Effects Analysis

D.H. Stamatis, Failure Mode and Effects Analysis, Milwaukee, WI, ASQC Quality Press, 1995.

White Box techniques

Coverage: control flow, data flow, statement and branch

Steve Cornett, "Code Coverage Analysis", Bullseye Testing Technology, 2005. A short document on coverage, mostly white box

Joseph R. Horgan, Saul London, and Michael R. Lyu, "Achieving Software Quality with Testing Coverage Measures", IEEE Computer, September 1994, pages 60-69, 1994

Brian Marick , "How to Misuse Code Coverage", 1997

S Rapps, E J Weyuker , "Data Flow Analysis Techniques for Test Data Selection", Sixth International Conference on Software Engineering, Tokyo, Japan, September, 1982
Dataflow techniques for test case generation

NCover- A free coverage tool for the .NET environment - does statement coverage only

NCoverExplorer includes NCover.

Microsoft’s Visual Studioincludes a coverage tool built in.

Cobertura - open source coverage tool for Java. Does both statement and branch coverage. Cobertura is a fork of JCover.

Elliotte Rusty Harold, "Measure test coverage with Cobertura", IBM DeveloperWorks, May 2005.

Clover is a Java test coverage tool developed for commercial application that is free for noncommercial activities Cenqua's Clover is a somewhat more polished, payware test-coverage tool that does essentially the same job as Cobertura with fewer rough edges. Clover 2 ($1200 and up) is now available from Atlassian, with new features.

Source Monitor is a free tool that does cyclomatic complexity and some other metrics. For C++, C, C#, VB.NET, Java, Delphi, Visual Basic (VB6) or HTML. Does method and function level metrics for C++, C, C#, VB.NET, Java, and Delphi.

JCover is a Java test coverage tool

Cyclomatic Complexity

S D Conte, H E Dunsmore, V Y Shen, Software Engineering Metrics and Models, Benjamin/Cummings, 1986

Curtis, B., S.B. Sheppard, and P. Milliman , "Third time charm: Stronger prediction of programmer performance by software complexity metrics", Proceedings of the 4th international conference on Software engineering, July 1979, pp 356-260.

T J McCabe , "A Complexity Measure", IEEE Transactions on Software Engineering, Vol 2 No 4, December, 1976

Schroeder, Mark, "A Practical Guide to Object-Oriented Metrics", IT Pro Nov/Dec 1999, pp 30-36, 1999.

Edmond VanDoren, "Cyclomatic Complexity" SEI

Code metrics tool NDepend

McCabe QAMetrics including cyclomatic complexity and essential complexity help identify where a program is more likely to contain errors. Platforms: Sun - Solaris 2.5.1+ , PC - Windows NT 4.0/95, IBM - AIX 4.2+, HP - HP-UX 10.2+, SGI - IRIX 5.3+

CMT++(Complexity Measures Tool for C/C++) calculates the basic McCabe, Halstead and lines-of-code metrics for C and C++ code. Platforms: Windows 2000/NT/9x, HPUX, Solaris, Linux.

METRIC works as a stand-alone product or as part of the TestWorks/Advisor tool suite. METRIC automatically computes the Cyclomatic Complexity metrics, which measure logic complexity in routines; and size metrics, such as number of lines, comments and executable statements. Platforms SPARC SunOS 4.1.3; SPARC Solaris 2.3, 2.4, 2.5, 2.5.1, 2.6; x86 Solaris 2.4, 2.5; RS-6000/AIX 3.2.5, 4.1.2, 4.2; HP-9000/7xx HP-UX 9.05, 10.10; HP-9000/8xx HP-UX 9.04, 10.01, 10.10; x86 SCO/ODT 2.0, 5.0(3.0); SGI Irix 5.3, 6.2; DEC-Alpha OSF1 3.2, 4.0; NCR 3000; SCO/UnixWare 2.1.1; DOS.

Metrics Tools for Java, VBScript collect standard metrics on source code, including SLOC, NCSLOC, Cyclomatic Complexity, Decision Density. Platforms : Windows/NT/2K/XP.

Understand for Java, generates cyclomatic complexity metrics and also generates a bunch of other metrics.

Understand for C++ does cyclomatic complexity and other metrics for C++.

Source Monitor (for C#) does cyclomatic complexity and other metrics for C#.

Listing of static analysis tools: Includes metrics tools and cyclomatic complexity tools as well as tools like lint. Gives some basic information about each listing.

MC/DC

John Chilenski, Steven Miller, "Applicability of Modified Condition/Decision Coverage to Software Testing", Software Engineering Journal, September, 1994

FAA, Software Considerations in Airborne Systems and Equipment Certification, Document RTCA/DO-178B, RTCA, Inc. December, 1992
This book lists the requirements for testing avionics, depending on how life-critical the system is. MC/DC is required for the most life critical systems.

Kelly J. Hayhurst, Dan S. Veerhusen, John J. Chilenski, and Leanna K. Rierson, "A Practical Tutorial on Modified Condition/Decision Coverage", NASA, 2001

Leanna Rierson, Kelly Hayhurst, and Dan Veerhusen, "Modified Condition/Decision Coverage (MC/DC): An Interactive Video Teletraining Course", FAA, May 2001

Validated Software is a vendor of coverage tools for use with RTCA DO-178B testing requirements, i.e. these tools can do MC/DC coverage

Black Box Techniques

Requirements

Donald Gause, Gerald Weinberg, Exploring Requirements: Quality Before Design, Dorset House, 1989

Dean Leffingwell, "Calculating the Return on Investment from More Effective Requirements Management," American Programmer, 1997 10(4):13-16.

Suzanne Robertson and James Robertson, Mastering the Requirements Process, 2nd Edition, Addison-Wesley, 2006

Karl E. Wiegers, Software requirements. 2nd Ed. Redmond, Wash. Microsoft Press, 2003.

Use Cases

Steve Adolph and Paul Bramble, Patterns for Effective Use Cases, Addison-Wesley, 2003.

Alistair Cockburn, Writing Effective Use Cases, Addison-Wesley, 2000.

Tool for use case and requirements: Case Complete -- tool allows information for test cases to be entered alongside the use case details, then can generate test cases.

Scenario Testing

Cem Kaner, "Pattern: Scenario Testing"

Cem Kaner, "Cem Kaner on Scenario Testing", STAR East May 2003

Soap Opera Testing

Hans Buwalda, "Soap Opera Testing", Better Software Magazine, February 2004

Model Based Testing

Ian Horrocks, Constructing the user interface with statecharts, Reading, MA: Addison-Wesley, 1999.

Harry Robinson, "Intelligent Test Automation", Software Testing and Quality Engineering September/October 2000

Harry Robinson, "Things That Find Bugs in the Night", StickyMinds.com

Harry Robinson, "Obstacles and Opportunities for model-based testing in an industrial software environment", or PowerPoint slides

Harry Robinson, "Model Based Testing", slides from tutorial at Star East 2005

Model-based testing website

Pascale Thevenod-Fosse and Helene Waeselynk , "STATEMATE Applied to Statistical Software Testing", ACM International Symposium on Software Testing and Analysis, 1993, pages 99-109. (Available in the ACM Digital Library if you subscribe)

Generic link to papers on model based testing on Geocities

Pairwise Testing/ AllPairs

Testing n-way combinations, and the effectiveness relative to testing all possible combinations

Articles and books

James Bach and P Schroeder, "Pairwise Testing: a Best Practice that Isn’t", 22nd Annual Pacific Northwest Software Quality Conference, Portland, October 2004

D. M. Cohen et al. , "The AETG system: An Approach to Testing Based on Combinatorial Design", IEEE Transactions on Software Engineering, Vol. 23, No. 7, July 1997

Richard D. Kuhn, and Michael J Reilly, "An Investigation of the Applicability of Design Experiments to Software Testing", 27th NASA/IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, 4-6 December 2002
They looked at what percent of errors are found with different levels of n-way testing

Keith Stobie, "Too Darned Big to Test", ACM Queue, February 2005, pages 30-37.

Keith Stobie talk at WSA QA SIG September 2005,"It's Too Darned Big to Test"

Delores R. Wallace and D. Richard Kuhn, "Failure Modes in Medical Device Software: An Analysis of 15 years of Recall Data", International Journal of Reliability, Quality and Safety Engineering, Vol. 8, No. 4, 2001

Tools to generate pairwise or n-way combinational test cases

AllPairsby James Bach. It’s written in PERL. (free, open source, GPL). It can only cover all pairs of features. It doesn't support any restrictions. It takes as input a tab-delimited table listing the actual parameter values of the attributes you want to test. The output is a table suitable for dumping into Excel listing the test cases, parameter value by parameter value. It also produces an index of pairs saying which test cases cover each pair. 20 dimensions of 10 features each, all pairs, requires 230 testcases.

AETGfrom Telcordia (commercial, $6000 for two seats for a year). It's web-based. It can cover all pairs (or triples or arbitrary n-tuples) of features. It supports restrictions, disallowing certain feature combinations. It can extend an existing test suite. It can often use deterministic methods (as opposed to pseudorandom) to generate test cases. 20 dimensions of 10 features each, all pairs, requires 180 test cases.

Ward Cunningham provides further discussion and the source code of a Java program to generate all pairs combinations

Jenny will do pairs, triplets, etc. You say what you want with parameters. Written by Bob Jenkins (free, open source, public domain). It covers all n-tuples of features and supports restrictions. It can extend an existing test suite. It always uses pseudorandom methods to produce test cases. 20 dimensions of 10 features each, all pairs, requires 195 test cases. It’s written in C.

Pairwise.org has a listing of available tools and some references to effectiveness of pairwise

Orthogonal Arrays : another way to generate test cases (the tools above are easier)

Robert Brownlie, et al., "Robust Testing of AT&T PMX/StarMAIL Using OATS, AT&T Technical Journal, Vol. 71, No. 3, May/June 1992, pp. 41-47.

Robert Mandl, "Orthogonal Latin Squares: An Application of Experiment Design to Compiler Testing", Communications of the ACM, Vol. 128, No. 10, October 1985, pp. 1054-1058.

Madhav S. Phadke, Quality Engineering Using Robust Design, Prentice Hall, 1989

Madhav S. Phadke, "Planning Efficient Software Tests", Crosstalk, October 1997

Madhav S. Phadke, "Design Of Experiment for Software Testing", January 2003

Phadke Associates sells tools that will generate orthogonal arrays

Here’s a website with a comprehensive catalog of orthogonal arrays

Usability Testing

Jakob Nielsen, Usability Engineering, San Francisco, Academic Press, 1993

Donald Norman, The Invisible Computer, Cambridge, MA, The MIT Press, 1999.

Jeffrey Rubin, Handbook of Usability Testing, Wiley, 1994

Security Testing

Ryan English, "What Lies Beneath: Hunt Down Security Vulnerabilities with Penetration Testing", Better Software Magazine, May 2006, page 26

Steven Splaine, Testing Web Security: Assessing the Security of Web Sites and Applications . Wiley, 2002

James A Whittaker and Herbert H Thompson, How to Break Software Security, Addison-Wesley, 2003

DevInspect from SPI Dynamics (now HP) - accelerates the construction and delivery of secure Web applications and Web services by finding and fixing security vulnerabilities during development, like unsanitized inputs, undeclared values, buffer overflows, etc.

QAInspect from SPI Dynamics (now HP) provides the appropriate Web application security testing tools for systems testing staff. Find vulnerabilities in specific usage scenarios or pages during the Initial QA process, and confirm they have been removed during final regression testing.

WebInspect - is the first and only web application security assessment tool to be re-architected to thoroughly analyze today's complex web applications built on emerging Web 2.0 technologies like AJAX, SOAP, SOA, JavaScript and Flash.

CodeScan from CodescanLabs, finds and corrects vulnerabilities in web source code. Currently it supports VBscript (.asp) and php code. Soon it will support .NET, Java and ColdFusion.

Fortify Source Code Analysis (SCA) is for source code security analysis. Its powerful, multi-algorithm analyzers automate the extremely difficult and tedious activity of identifying potentially dangerous functions, configurations and data flows within an entire code base -- even through multiple tiers and over millions of lines of code.

Fortify Tracer gives accurate, consistent answers to the key question "how many security-sensitive functions in my Web application were exercised?"

Fortify® Defender monitors custom business web applications from the inside out so it can deliver reports on who attacked, how often, the technique used and much more for every instance of a web application. Fortify Defender not only addresses PCI Data Security Standards but also key software security compliance requirements including OWASP Top Ten, HIPAA and more.

Fortify® Manager is a security dashboard and control center for Security and Development teams to centrally manage software security policies and report on the results from multiple deployments of Fortify SCA.

CxSuite from CheckMarx offers a solution for each participant of the Software Development Life Cycle:

CxAudit is used by internal and external auditors for translating business logic, company policy and organization standards into sets of queries, and assigning policies and sets of relevant queries for each specific project

CxDeveloper used by developers and team leaders for running sets of predefined and customized queries from the Integrated Development Environment (IDE) and browsing tracked security flaws directly in the code.

Team leaders use CxDeveloper for monitoring both full projects as well as reviewing each developer’s code.

Risks digest online

Web Systems Testing

Mike Andrews and James A. Whittaker, How to Break Web Software, Addison Wesley, 2006

Hung Q. Nguyen, Bob Johnson, and Michael Hackett, Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems, 2nd Ed, Wiley, 2003

Steven Splaine and Stefan P. Jaskiel, The Web Testing Handbook, STQE Publishing, 2001

Regression Testing

Cem Kaner, "Avoiding Shelfware: A Manager’s View of Automated GUI Testing", 1998.

Brian Marick, "How Many Bugs do Regression Tests Find?"

Brian Marick, "Classic Testing Mistakes", STAR ’97 Conference, 1997.

GUI Regression Testing

Chris Agruss, "Automating Software Installation Testing", Software Testing and Quality Engineering Jul/Aug 2000 (Vol. 2, Issue 4).

James Bach, "Test Automation Snake Oil".

Hans Buwalds, "Testing Using Action Words".

Hans Buwalds, "Automated Testing with Action Words: Abandoning Record and Playback".

Elizabeth Hendrickson, "The Difference Between Test Automation Failure and Success", 1998.

Cem Kaner, "Avoiding Shelfware: A Manager’s View of Automated GUI Testing", 1998.

John Kent, "The Ghost in the Machine Part 3: Data Driven Test Automation", Professional Tester, April 2003.

John Kent, The Ghost in the Machine Part 4:Advanced Automated Testing Architectures, Professional Tester, April 2004.

Bret Pettichord, "Success with Test Automation", 1996.

Bret Pettichord, "Seven Steps to Test Automation Success", 2001.

Keith Zambelich, "Totally Data-Driven Automated Testing".

Test Automation Architectures

Doug Hoffman, "Test Automation Architectures: Planning for Test Automation", 1999.

Doug Hoffman, "Mutating Automated Tests", STAR East ’00.

Cem Kaner and John Vokey, "A Better Random Number Generator for Apple’s Floating Point BASIC", 1982.

John Kent, The Ghost in the Machine Part 4:Advanced Automated Testing Architectures, Professional Tester, April 2004.

Test Oracles

- ways of partially or fully knowing the expected results of a series of test cases

Douglas Hoffman, "A Taxonomy for Test Oracles", Quality Week 1998.

Douglas Hoffman, "Heuristic Test Oracles", Software Testing and Quality Engineering March/April 1999.

Douglas Hoffman, "Using Oracles in Testing and Test Automation", part 1, part 2,part 3 .

Fuzz Testing

Generating all sorts of random data and seeing what your system does

Barton P. Miller, David Koski, Chin Pheow Lee, Vivekananda Maganty, Ravi Murthy, Ajitkumar Natarajan, Jeff Steidl, "Fuzz Revisited: A Re-Examination of the Reliability of Unix Utilities and Services", 2000

Monkey Testing

Random testing

Noel Nyman, " In Defense of Monkey Testing "

Noel Nyman, "Using Monkey Test Tools", Software Testing and Quality Engineering, January/February 2000.

Thomas R. Arnold, Visual Test 6 Bible, IDG Books, 1998. Chapter 14 of the book describes monkey testing and shows you how to add features to Freddie, a dumb monkey tool which is supplied on the CD accompanying the book.

Monkey testing tools

Smart Monkeys know your application, so are developed for each application, usually using a state table or model of the software.

Dumb monkeys:

Rational’s Test Factory uses dumb monkey techniques to explore applications and crate automation to maximize test coverage while minimizing test time.

Rational Visual Test (for the Windows platform).

Freddie, dumb monkey on the CD accompanying the book by Arnold, Visual Test 6. Bible.

Software Development Methodologies, Software Engineering in general

Boehm, Barry and Richard Turner, 2004. Balancing Agility and Discipline: A Guide for the Perplexed, Boston,Mass.>: Addison Wesley, 2004.

Steve McConnell, Rapid Development, Microsoft Press, 1996

Steve McConnell, Resources at the Survival Guide web site and his company, Construx.

Roger Pressman, Software Engineering: A Practitioners Approach, 4th Ed, McGraw Hill, 1996

FAA, Software Considerations in Airborne Systems and Equipment Certification, Document RTCA/DO-178B, RTCA, Inc. December, 1992

Test First Techniques for Unit Testing

Unit or Developer Testing

Michael C. Feathers, "The Self-Shunt Unit Testing Pattern", May 2001

Michael C. Feathers, "A Set of Unit Testing Rules", Sept 9, 2005.

Natalia Juristo, Ana M. Moreno, Sira Vegas and Martin Solari, "In Search of What we Experimentally Know about Unit Testing", IEEE Software, November/December 2006, pages 72-80

Arie van Deursen, Leon Moonen, Alex van den Bergh, and Gerard Kok, "Refactoring Test Code"

Refactoring

Martin Fowler, et al., Refactoring: Improving the Design of Existing Code , Addison-Wesley, 1999 (introductory)

Joshua Kerievsky, Refactoring to Patterns, Addison-Wesley, 2004 (intermediate)

Legacy Code

Michael C. Feathers, "Working Effectively with Legacy Code", Object Mentor site

Michael C. Feathers, Working Effectively with Legacy Code, Prentice Hall, 2005

Concurrency

Debugging and testing concurrency

Kang Su Gatlin, "The Trials and Tribulations of Debugging Concurrency", ACM Queue, October 2004, pages 66-73

Brian Goetz, Tim Peirels, Joshua Bloch, Joseph Bowbeer, David Holmes and Doug Lea, Java Concurrency in Practice, Addison Wesley, 2006.
This book is about developing concurrent programs, but also discusses testing.

Doug Lea, Concurrent Programming in Java, 2nd Ed, Addison Wesley, 2000
This book does not discuss testing much, but is about designing and writing concurrent programs.

Johannes Link, Unit Testing in Java, Morgan Freeman, 2002
Chapter 10 discusses testing concurrent programs.

Fail Fast

If your source code has a problem, write it so it will fail immediately with an error message rather than waiting. Make debugging much easier.

Jim Shore, "Fail Fast", IEEE Software, September/October 2004
On assertions and using them to fail on null values, etc.

Test Driven Development/ Design (Test First)

Scott W. Ambler, "Introduction to Test Driven Design"

Scott W. Ambler, "Agile Testing Strategies", Dr. Dobb’s Testing and Debugging Update, December 12, 2006

David Astels, Test-Driven Development: A Practical Guide, Prentice Hall PTR, 2003 (intermediate)

Kent Beck, "Simple Smalltalk Testing: With Patterns"

Kent Beck, Test-Driven Development, By Example, Addison Wesley, 2003 (introductory) See also articles at JUnit.org

Rod Claar, The Case for Unit Testing, recorded presentation with audio and slides

Mike Clark, "A Dozen Ways to Get the Testing Bug in the New Year"

Lisa Crispin, "Driving Software Quality: How Test-Driven Development Impacts Software Quality", IEEE Software November/December 2006, pages 70-71

Thomas Dohmke and Henrik Gollee, "Test-Driven Development of a PID Controller", IEEE Software May/Hune 2007, pages 47-50. The sIUnit framework lets users apply TDD to dynamic systems.

Michael Ellims, James Bridges, and Darrel C. Ince, "The Economics of Unit Testing", Empirical Software Engineering, Volume 11, Number 1, March 2006, pages 5-31

Ron Jeffries and Grigori Melnik, TDD: The Art of Fearless Programming, IEEE Software, May/June 2007, pp 24-30.

Robert C. Martin, Professionalism and Test-Driven Development, IEEE Software, May/June 2007, pp 32-36

B. George and L. Williams, "A Structured Experiment in Test-Driven Development", Information and Software Technology, Vol. 46, No. 5, 2004, pages 337-342

Thomas Hammell, with Russell Gold and Tom Snyder, "Getting Started with Test Driven Development", JavaWorld December 2004

Ron Jeffries, Extreme Programming Adventures in C#, Microsoft Press, 2004 (introductory).

Michael J. Johnson, E. Michael Maximilein, Chih-Wei Ho and Laurie Williams, "Incorporating Performance Testing in Test-Driven Development", IEEE Software, May/June 2007. How to do early performance testing using unit testing frameworks.

Jonathan Kohl, "Test Driven Development from a Conventional Software Testing Perspective, Part 1", article on Informit.com, April 14, 2006
A conventional tester with some programming skills pairs with a developer to learn TDD

Jonathan Kohl, "Test Driven Development from a Conventional Software Testing Perspective, Part 2", article on Informit.com, April 21, 2006
A conventional tester with some programming skills pairs with a developer to learn TDD

Jonathan Kohl, "Test Driven Development from a Conventional Software Testing Perspective, Part 3", article on Informit.com, May 4, 2006.
A conventional tester with some programming skills pairs with a developer to learn TDD

Jeff Langr, " Evolution of Test and Code Via Test-First Design"

Brian Marick, Everyday Scripting with Ruby: For Teams, Testers, and Users, Pragmatic Bookshelf, 2007 (introductory)

Gerard Meszaros, XUnit Test Patterns, Addison-Wesley, 2007 (intermediate-advanced)

Roy Miller, "Demystifying Extreme Programming: Test-Driven Programming", IBM DeveloperWorks, April 2003.

James Newkirk and Alexey Vorontzov, Test-Driven Development in Microsoft .NET, Microsoft Press, 2004 (intermediate)

J.B. Rainsberger, Junit Recipes, Manning Publications, 2004 (advanced)

Keith Ray, TDD blog

Alec Ruiz and Yvonne Wang Price, "Test-Driven Development with TestNG and Abbot’, IEEE Software May/June 2007, pages 51-57. Recommendations to simplify test-driven GUI development.

Keith Stobie, "Test Result Checking Patterns", 2001.

Dave Thomas and Andy Hunt, " Learning to Love Unit Testing", STQE magazine, January/February 2002, pages 32-47

Bas Vodde and Lasse Koskela, "Learning Test-Driven Development by Counting Lines", IEEE Software, May/June 2007, pages 74-79.Two people from Nokia discuss their experiences in designing a class to teach Nokia developers TDD.

L. Williams, E. M. Maximilien, and M. Vouk, " Test-Driven Development as a Defect-Reduction Practice ", Proc. 14th Int’l Symp. Software Reliability Eng. (ISSRE 03), IEEE Press, 2003, pp34-45

Test Driven Development mailing list.

Effect of TDD on Productivity and Quality

Overview papers

David Janzen and Hossein Saiedian, "Test-Driven Development: Concepts, Taxonomy, and Future Direction," IEEE Computer, September 2005, pages 43-50

David Janzen and Hossein Saiedian, "On the Influence of Test-Driven Development on Software Design," Proceedings of the 19th Conference on Software Engineering Education and Training (CSEET ’06)

Ron Jeffries and Grigori Melnik, "TDD: The Art of Fearless Programming,", IEEE Software, May/June 2007, pp 24-30.

Papers describing experiments on the effect of TDD on productivity and quality

J Sanchez, Laurie Williams, and E. Michael Maximilien, "A Longitudinal Study of the Use of a Test-Driven Development Practice in Industry", Proc. Agile 2007 Conference, IEEE CS Press, 2007

T. Bhat and N. Hagappan, "Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies," Proc Int’l Symp Empirical Software Eng (ISESE 06), ACM Press, 2006, pp 356-363.

A. Canfora et al., "Evaluating Advantages of Test Driven Development: A Controlled Experiment with Professionals," Proc Int’l Symp. Empirical Software Engineering (ISESE 06), ACM Press, 2006, pp. 364-371

L. Damm and L. Lundberg, "Results from Introducing Component-Level Test Automation and Test-Driven Development," J. Systems and Software, vol. 79, no. 7, 2006, pp. 1001-1014

M. Melis et al., "Evaluating the Impact of Test-First Programming and Pair Programming though Software Process Simulation," J Software Process Improvement and Practice, vol. 11, 2006, pp 345-360

C. Mann, "An Exploratory Longitudinal Case Study of Agile Methods in a Small Software Company," master’s thesis, Dept. Computer Science, Univ. of Calgary, 2004

A. Geras, et al., "A Prototype Empirical Evaluation of Test Driven Development," Proc 10th Int’l Symp. Software Metrics, (Metrics 04), IEEE CS Press, 2004, pp 405-416

B. George and Laurie Williams, "An Initial Investigation of Test Driven Development in Industry", Proc. ACM Symp. Applied Computing, ACM Press, 2003, pp 1135-1139

R.A. Ynchausti, "Integrating Unit Testing into a Software Development Team’s Process," Proc. 2nd Int’l Conf Extreme Programming and Flexible Processes in Software Eng. (XP01), 2001, pp 79-83.

B. George and Laurie Williams, "A Structured Expeiment of Test-Driven Development," Information and Software Technology, vol. 46, no. 5, 2004, pp. 337-342

E. Michael Maximilien and Laurie Williams, " Assessing Test-Driven Development at IBM," Proc. 25th Int’l Conf. Software Eng (ICSE 03), IEEE CS Press, 2003, pp 564-569.

Laurie Williams, E. Michael Maximilien, and Mladen Vouk, "Test Driven Development as a Defect-Reduction Practice," Proc 14th Int’l Symp Software Reliability Eng. (ISSRE03), IEEE Press, 2003, pp 34-45.

Unit Testing for Java

Agitarautomated unit testing for Java. Generates test cases.

JUnit Factory from Agitar is a free service. You send them your Java source code and they send back JUnit test cases for the code.

JTest from Parasoft. Has several functions. Automatically generates and runs JUnit tests. Has a coverage tool built in, and a static analysis tool built in. You can add to the generated tests.

Junit

TestNGby Cedric Beust, which pioneered the annotation based testing style now in Junit 4.

Cedric Beust and Hani Seuliman, Next Generation Java Testing: TestNG and Advanced Concepts, Addison Wesley, 2007.

Fillipo Diotalevi, "TestNG makes Java unit testing a breeze", IBM DeveloperWorks, January 2005.

David Carew, "Keeping critters out of your code: How to use WebSphere and Junit to prevent programming bugs", tutorial, IBM DeveloperWorks, June 2003.

Unit Testing for C/C++

C++Test from Parasoft. Has fewer functions than the JTest tool, but they offer Insure++ which has the static analysis piece. Automatically generates and runs CppUnit tests and has a coverage tool built in.

Open Source Unit Test Frameworks

Paul Hamill, Unit Test Frameworks, O’Reilly, 2004

Unit testing frameworks links to over 82 unit testing frameworks libraries for different languages

JUnit and Test Driven Development

Matt Albrecht, "Using Multi-Threaded Tests" Discusses the GroboUtils class MultiThreadedTestRunner (an enhancement to Junit) to test multi-threaded Java applications

Kent Beck and Erich Gamma, "Junit: A Cook’s Tour"

Mike Clark, "JUnit Primer"

Malcolm Davis, "Incremental Development with Ant and Junit", IBM DeveloperWorks, November 2000.

Erich Gamma, and Kent Beck, "Junit Test Infected: Programmers Love Writing Tests"

Antonio Goncalves, "Get Acquainted with the New Advanced Features of Junit 4", July 24, 2006.

Elliotte Harold, "An Early Look at Junit 4", September 13, 2005.

Andrew Hunt and David Thomas, Pragmatic Unit Testing, In Java with JUnit,The Pragmatic Bookshelf, 2003 (introductory and recommended)

Johannes Link, Unit Testing in Java, Morgan Freeman, 2002

Vincent Massol with Ted Husted, Junit in Action, Manning Publications Co., 2004

J.B. Rainsberger,"Junit: A Starter Guide"

Andy Schneider, "JUnit Best Practices: Techniques for building resilient, relocatable, multithreaded Junit Tests, JavaWorld December 2000

William C Wake, "The Test/Code Cycle in XP", Part 1 and Part 2 A fairly long worked example of using TDD and using Junit to test a UI.

Don Wells, "Unit Tests", Part1 and Part 2

NUnit and Test Driven Development

Andrew Hunt and David Thomas, Pragmatic Unit Testing, In C# with NUnit, The Pragmatic Bookshelf, 2003 (introductory and recommended)

Pablo Santos and Francisco J. Garcia, " Distributed Unit Testing", Dr Dobbs Portal, October 2006 On an extension to NUnit to support distributed unit testing. A link to the source code .

Unit Testing of real time dynamic systems

Thomas Dohmke and Henrik Gollee, "Test-Driven Development of a PID Controller", IEEE Software, May/June 2007.

SlUnit, for dynamic systems. It’s designed for the Simulink graphical programming language for modeling and graphical design.

Unit testing of GUIs

Alex Ruiz and Yvonne Wang Price, "Test-Driven GUI Development with TestNG and Abbot", IEEE Software, May/June 2007.

Alex Ruiz, "Programmatic GUI Tests with TestNG-Abbot (with Demo)."

Tools for Java (libraries)

Jacaretofor record and playback

Marathonfor record and playback

Jemmyfor programmatic GUI testing

UISpec4Jfor programmatic GUI testing

Abbotfor both record and playback and programmatic GUI testing. It’s a Java library for testing Swing GUIs. It includes the script editor Costello for creating tests for existing GUIs and a Java API that can help create programmatic tests.

jfcUnitfor both record and playback and programmatic GUI testing

TestNG is a Java testing framework designed to cover different levels of tests, from unit to functional to integration tests. TestNG has parameterized tests, as does the newest release of JUnit, Junit 4.x. It supports data-driven testing. Abbot isn’t designed to work with TestNG, only Junit. There’s an open source project TestNG-Abbot which provides an easy way to use TestNG and Abbot together.

Performance testing in TDD

Michael J Johnson, E. Michael Maximilien, Chih-Wei Ho and Laurie Williams, "Incorporating Performance Testing in Test-Driven Development", IEEE Software May/June 2007. This is a good introduction to a methodology for incorporating performance testing early as part of developer testing. They used JUnit, so their method is usable by people working in other languages and unit test frameworks.

Chih-Wei Ho et al, "On Agile Performance Requirements Specification and Testing", Proc. Agile 2006 Int’l Conference, IEEE Press, 2006, pp 47-52.

E.J. Weyuker and F.I. Vokolos, "Experience with Performance Testing of Software Systems: Issues, An Approach, and Case Study", IEEE Transactions on Software Engineering, Vol 26, no 12, Dec. 2000, pp 1147-1156.

Tools to help:

Jacl, an interactive scripting language that allows manual interactive exercising of classes by creating objects and calling their methods.

JUnitPerf,a JUnit based performance tool

SOA testing

SOAPscope Testerfrom Mindreef. SOAPscope Tester combines a Web services testing and SOA Quality Management platform with powerful load testing capabilities. It provides an integrated set of tools for Web services performance and load testing, design-time support, prototyping, change-time and run-time support

Mock Objects

Martin Fowler, "Mocks Aren’t Stubs"

Steve Freeman, Nat Pryce, Tim Mackinnon, Joe Walnes , "Mocks Roles, not Objects", OOPSLA 2004

Tim Mackinnon, Steve Freeman, Philip Craig, "Endo Testing: Unit Testing with Mock Objects", in Extreme Programming eXamined, Addison Wesley, 2001

Ivan Moore and Sebastian Palmer, "Making a Mockery", in Proceedings of XP2002: 3rd International Conference on eXtreme Programming and Flexible Processes in Software Engineering.

Dave Thomas and Andy Hunt, "Mock Objects", IEEE Software, May/June 2002, pages 22-24

Mock Objects Frameworks and Tools

For Java

Easy Mock

Extender

MockCreator

MockMaker

Mockry

MockObjects Framework provides some ready-made mock objects for standard JCK API’s. Also some reusable expectation classes.

jMock

Cactus allows us to test components in their containers in the J2EE environment

For .Net

NMock

.NET easyMock

Testing Database Systems

If you separate the idea of testing the interfaces to your database from testing the database internals, then the usual unit testing frameworks (xUnit) work for testing the interfaces. Tools are beginning to be available to help with testing the internals. See Visual Studio Team System for Database Professionals (for SQL Server), Eclipse Data Tools Platform, and Qute from Quest (for Oracle) for examples.

Unit Testing Databases

Scott W. Ambler, Agile Database Techniques: Effective Strategies for the Agile Software Developer, Wiley and Sons, 2003.

Scott W. Ambler, "Ensuring Database Quality", Dr. Dobb’s Journal, Vol 31, Issue 12, December, 2006, pages 63-65.

Scott W. Ambler and PramodSadalage, Refactoring Databases: Evolutionary Database Design, Addison-Wesley, 2006.

Scott W Ambler, "Database Testing: How to Regression Test a Relational Database", 2006.

Scott W. Ambler, "Database Regression Testing", 2006.

Scott W. Ambler, "Test-Driven Development of Relational Databases", IEEE Software, May/June 2007, pp 37-43. This is an excellent article and well worth reading.

David Astels, Test-Driven Development: A Practical Guide, Prentice Hall PTR, 2003
S
ee Chapter 5, page 97 on Daedalos JUnit Extensions (djux) for testing databases. Djux supplies methods useful for checking if a row was added to a database, for example.

Shirley A. Becker and Anthony Berkemeyer, "The Application of a Software Testing Technique to Uncover Errors in Database Systems", Proceedings of the 20th Pacific Northwest Software Quality Conference, 1999, pp 173-183. The link is to the whole .pdf of the conference proceedings - print pages 170-180 for this paper.

Richard Dallaway, "Unit Testing Database Code", 2006.

Martin Fowler and Pramod Sadalage, "Evolutionary Database Design", 2003. Good, practical advice on what works in an agile project to allow evolutionary changes to the database design.

Vincent Massol with Ted Husted, Junit in Action, Manning Publications Co., 2004
See Chapter 11 on Unit Testing Databases.

Gerard Meszaros, XUnit Test Patterns, Addison-Wesley, 2007 (intermediate-advanced). Discusses Testing with Databases in Chapter 13.

J.B. Rainsberger, Junit Recipes, Manning Publications, 2004 (advanced). Chapter 10 is on Testing and JDBC and Chapter 11 is on testing Enterprise Java Beans, including data access.

Bill Siggelkow, "DBUnit Made Easy", 2005. Shows how DBUnit can be used to extract data from an existing database into an XML file, and then be used to reload the data into a test database.

Alex Vollmer, "Unit-Testing Hibernate with HSQLDB", 2005. Hibernate is an object-relational mapping mechanism. HSQLDB is an in-memore SQL database. You can use Hibernate to unit test against HSQLDB rather than your real production database.

Agile Database Development

Scott Ambler, Agile Database Techniques: Effective Strategies for the Agile Software Developer, Wiley and sons, 2003.

Scott Ambler, "Agile Best Practices for Data Warehousing (DW)/Business Intelligence (BI) Projects", 2007.

Scott W. Ambler, "Evolutionary/Agile Database Best Practices", 2007.

Look on the www.agiledata.org site for more articles on agile database development.

Scott Ambler and Pramod Sadalage, Refactoring Databases: Evolutionary Database Design, Addison-Wesley, 2006.

Scott Ambler, "Development Sandboxes: An Agile Best Practice", 2003.

Martin Fowler and Pramod Sadalage, "Evolutionary Database Design", 2003.

System Testing Databases

Chris McMahon, "Old School Meets New Wave", Better Software Magazine, June 2006, pages 28-32, (on testing middleware). On using scripted automation to do system testing for a message broker implementation and an agile project.

Don Slutz, "Massive Stochastic Testing of SQL", Proceedings of the Very Large Database Conference 1998. They developed a tool to generate statistically random SQL. inputs.

Database Testing Tools

- unit testing tools

These tools enable you to regression test your database.

DBFit

DbUnitprovides several useful functions for unit testing databases. For example it can load a database from an XML file.

NDbUnit

OUnit for Oracle- from Quest - nowQUTE. QUTE from Quest validates Oracle PLSQL code.

SQLUnit

TSQLUnit (for testing t-SQL in MS SQL Server)

Microsoft's Visual Studio Team System for Database Professionals is an IDE that comes with data tools. This product includes testing tools for Microsoft’s SQL serverXTUnit

Database Testing tools - for load testing

These tools simulate high usage loads on your database, enabling you to determine whether your system’s architecture will stand up to your true production needs.

Empirix

Mercury Interactive - now owned by HP.

RadView

Rational Suite Test Studio- now owned by IBM.

Web Performance

Database testing tools - test data generators

Developers need test data against which to validate their systems. Test data generators can be particularly useful when you need large amounts of test data, perhaps for stress or load testing.

For small amounts of test data, tools like DbUnit can help. You can store the data as XML files and use DbUnit to load it.

Data Factory

Datatect

DTM Data Generator.

Turbo Data

Other data related tools

Cactusallows you to start the test inside a container, e.g. to test a database that runs on a web server.

Daedalos Junit Extensions, Djux for short provides the ability to open the database at the start of a series of tests and close it at the end of the series, which makes the set of tests run faster; it also provides some methods for checking databases, for example to check if a row is present.

Eclipse Data Tools Platform is an IDE that comes with data tools.

JBoss provides a J2EE container and a database you can use for testing. HypersonicSQL is the in-memory database. JBoss comes preconfigured with HypersonicSQL.

Jetty provides a web server you can embed in your test class. Jetty is fast and lightweight, and can be completely controlled in Java from your test case.

Redgate tools for SQL Server can compare schemas, compare SQL data, etc.

Data Quality

Lydia Ash, "Data Set Analysis : Approaches to Testing when the Build is the Data", CAST 2007 presentation, Bellevue WA, July 2007.

Laure Berti-Equille, "Quality-Aware Data Management and Mining, Institut National de Recherche in Informatique et en Automatique, 2007.

Laure Berti-Équille, « La qualité des données comme condition à la qualité des connaissances : un état de l'art, in: Mesures de qualité pour la fouille de données. Numéro spécial, Revue Nationale des Technologies de l'Information (RNTI »),Cépaduès 2004.

Laure Berti-Équille , F. Moussouni. "Quality-Aware Integration and Warehousing of Genomic Data" in: Proceedings of the 10th Intl.Conference on Information Quality, IQ'05, Massachusetts Institute of Technology, Cambridge, MA, USA, November 2005.

Laure Berti-Équille, « Nettoyage des données XML : combien ça coûte ? », in: Actes du 1er Atelier Qualités des Données et des Connaissances, DKQ'05, en conjunction avec la conférence Extraction et Gestion des Connaissances, EGC'05, Paris, France, January 2005.

Laure Berti-Équille, "Cost of Low-Quality Data over Association Rules Discovery", in: Proceedings of the International Symposium on Applied Stochastic Models and Data Analysis, AMSDA'05, >Brest, France, May 2005.

Laure Berti-Équille, "Recommendation of XML Documents exploiting Quality Metadata and Views", in: Proceedings of the 2nd International Workshop on Data and Information Quality, DIQ'05, in conjunction with the 17th Conference on Advanced Information Systems Engineering, CAiSE'05, Porto, Portugal, June 2005.

Laure Berti-Équille, "Quality-Adaptive Query Processing over distributed Sources", in: International Conference on Information Quality, M.I.T., Boston, U.S., November 2004.

T. Dasu T. Johnson, Exploratory Data Mining and Data Cleaning John Wiley and Sons, 2003.

Larry English, Improving Data Warehouse and Business Information Quality, New York, Wiley, 1999.

M. Hernandez, S. Stolfo, "Real-World Data is Dirty: Data Cleansing and theMerge/Purge Problem" in: Journal of Data Mining and Knowledge Discovery, 1998, vol. 2, no 1, p. 9-37.

David Isaac and Christopher Lynnes, "Automated Data Quality Assessment in the Intelligent Archive", white paper prepared for the Intelligent Data Understanding program, January 2003.

Jack Olson, Data Quality: The Accuracy Dimension, Morgan Kaufmann, 2002.

T. Redman, Data Quality for the Information Age, Artech House, 1996.

Agile Testing

Jeannitta Andrea, "Envisioning the Next Generation of Functional Testing Tools, IEEE Software, May/June 2007, pages 58-65.

James Bach, "The Challenge of "Good Enough" Software", 1995.

Lisa Crispin and Tip House, Testing Extreme Programming, Addison Wesley, 2003.

Robert Culbertson, Chris Brown and Gary Cobb, Rapid Testing, Prentice Hall PTR, 2002

Jonathan Kohl, "Conventional Software Testing on a Scrum Team", article on Informit.com, September 30, 2005
A professional tester joins a Scrum team

Brian Marick, Agile Testing home page,

Brian Marick, "Roadmap for Testing on an Agile Project"

Brian Marick, "Agile Methods and Agile Testing".

Chris McMahon, "Old School Meets New Wave", Better Software Magazine, June 2006, pages 28-32, (on testing middleware)
On using scripted automation to do system testing for a message broker implementation and an agile project.

Exploratory Testing

James Bach, "A Low-Tech Testing Dashboard", presentation at Star ’99 East

James Bach, "General Functionality and Stability Test Procedure"
Document for testing the functionality and stability of a software application for the purpose of certifying it for Windows 2000

James Bach, "Heuristic Risk-Based Testing", Software Testing and Quality Engineering November 1999

Jonathan Bach, "Session-Based Test Management", Software Testing and Quality Engineering, November 2000

James Bach, "Boost Your Testing Superpowers", Star ’99 East
Simple and cheap testing tools

James Bach, "What is Exploratory Testing", Stickyminds column

James Bach, Rapid Software Testing, course notes, Fall 2002

James Bach, "Heuristic Test Strategy Model", April 2003

James Bach, "Heuristics of Software Testability", April 2003

James Bach, "Exploratory Testing Explained", April 2003

James Bach, "Reasons to Repeat Tests", 2004

Jonathan Bach, "Testing in Session: A Method to Measure Exploratory Testing", Washington Software Association QA SIG, May 13, 2003
Slides of a presentation

Michael Bolton, "Master of Your Domain", 2006
Other articles there as well

Elisabeth Hendrickson, "Rigorous Exploratory Testing", April 19, 2006

Test Process

Testing Process/ Test Planning

Rex Black, Managing the Testing Process, 2nd Ed, Microsoft Press, 2002

Rex Black, Critical Testing Processes: Plan, Prepare, Perform, Perfect, Addison Wesley, 2004

Cem Kaner, " Negotiating testing resources: A Collaborative Approach", 1995,

Edward Kit, Software Testing in the Real World: Improving the Process, Addison-Wesley, 1995.

William E. Lewis, Software Testing and Continuous Quality Improvement, Auerbach, 2000

Brian Marick, "The Testing Team’s Motto"

Brian Marick, James Bach and Cem Kaner, "A Manager’s Guide to Evaluating Test Suites", Quality Week Conference, 2000

William E. Perry, Effective Methods for Software Testing, Wiley, 1995

John Watkins, Testing IT: An Off-the-Shelf Software Testing Process, Cambridge University Press, 2001.

Test Process Improvement

Edward Kit, Software Testing in the Real World, Addison-Wesley, 1995

Tim Koomen, Martin Pol, Test Process Improvement, Addison-Wesley, 1999

Brian Marick, "Classic Testing Mistakes", STAR ’97 Conference, 1997

Testing Embedded Software

Bart Broekman and Edwin Notenboom, Testing Embedded Software, Addison Wesley, 2003.

Arnold S. Berger, Embedded Systems Design, CMP Books, 2002
Chapter 9 is devoted to testing.

Test Automation

These are mainly on automation involving record and playback tools - or in the case of the Li books, in writing your own automation tool. See also the references on FIT for an open source automation framework that doesn’t test through the GUI.

GUI Automation

T. Arnold, Software Testing with Visual Test 4.0, IDG Books, 1996.

James Bach, "Test Automation Snake Oil", 1999.

Hans Buwalda, "Automated Testing with Action Words, Abandoning Record and Playback", Proceedings of the EuroStar Conference, 1996.

Hans Buwalda and Maartje Kasdorp, "Getting Automated Testing Under Control", Software Testing and Quality Engineering, November/December 1999

Hans Buwalda, Dennis Janssen and Iris Pinkster, Integrated Test Design and Automation Using the Test Frame Method, Addison Wesley, 2002

Elfriede Dustin, Jeff Rashka, and John Paul, Automated Software Testing: Introduction, Management and Performance, Addison Wesley, 1999

Mark Fewster and Dorothy Graham, Software Test Automation, Addison-Wesley, 1999

C. Groder, "Building Industrial-Strength GUI Tests", Proceedings of the 8th International Conference on Software Testing, Analysis and Review (STAR conference), p 381-396, 1997.

Linda Hayes, The Automated Testing Handbook, Software Testing Institute, 2004

Douglas Hoffman, "Mutating Automated Tests", 2000

Douglas Hoffman, "Cost Benefits Analysis of Test Automation", presentation at StarWest 1999. Good analysis of what is involved in trying to compute the cost/benefits of implementing test automation, with sample calculations.

Cem Kaner, Walter P. Bond and Pat McGee, "High Volume Test Automation"

Cem Kaner, "Improving the Maintainability of Automated Test Suites", Quality Week, 1997.

Cem Kaner, "Avoiding Shelfware: A Manager’s View of Automated GUI Testing", 1998. Useful article on what works and doesn’t in GUI automation. List of 19 common mistakes and 27 questions about your requirements for automation. Good discussion of automation architectures.

Cem Kaner, "Architectures of Test Automation", August 2000

Kanglin Li and Mengqi Wu, Effective Software Test Automation: Developing an Automated Software Testing Tool, Sybex, 2004

Kanglin Li and Mengqi Wu, Effective GUI Test Automation: Developing an Automated GUI Testing Tool, Sybex, 2005

Chris McMahon, "Old School Meets New Wave", Better Software Magazine, June 2006, pages 28-32, (on testing middleware)
On using scripted automation to do system testing for a message broker implementation and an agile project.

Brian Marick , "Bypassing the GUI", STQE magazine, September/October 2002, pages 41-47
On using scripting to do test automation behind the GUI.

Daniel J Mosley and Bruce A. Posey, Just Enough Software Test Automation, Prentice Hall PTR, 2002

Carl J Nagle, "Test Automation Frameworks"
Also open source frameworks

Bret Pettichord, "Success with Test Automation", 2001

Kerry Zallar, "Are You Ready for the Test Automation Game?", STQE magazine, November/December 2001.

Keith Zambelich, "Totally Data-Driven Automated Testing", white paper. This is a very good paper that makes clear the technical differences between record-and playback, data driven and keyword driven approaches.

GUI Tools (open source)

Canoo Web Test

Watir

Selenium

Abbot

AutoIT is a free record and playback tool. AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting.

Systir

GUI Tools (vendors)

The traditional record/playback tool vendors usually offer a suite of 3 tools: a test case tracking tool, a record and playback tool, and a load testing tool. There are several vendors: Mercury, Compuware, Rational/IBM, Cyrano, Segue, and others.

TestComplete from Automated QA delivers automated functional, unit, regression, manual, data-driven, object-driven, distributed and HTTP load, stress and scalability testing in one easy-to-use and totally integrated package at an affordable price. TestComplete Enterprise includes 5 virtual users for web application load testing. You can purchase add-ons to 50, 250, or unlimited virtual users.

www.rational.com

Rational Software’s site. Info about Software testing tools (now owned by IBM)

Segue offers a wide range of testing tools and related services

Software Research, Inc offers testing tools, including capture/playback, test management, code coverage, and source-code analysis

Compuware QACenter and other tools for automating testing

CyranoQTest for Web Load Testing

Mercury offers a suite of tools, including Test Director, recently renamed Quality Center

PushToTest -- an open source automated testing tool which handles service oriented architecture (SOA) applications, as well as Ajax web applications, desktop applications, etc.

Visual Studio Team System and Visual Studio Team Test - test automation from Microsoft

Mark Michaelis, "A Unit Testing Walkthrough with Visual Studio Team Test", 2005.

Jason Anderson, Tom Arnold, "Visual Studio Team Test: The many test types and how to manage the many tests you generate", 2005.

Doug Seven, "Test Driven Development with Visual Studio 2005 Team System", March 2006.

MSDN blog, "Visual Studio Team Test - Starting Out", 2006. Links to many articles and blog entries.

FIT and FitNesse

FIT stands for Framework for Integrated Tests. FIT is an open source framework for automating testing of business rules.

The original implementation is in Java, but FIT has been ported to several other languages. FitNesse runs on top of FIT and allows the test cases to be on wiki pages, and thus shared.

Johnson, Kris, "Introduction to FIT and FitNesse", 2005.

FIT and FitNesse tool sites

FIT website. Documentation on using FIT is here, also example source code
You also need the FitLibrary for additional fixture types and POI .

Marick, Brian, StepFixture, an addition to FIT.

FitNesse runs on a web server, which makes it easy to share Fit test tables among many people working on a project. Chapter 27 in the book by Mugridge and Cunningham on Fit discusses FitNesse.

User Guide

FitNesse Yahoo Group - online discussion forum, mailing list and file repository. This is where to ask for help.

Storyteller - tool for efficient management of automated testing with FitNesse

Java

Foy, Cory, "Using Symbols in Java Fitnesse Fixtures".

McAuliffe, Jean, "Testing at the Speed of Need", talk at NetObjectives on Lean-Agile System Testing, January 2007. Includes slides on FitNesse.

Mugridge, Rick and Ward Cunningham, Fit (Framework for Integrated Tests) for Developing Software, Prentice Hall, 2005.

Vlagsma, Paul, "FitBook Examples for .NET 2.0, .NET 1.1 and Java".
Downloadable source code for the examples in the FIT book, in C#, J# and VB.Net with either .NET 2.0, .NET 1.1 or JRE1.5.

.NET

Adzic, Gojko, "Gettting Fit with .Net - Quick Introduction to Testing .Net applications with FitNesse", .pdf short tutorial on using FitNesse with .NET.

Adzic, Gojko, Test Driven .NET Development with FitNesse, available as a .pdf or as a printed book. Most FIT and FitNesse information is for the Java version, including the book by Rick Mugridge and Ward Cunningham. This book supplies the details of how the .NET port works, and the subtle technical differences when using FitNesse for .NET applications.

Foy, Cory, "Fitnesse and .NET - a basic tutorial".

Foy, Cory, "HowTo: Fitnesse ActionFixture in C#".

Foy, Cory, "Getting Up and Running with FitNesse.NET", 2005.

Mattfield, Jeff, "Building FitNesse and FitLibrary for .Net 2.0", February 10, 2006.

Vlagsma, Paul, "FitBook Examples for .NET 2.0, .NET 1.1 and Java".
Downloadable source code for the examples in the FIT book, in C#, J# and VB.Net with either .NET 2.0, .NET 1.1 or JRE1.5.

FitNesse.NET is a .NET port of FitNesse. Online acceptance tests for the .NET implementation - a good source of ideas, use to find new features and how to use them.

Subversion repository for NitNesse.Net on SourceForge - browse the latest source code for FitNesse.NET integration to see how things really work

Mike Stockdale’s page on FitNesse.NET plans

Blogs with good articles on Fitnesse and .Net Jeremy MillerXman892

Integrating FitNesse and Nant

Integrating FitNesse and CC.Net

Ruby

Foy, Cory, "Fitnesse and Ruby - a basic tutorial".

Mutation Testing

Mutation testing involves creating, perhaps using a tool, many copies of a program, each of which has a mutation, i.e. a source code change. Then you test whether your test cases can catch all the mutations. If there is a mutation that isn’t caught, your test suite wouldn’t catch that error, and needs enhancing. There are some tools available to generate the mutated programs and test them all against a set of testcases.

Articles

Mattias Bybro, "A Mutation Testing Tool for Java Programs", Master’s Thesis, 2003.

Sunwoo Kim, John A. Clark, and John A. McDermid, "Class Mutation: Mutation Testing for Object Oriented Programs", 2000.

Adam Kolawa, "Mutation Testing: A New Approach to Automatic Error-Detection", 1999.

James McCaffrey, "Create a Simple Mutation Testing System with the .NET Framework", MSDN Magazine, April 2006.

A. Jefferson Offutt, "A Practical System for Mutation Testing: Help for the Common Programmer", >Twelfth International Conference on Testing Computer Software, June1995

A. Jefferson Offutt and Roland H Untch, "Mutation 2000: Uniting the Orthogonal ",Mutation2000 Conference, October 2000

Yuan Laurie Williams, " Mutation Testing ", 2004, six powerpoint slides.

Mutation Testing tools

Mothra, a mutation testing tool for Fortran. It’s available free for research and educational use but not for commercial use. The page also links to papers on the theory of mutation testing.

Site listing mutation testing tools:

Mutation testing for Java

Jesterperforms random mutations on the source code being tested; it then verifies if your tests still pass.

MuJava, a mutation testing tool for Java

TSP and PSP (Personal Software Process and Team Software Process) from the SEI

PSP emphasizes improving estimation through collecting estimated size and effort and actual size and effort for every program, and very thorough Inspections for finding defects, as well as testing. TSP puts PSP trained people into teams that can self-organize, estimate their work well, and track progress using Earned Value.

PSP

Watts Humphrey, "The Personal Software ProcessSM (PSPSM)", Software Engineering Institute, 2000.

Watts Humphrey, PSP: A Self-Improvement Process for Software Engineers, Addison-Wesley, 2005.

Watts Humphrey, Introduction to the Personal Software Process, Addison Wesley, 1996.

TSP

Noopur Davis and Julia Mullaney, "The Team Software Process SM (TSP SM ) In Practice: A Summary of Recent Results", SEI Technical Report CMU/SEI-2003-TR-014, September 2003.

Watts Humphrey, "The Team Software ProcessSM (TSPSM)", Software Engineering Institute, 2000.

Watts Humphrey, Introduction to the Team Software Process, Addison-Wesley, 1999.

Watts Humphrey, TSP(SM)-Coaching Development Teams, Addison-Wesley, 2006.

Kelly Kimberland, "Microsoft’s Pilot of TSP Yields Dramatic Results", February 2004.

Testing Tools

Testing Tools sites - sites that link to tools sites or have information about tools

System testing, load testing, performance testing tools site,maintained by Cordell Vail, has links to both testing tools sites (sites that list tools) and to white papers and articles on testing tools. A good place to start " many links from this site to other tools resources. It has sections by type of tools (load and performance testing, record and playback, etc.).

Open source testing tools site that lists open source testing tools.

Good site, lists many types of tools on individual pages, and has a paragraph of information about each.

Testing tool for creating, printing and running tests.

Assessment Systems Corporation has books, software and various automated testing tools.

Introducing automated tools to your team.

CSST technologies provides products and services for testing client-server applications.

Dr Dobb’s website of software tools.

Methodology plus tools.

IPL Software Testing Products Library.

Links to a number of test tool companies. Excellent site.

Stellar Logic Corporation provides tools, services and information.

Information on McCabe TestCompress automated testing software.

Load testing and website monitoring services.

information on automated testing tools.

Home page for access to test tools lists in many categories " GUI test drivers, unit test tools, static analysis tools, test design tools and many others.

McCabe and Associates " QA consulting firm with products and processes.

Michael J. Johnson, E. Michael Maximilein, Chih-Wei Ho and Laurie Williams, "Incorporating Performance Testing in Test-Driven Development", IEEE Software, May/June 2007. How to do early performance testing using unit testing frameworks.

Links to useful freeware, shareware, and cheapware programs for testing

http://www.zdnet.com

www.tucows.com

www.shareware.com

www.pcmagazine.com

www.cnet.com

www.qadownloads.com

www.softpanorama.org

Static Analysis tools

These tools read the source code and check for errors without executing the code. They can find such errors as the possibility of dividing by zero, uninitialized pointers, etc. Some statically check the code itself, as Lint does. Others check the running code, by instrumenting it and analyzing the run trace. Some are research projects only, but they indicate possibilities for the future. Types of analysis include control flow analysis, to insure the code is executed in the right sequence, is well structured, to locate any syntactically unreachable code (dead code), and highlight parts of the code, e.g. loops, where termination needs to be considered. Data flow analysis shows that no execution paths in the software exist that would access a variable not set to a value. Information flow analysis identifies how execution of a unit of code creates dependencies between the inputs to and the outputs from that code. Information flow analysis can be augmented in some tools by using annotions which document certain assumptions about functions, variables, parameters and types.

Other techniques include Path Function Analysis (also called Semantic Analysis or Symbolic Execution) which reveals exactly what the code does in all circumstances for the whole range of input variables for each program section.

Also there is Formal Verification (also called Compliance Analysis) which involves proving, in an automated process and as far as possible, that the program code is correct with respect to the formal specification of its requirements. This is the most rigorous of the techniques, but the rigor is at the expense of cost; productivity is about 5LOC/day.

Articles on static analysis

German, Andy, "Software Static Code Analysis Lessons Learned", Crosstalk, Nov 2003. Discusses static analysis, what can be found, and then discusses several tools.

Koved, Tim, "SPADE and SABER: Improving Systems Through Error Reduction", talk for Microsoft, 2003.

Lamport, Leslie, " TLA - The Temporal Logic of Actions".

Larus, James R, Thomas Ball, Manuvir Das, Robert DeLine, Maneul Fahndrich, Jon Pincus, Sriram K Rajamani, and Ramanathan Venkatapathy, "Righting Software", IEEE Software May/June 2004, pages 92-100 on correctness tools " static analysis tools " developed at Microsoft Research

Poll, Eric, Joe Kiniry and David Kok, " ESC/Java: extended static checking for Java". This talk  shows how to use ESC/Java, with examples.

Reimer, Darrell, Edith Schonberg, Kavitha Srinivas, Harini Srinivasan, Bowen Alpern, Robert D. Johnson, Aaron Kershenbaum, Larry Koved, "SABER: Smart Analysis-Based Error Reduction", ISSTA ‘04, at ACM website with digital library subscription. See also talk on SABER by Larry Koved.

Robbins, John, "Bad Code? FxCop to the Rescue", June 2004. Blog about using FxCop to find bugs – about what FxCop is and about how to write additional rules. FxCop is now integrated into Visual Studio

Robbins, John, " Three Vital FxCop Rules", September 2004. Second part of article on FxCop – 3 additional rules. FxCop is now integrated into Visual Studio.

Savoia, Alberto, "From Primitive to Prominent: The Past, Present and Future of Automated Code Analysis", Better Software Magazine, April 2006, pages 18-22, (this link is to the whole issue; print pages 18-22).

Vaughan-Nichols, Steven J, "Building Better Software With Better Tools". IEEE Computer September 2003, Vol 36, No. 9, pages 12-14

Yu, Yuan and Tom Rodeheffer, " RaceTrack: Detecting Potential Races in Managed Code ", 2004.

Tools for static analysis

TLA " temporal logic of actions. TLA is a logic for specifying and reasoning about concurrent and reactive systems. Specification language, TLA+. Tools also available.

Spec# - a programming language which is an extension of C#. It extends the type system to include non-null types and checked exceptions. It provides method contracts in the form of pre- and post-conditions as well as object invariants. It includes a Spec# compiler and Spec# static program verifier. Spec# guarantees maintaining object invariants in OO programs in the presence of callbacks, threads and inter-object relationships. Currently alpha 9/2005.

PreFix- a simulation tool for C/C++- a model property checking tool. In the latest Visual Studio release.

PreFast - checks for syntactic and semantic errors. May only be available inside Microsoft.

SLAM - software, languages, analysis and model-checking project. SLAM simulates a system running a piece of software to make sure a given piece of program logic will work on every potential execution path.

RaceTrack - limited to Microsoft internal use only. For detecting potential races in managed code.

Zing - Zing is a software model checking project at Microsoft Research. For noncommercial research uses only. Requires Visual Studio .NET 2003.

SpinACM, " On-The-Fly, LTL Model Checking with SPIN" . Spin is a popular open-source software tool, used by thousands of people worldwide, that can be used for the formal verification of distributed software systems

FxCopFxCop is a code analysis tool that checks .NET managed code assemblies for conformance to the Microsoft .NET Framework Design Guidelines. It uses reflection, MSIL parsing, and call graph analysis to inspect assemblies for more than 200 defects in the following areas:

  • Library design
  • Localization
  • Naming conventions
  • Performance
  • Security

FxCop includes both GUI and command line versions of the tool. FxCop is open source and free.

CoVerity automates the detection of software defects and security vulnerabilities for complex software as developers write code. With Coverity, development teams identify critical software problems that could have a catastrophic impact. Our products help companies improve software quality and security, decrease time to market and optimize developer productivity. Coverity’s next-generation static analysis technology finds complex defects, scales to large code bases and does not waste developer time with false positives. CoVerity works on C and C++. It is not free, but there’s a free trial.

Eclipsehas a tool called Hyades which is an integrated test trace and monitoring environment. Hyades is based on the OMG’s UML testing profile. Hyades is integrated into the Eclipse IDE.

Jackpotfrom Sun consists of three major parts, source code metrics, JSCM which is for use with NetBeans, and a static code checker that uses lexical scanning, parsing, and semantic and dataflow analysis.

List of tools for static analysis, both free and commercial, from Wikipedia

PolySpace Embedded Software Verification. PolySpace products verify C, C++, and Ada code for embedded applications by detecting run-time errors before code is compiled and executed, using formal methods not only to detect errors, but to prove mathematically that certain classes of run-time errors do not exist.

MALPAS Malvern Program Analysis Suite supplied by Advantage. This has been used in the UK for static analysis of military programs.

SPARK is a subset of ADA for high integrity programming. Control flow analysis is subsumed into the SPARK grammar and is thus performed on the fly. Data and information flow requirements are expressed as SPARK program design rules. This has also been used in the UK for military programs.

LDRA Testbed (Liverpool Data Research Associates Ltd. Testbed). Many languages are covered: Ada, C, C++, COBOL, Coral 66, Pascal, PL/1, PS/MX86, and Intel and Motorola assemblers. The LDRA testbed will perform control flow, data flow and information flow analysies., with information flow enhanced by the use of annotions. This has also been used in the UK for military programs.

SPADE The SPADE tools provide flow analysis and verification facilities based on a model of the program being analysed. The model is generated by automatic translators which operate on the original source code.
Translators are available for a Pascal subset and for the following assemblers: 6800, 68020, 8096 and Z8002.

QA C Programming Research Ltd Tools for static analysis of C, C++, Java & Fortran.

Cantata++ from IPL is primarily a testing tool which generates unit and system test cases, and has a coverage tool built in. Also does static analysis: code complexity and size metrics for C, C++ and Java.

AdaTEST from IPL is primarily a testing tool which generates unit and system test cases, and has a coverage tool built in. Also does static analysis: code complexity and size metrics

Alsys C-SMART and RAVEN Certifiable Small Ada Run-Time from Aonix.

PC-Lint from Gimpel Software, for C and C++.

SpLINT for checking C programs. From the University of Virginia

Extended Static Checking (ESC) for Java Version 2. This tool uses annotations in its extended static checking. See paper above by Poll and many others findable with Google.

CodeSonar from GrammaTech finds bugs in C and C++ code.

Static Analysis for Security Problems

DevInspect from SPI Dynamics (now HP) - accelerates the construction and delivery of secure Web applications and Web services by finding and fixing security vulnerabilities during development, like unsanitized inputs, undeclared values, buffer overflows, etc.

QAInspect from SPI Dynamics (now HP) provides the appropriate Web application security testing tools for systems testing staff. Find vulnerabilities in specific usage scenarios or pages during the Initial QA process, and confirm they have been removed during final regression testing.

WebInspect - is the first and only web application security assessment tool to be re-architected to thoroughly analyze today's complex web applications built on emerging Web 2.0 technologies like AJAX, SOAP, SOA, JavaScript and Flash.

CodeScan from CodescanLabs, finds and corrects vulnerabilities in web source code. Currently it supports VBscript (.asp) and php code. Soon it will support .NET, Java and ColdFusion.

Fortify Source Code Analysis (SCA) is for source code security analysis. Its powerful, multi-algorithm analyzers automate the extremely difficult and tedious activity of identifying potentially dangerous functions, configurations and data flows within an entire code base -- even through multiple tiers and over millions of lines of code.

Fortify Tracer gives accurate, consistent answers to the key question "how many security-sensitive functions in my Web application were exercised?"

Fortify® Defender monitors custom business web applications from the inside out so it can deliver reports on who attacked, how often, the technique used and much more for every instance of a web application. Fortify Defender not only addresses PCI Data Security Standards but also key software security compliance requirements including OWASP Top Ten, HIPAA and more.

Fortify® Manager is a security dashboard and control center for Security and Development teams to centrally manage software security policies and report on the results from multiple deployments of Fortify SCA.

CxSuite from CheckMarx offers a solution for each participant of the Software Development Life Cycle:

CxAudit is used by internal and external auditors for translating business logic, company policy and organization standards into sets of queries, and assigning policies and sets of relevant queries for each specific project

CxDeveloper used by developers and team leaders for running sets of predefined and customized queries from the Integrated Development Environment (IDE) and browsing tracked security flaws directly in the code.

Team leaders use CxDeveloper for monitoring both full projects as well as reviewing each developer’s code.

Static Analysis tools for Java

Some analyze the source code and others analyze the byte code

SABER- can be installed as an addition to IBM’s WebSphere. Does fairly sophisticated static analysis of Java for hard to find errors. SABER stands for Smart Analysis Based Error Reduction. See paper on it in the section above, at the ACM website if you have ACM access.

PMD- source code analysis

Checkstyle - source code analysis

Hammurapi - source code analysis

FindBugs - byte code analysis

JLint - byte code analysis

Runtime Verification tools

Klaus Havelund and Grigore Rosu, "Java PathExplorer - a Runtime Verification Tool", 2000, an experimental tool for verifying Java programs. Developed by NASA Ames Research Center.

Klaus Havelund and Grigore Rosu, "Java Path Explorer " A Runtime Verification Tool", slides on Java Path Explorer.

Bill Pierce, "Diagnose Common Runtime Problems with hprof", JavaWorld, December 2001.

Memory Leak Detector Tools

Commercial - Purify Plusfrom Rational/IBM. Comes with statement coverage tool. For C, C++ and Java

JProfiler - for memory leaks and other uses, for Java. Not free, but cost is moderate, around $500.

Visual Leak Detector open source, for C and C++

Insure++ from Parasoft, for C++

Memory Leak Detection - The Code Project - C++ / MFC

ValGrind: Free memory leak detector for Linux

Tools for recording what happens on the screen

Good for recording when you’re doing exploratory testing, in case you don’t exactly remember the set of steps that led to a problem showing up

Not free, not expensive: Camtasia

Free: CamStudio

AppSight Black Box from BMC Software: Like a black box flight recorder for your applications. Just as a "black box" flight recorder on an aircraft captures a comprehensive, real-time record of each flight, AppSight’s patented Black Box software technology captures a "digital documentary" of application execution at multiple synchronized levels, based on a dynamic, user-defined recording profile. Black Boxes can be deployed on servers and/or clients to provide a unified view of application activity. Available for Windows and for J2EE. This records more internal details than the products above. It can be deployed at client sites to record information on problems not reproducible otherwise. Not free, quite expensive.

Tool for finding similarities in code - to eliminate duplication

Simian - this tool does similarity analysis in almost any text file, finding duplications of code

Tools for Stress, Load and Performance Testing

Cordell Vail and Joe Towns,Load/Performance Type Testing Tools at a Price You Can Afford’. They work at an organization without a lot of money, and searched for tools that cost less and found one that worked for them. This is a recording with audio of the presentation.

The traditional record/playback tool vendors usually offer a load/performance testing tool as a part of a suite of 3 tools: a test case tracking tool, a record and playback tool, and a load testing tool. There are several vendors: Mercury, Compuware, Rational/IBM, Cyrano, Segue, and others.

www.rational.com

Rational Software’s site. Info about Software testing tools (now owned by IBM)

Segue offers a wide range of testing tools and related services

Software Research, Inc offers testing tools, including capture/playback, test management, code coverage, and source-code analysis

Compuware QACenter and other tools for automating testing

Cyrano QTest for Web Load Testing

Mercury offers a suite of tools, including Test Director, recently renamed Quality Center

Stress/Load/ Performance testing for Java

JMeter

OptimizeIt, a performance testing and enhancement tool for Java and JavaBeans

Web Testing tools

Various Web tools, including SiteMapper, a program that maps web sites, and SubmitWolfPRO, a Web site submission tool.

HTML Power Tools for Windows

freeware tools for the Web, and news

XML Spy is an XML editor. Free trial version available

Web HTML Validation and Link Checking Tools/Sites

Web Techs, a free online HTML validator.

a link to link verifiers

HTML validator by W3C

A Real Validator, HTML validation software with a 30 day trial version

free download of CSE Validator Lite, an HTML validator

Rational SiteCheck follows all links and checks that they are valid.

Profilers

Rational Quantify is now integrated into Rational PurifyPlus - performance profiling tool available for Windows, Linux, and UNIX platforms. It provides accurate and repeatable timing data, enabling a quick analysis of your application's run-time execution. Quickly directs you to the areas in your application that are taking the most time to execute.

Performance Validator from Software Verification is available for several languages: .Net, C#, VB.Net, C++, Delphi, Visual Basic, Fortran 95, Java, JavaScript, Lua, Perl, PHP, Python, Ruby.

Intel VTune Performance Analyzer - makes application performance tuning easier with a graphical user interface and no recompiles required. It is compiler and language independent so it works with C, C++, Fortran, C#, Java, .NET and more†.VTune analyzer offers call graph analysis with an extensive set of tuning events for all the latest Intel® processors.

Performance Profiling Tools for Windows Presentation Foundation. WPF provides a suite of performance profiling tools that allow you to analyze the run-time behavior of your application and determine the types of performance optimizations you can apply. WPFPerf is included in the Windows SDK. This tool is included in .NET Framework 3.0. There are five tools included in WPFPerf., an event trace, used for analyzing events and generating an event log file; a perforator, used for analyzing rendering behavior; an ETW Trace Viewer used to record, display and browse Even Tracing for Windows log files; a Visual Profiler, used for profiling the use of WPF services, such as layout and event handling, by elements in the visual tree; and a Working Set Analyzer, used to analyze the working set characteristics of an application.

Remy Sharp, Performance Profiling Java Script, using IE. He’s developed a library that allows him to do performance profiling, relatively unobtrusively, in Internet Explorer.

AQTime from Automated QA is both a memory and resource debugging tool and a performance profiling toolset for Microsoft, Borland, Intel, Compaq and GNU compilers. AQtime supports .NET 1.0, 1.1 and 2.0 applications and Windows 32- and 64-bit applications. AQtime collects crucial performance and memory/resource allocation information at runtime.

JProfiler for memory leaks and performance profiling, for Java. Not free, but cost is moderate, around $500.

For Ajax developers jsLex gives developers a complete picture of the performance issues with-in their Ajax application. By auto injecting profiling code using jsLex ant task, developers don't need to modify their code to track down bottlenecks. Because no plugin is required, jsLex will work in all browsers. JsLEX is open source.

NetBeans Profiler. This is a NetBeans IDE add-on. It can do task-based profiling, help you speed up performance, slim down memory usage, and untangle your threads.

Metz, Edu and Raimondas Lencevicius, Efficient Instrumentation for Performance Profiling, Nokia Labs 2003

Glavich, Paul, Profiling the performance of a .NET application.

ANTS Profiler provides you with a quick and easy way of identifying performance bottlenecks within your .NET applications. Additional support for the latest .NET 3.0 features, the Vista operating system, and Visual Studio integration, cements ANTS Profiler as a leading tool in performance profiling. Open source.

Performance and memory profiling tools for Mozilla.

Shirazi, Jack and Kirk Pepperdine, Eye on performance: Profiling on the edge: What do you do when your tools are too fat? Tuning isn't always about speed.. Your first course of action is normally to monitor the application with a profiler. But profiling is not always practical. In this installment of Eye on performance, Jack and Kirk relate their recent experiences with profiling a fat client -- so fat, in fact, that it left no room for a profiler.

Java Performance Tuning page lists books and tools for Java profiling./ Lists both open source (free) tools and commercial tools many many tools listed

Automated build environments and Continuous Integration Servers

Articles

Mike Clark, Pragmatic Project Automation : How to Build, Deploy, and Monitor Java Applications, Pragmatic Programmers, 2004

Steve McConnell, "Daily Build and Smoke Test"

Martin Fowler, "Continuous Integration"

Dave Thomas and Andy Hunt, Pragmatic Version Control using CVS, Pragmatic Programmers, 2004

Tools for Continuous Integration

A matrix of features of a number of continuous integration servers

A blog on the uses of continuous integration servers with comments on a couple of tools for the job, for a Windows environment

Gumpis Apache's continuous integration tool. It is written in Python and fully supports Apache Ant, Apache Maven and other build tools.

Justin Gehtland, "Continuous Integration with CruiseControl.NET and Draco.NET"

Article on continuous integration in the .Net environment, using NAnt and CruiseControl. This is shorter than Justin Gehtland’s article above, but also includes coding examples to help you get started

Ant is a build tool, which is the de facto standard for Java applications

Malcolm Davis, "Incremental Development with Ant and Junit", IBM DeveloperWorks, November 2000.

CruiseControl.NET

Draco.NET

NAnt Ant for .NET applications

MavenMaven contains several plugins to make setting up automated builds easier

Ambient Orb

Ambient Orb WDK

Tools for Web Security Testing or checking hackers’ techniques

Books and articles

Mike Andrews and James A. Whittaker,How to Break Web Software, Addison Wesley, 2006. This book describes most of the tools below. It’s very good and very detailed on looking for hacking vulnerabilities on web sites.

Michael Howard, David LeBlanc, and John Viega,19 Deadly Sins of Software Security. McGraw-Hill, 2005.

Jack Koziol, David Litchfield, Dave Aitel, and Chris An,:The Shellcoder's Handbook: Discovering and Exploiting Security Holes , Wiley, 2004

Johnny Long,Google Hacking for Penetration Testers, Syngress Publishers, 2001

Bala Neerumalla, "New SQL Truncation Attacks And How To Avoid Them", MSDN Magazine, November 2006.

Aleph One, " Smashing the Stack for Fun and Profit ".

Tools

IEHttpHeaderstool, which help uncover what is being sent between pages. Also on the CD that comes with the book, How To Break Web Software, by Mike Andrews and James Whittaker.

the Paros tool helps uncover what is being sent between pages. Also on the CD that comes with the book, How To Break Web Software, by Mike Andrews and James Whittaker.

SPIKE Proxy" tests parameter manipulation and CGI buffer overflow. Also on the CD that comes with the book, How To Break Web Software, by Mike Andrews and James Whittaker.

SSLDiggeris available on the Foundstone website : go to resources, then free tools. It allows you to test an SSL-enabled web server to determine which encryption algorithms it supports. Also on the CD that comes with the book, How To Break Web Software, by Mike Andrews and James Whittaker.

Wgetis included with most Linux and BSD distributions. It’s a simple yet powerful command-line tool for accessing, downloading, or mirroring Web server content. Also libraries.

cURL is a command line tool that is also a pen tester. It has similar functionality to Wget.

Blackwidow- a web spider or crawler tool. 30-day free trial is available, tool costs 39.95 after that.

Cygwin, which is a Unix environment for Windows. Provides, for example, the grep utility on a Windows system. Also on the CD that comes with the book, How To Break Web Software, by Mike Andrews and James Whittaker.

The Regulator- helps create search expressions for grep. Also on the CD that comes with the book, How To Break Web Software, by Mike Andrews and James Whittaker.

FITScanner is available on the CD that comes with the book How To Break Software Security, by James Whittaker and Herbert Thompson

Nikto, helps to find known vulnerabilities in a web server.

Wiktoadds to Nikto the Google Hacking Database and using the Google search engine to case your client.

GHDB, a database of hacks.

Stunnel allows you to set up a tunnel to a machine using Secure Sockets Layer. Stunnel is the "Universal SSL Wrapper" : it can be both a server and a client.

IISLockdown, a tool for locking down servers. Also on the CD that comes with the book, How To Break Web Software, by Mike Andrews and James Whittaker.

TextPadA useful text editor which can display and edit almost any file, and you can get free syntax definition files, so that TextPad appropriately highlights and indents documents (like Perl programs). - basic product isn’t free, add ons are free.

Cookie Pal- allows users more fine grained control over what cookies they will accept or reject.

Cookie Crusher- allows users more fine grained control over what cookies they will accept or reject.

http://www.securityspace.com/s_survey/data/man.200507/cookieReport.html

link to FAQ pages on cookies

paper on session fixation

BBCode

Examples of things to filter for

For more information onSQL injection techniques

chroot commandfor Apache servers

RainForrestPuppy, a pioneer of Web application security testing

checklist for locking down an application and Microsoft SQL Server

Ethereal(a network monitoring tool)

J0hnny(of Google hacking fame

HTTPrintidentifies web server and version by differences in responses to requests

SiteDiggerfrom Foundstone executes Google searches to see if your site is vulnerable to known Web server bugs

BugTraqsite that lists security vulnerabilites of web servers

CERTsite that lists security vulnerabilites of web servers

Brutusa tool for brute force hacking of authentication

Information on Cross-Site Tracing

Articles on buffer overflows

" Smashing the Stack for Fun and Profit "

http://www/securityfocus.com/archive/1/317142/2003-03-28/2003-04-03/0

http://blogs.msdn.com/michael_howard/

http://msdn.microsoft.com/security/securecode/columns/default.aspx

UTF-8 encoding

http://en.wikipedia.org/wiki/UTF-8

http://www.unicode.org/standard/standard.html

encoder/decoder tool - Napkin

Modifying an Apache server to remove weak ciphers

http://httpd.apache.org/docs/2.0/ssl/ssl_howto.html

mod-ssl http://wwww.modssl.org/docs/2.8/

Modifying an IIS server to remove weak ciphers

http://support.microsoft.com/?kbid=245030

Video Games Testing

Tools for video game testing

Fraps is a universal Windows application that can be used with all games using DirectX or OpenGL technology. In its current form Fraps performs many tasks and can best be described as:

  • Benchmarking Software - See how many Frames Per Second (FPS) you are getting in a corner of your screen
  • Screen Capture Software - Take a screenshot with the press of a key!
  • Realtime Video Capture Software - Fraps can capture audio and video up to 1152x864 and 100 frames per second!

Test Oracles -ways of partially or fully knowing the expected results of a series of test cases

Articles

Douglas Hoffman, "A Taxonomy for Test Oracles", Quality Week 1998,

Douglas Hoffman, "Heuristic Test Oracles", Software Testing and Quality Engineering March/April 1999.

Douglas Hoffman, "Using Oracles in Testing and Test Automation", part 1, part 2,part 3

Defect Prevention Process - doing root cause analysis on defects and figuring out what to do to prevent those defects in the future

Article

R. G. Mays, C. L. Jones, G. J. Holloway, and D. P. Studinski, "Experiences With Defect Prevention", IBM Systems Journal, Vol 29, No 1, 1990

Testing Patterns

Books and Articles

Robert Binder,Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison Wesley, 2000. This book is good for both white box and black box techniques applied to object oriented systems. It is also organized around patterns.

David DeLano and Linda Rising, " System Test Pattern Language ", 1997.

Michael C. Feathers, The Self-Shunt Unit Testing Pattern", May 2001.

Neil Harrison, "Developers Who Test: Practical Patterns for the Well-Rounded Developer", Better Software Magazine, May 2006, pp 33-37, available at (this is the whole magazine; print pages 33-37)

Elisabeth Hendrickson and Grant Larsen, "Architecture Achilles Heel Analysis", 2000.

Software Testing Patterns page on Brian Marick’s website " has links to further sites

The Impossibility of Testing Completely

Article

Doug Hoffman, "So Little Time, So Many Cases".

Legal Aspects of Testing - Negligence

Article

Cem Kaner, "Software Negligence and Testing Coverage", 1995,.

Debugging and researching bugs

Books

David J. Agans,Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems, AMACOM, 2002

Eric Allen,Bug Patterns in Java, Apress, 2002

Steve McConnell,Code Complete, 2nd Edition, Microsoft Press, 2004. Chapter 23 is on debugging

Everett N. McKay and Mike Woodring,Debugging Windows Programs: Strategies, Tools, and Techniques for Visual C++ Programmers, Addison Wesley, 2000

Robert C Metzger,Debugging by Thinking: A Multidisciplinary Approach, Elsevier, 2004

John Robbins,Debugging Applications, Microsoft Press, 2000

David A. Spuler, C++ and C Debugging, Testing and Reliability, Prentice Hall, 1994

Article

Andreas Zeller and Ralf Hildebrandt,"Simplifying and Isolating Failure--Inducing Input.", IEEE Transactions on Software Engineering, February 2002, Describes Delta-Debugging, which is an algorithm that shortens long testing paths to the shortest steps that will reproduce the error.

Testing Checklists/Bug Lists/Lists of Things to Look For

James Bach, "Heuristic Test Strategy Model"

Cem Kaner, Jack Faulk, Hung Quoc Nguyen,Testing Computer Software, 2nd Ed., International Thompson Computer Press, 1993. In the back of the book is a very long list of bug types.

Josh Ledgard, " Software Testing 6: Good Tests for Bad Parameters"

Brian Marick, "A Short Catalog of Test Ideas for ...."

James A Whittaker, How to Break Software, Addison-Wesley, 2003. See list of attacks in table of contents

Software Testing and Related Magazines

Journals

Software Testing Journal "Software Testing Verification and Reliability"

ASQ (American Society for Quality) publishes a quarterly Journal, Software Quality Professional

Newsletters

Software Research, Inc has Testing Techniques Newsletter (TNN Online) .

Email: ttn@soft.com

To request your free subscription or propose any type of article, send Email to "ttn@soft.com". TO SUBSCRIBE: Send Email to "ttn@soft.com" and include in the body of your letter the phrase "subscribe ".

Magazines

Software Magazine - has industry news

Better Software Magazine - can sign up for a free subscription

Software Test and Performance- can sign up for a free subscription

Software Quality Professional- published by ASQ (American Society for Quality)

Software Testing and Related Organizations

Organizations

National Institute of Standards and Technology, Software Quality Group. Articles on Software Quality. Links to related sites.

"The Economic Impacts of Inadequate Infrastructure for Software Testing" from NIST.

Software Development Forum’s center for information, connection and education.

Software Testing Institute (STI). Articles and book suggestions for testers. Industry and profession overview. Also has discussion forum.

ICSTEST International Conference on Software Testing is an annual event that is a forum for presentations, tutorials, discussions, and exchange of experience on software testing

Quality Assurance Institute site has info on SW Testing, consulting, education, assessments and certification programs.

Software and Information Industry Assocation has info on conferences, etc.

Software QA/Test Resource Center has FAQ’s resources lists tools, etc.

Tester conference site - STAR : Software Testing Analysis and Review.

Society for Software Quality

STA group offers excellent classes on software testing and automation. Based in the Seattle area.

Software Testing and Quality Engineering, a magazine for forums, publications, book reviews and other information about software testing. In 2004 renamed to Better Software magazine

Software Testing Center offers training, including online training. Based in California.

Washington Software Alliance provides resources for WA software industry. Has regular meetings.

WSA QA SIG - testing SIG - meets every 2 months; prior presentations on website.

Quality Assurance Forum, 17 St Catherine’s Road, Ruislip Middlesex HA4 7RX, UK

American Society for Quality (ASQ), 611 N Plankinton Ave., Milwaukee, WI, 53203. ASQ covers all forms of quality, including manufacturing, health care, etc. Has a software division, and has a software quality certification program.

IEEE Computer Society, PO Box 80452, Worldway Postal Center, Los Angeles, CA 98008.

  • ANSI/IEEE Std 829-1998 Software Test Documentation
  • ANSI/IEEE Std 1008-1987 Software Unit Testing
  • ANSI/IEEE Std 1012-1986 Software Verification & Validation Plans
  • available through IEEE Standards Sales in New Jersey (201) 981-0060
  • IEEE International Test Conference (ITC)
  • IEEE European Design and Test Conference (ED&TC)

Software Quality Association (South Australia) Inc.

Web Application Security Consortium, an international group which produces best-practice security standards for the World Wide Web.

Open Web Application Security Project (OWASP) is dedicated to finding and fighting the causes of insecure software.

Pacific Northwest Software Quality Conference, usually in October in Portland, OR

Seattle Area Software Quality Assurance Group has monthly free meetings on fourth Thursdays in Bellevue, WA. Quarterly $99 training days are held locally in Puget Sound area. Prior talks, both slides and recorded audio, are stored on website.

Association for Software Testing, a nonprofit association dedicated to advancing the understanding and practice of software testing. AST sponsors and annual conference, CAST.

SSQA - Silicon Valley Software Quality Association is a local professonal software QA organization with monthly meetings, newsletter, etc.

CSTER - Center for Software Testing Education and Research is a not-for-profit organization that supports the teaching and self-study of software testing.

ISO - Describes ISO (International Organization for Standardization), with links to other standards organizations.

ANSI - American National Standards Institute

NSSN - National Standards Systems Network. Search engine for standards

SEI - Software Engineering Institute - at Carnegie Mellon University. Creators of CMM and CMMI, as well as PSP and TSP.

LaRS at JPL - Laboratory for Reliable Software’s mission is to achieve long term improvements in the reliability of JPL’s mission critical software systems.

SDForum - Non-profit in San Jose with a big test lab and various other support facilities

European Software Institute - Industry organization founded by leading European companies to improve the competitiveness of the European software industry. Very interesting for the Euromethod contracted software lifecycle and documents.

Society for Technical Communication - links to research material on documentation process and quality.

Software Publishers Association - Software and Information Industry Association is the main software publishers’ trade association.

Testing Resources on the Web - Professonal For profit sites

STA group offers excellent classes on software testing and automation. Based in the Seattle area.

Software Research, Inc. - Also a consulting and tool building firm. Publishes the TTN-online newsletter.

Quality Assurance Institute (QAI) - Quality control focused on the needs of information technology groups.

Reliable Software Technologies - Consulting firm. Hot stuff on software reliability and testability. Big archive of downloadable papers. Lots of pointers to other software engineering sites.

Software Testing Institute - Quality Publications, industry research, and online services for the software development and testing professional.

Software Quality Engineering - Articles from STQE Magazine (now Better Software Magazine), form for software testing and quality engineering. Runs stickyminds site.

Software Productivity Centre - Methodology, training and research center that supports software development in the Vancouver, BC area.

Centre for Software Engineering - Committed to raising the standards of quality and productivity within Ireland’s software development community.

Quality.org - links to quality control source materials.

Other Interesting Web Sites

Lessons Learned in Software Testing - this is the home page for Cem Kaner, James Bach and Bret Pettichord’s book, Lessons Learned in Software Testing.

Bad Software home page - This is the home page for Cem Kaner’s book, Bad Software. Material on the law of software quality and software customer dissatisfaction.

QA Dude’s Quality Info Center - Over 200 quality links - pointes to standards organizations, companies, etc. Plus articles, sample test plans, etc.

Quality Auditor - Documents, links, listservs dealing with auditing of product quality.

The Object Agency - Ed Berard’s site. Object oriented consulting and publications. Interesting material.

Dilbert - Home of Ratbert, black box tester from Heck.

General sites for testing information, testing gurus, and forums

Testing Information

Software Testing and Quality Assurance discussions site

StickyMinds - site for software test managers, testers, and QA professionals to gather information and provide resources for one another - website attached to Better Software Magazine.

Bad Software - site hosted by

Info on software testing and links to other sites

SW Testing Teacher’s page. Goofy picture but has useful links.

FAQ’s about testing

Software Testing Links.

New site with testing info, tester idea exchange areas, job postings and more.

Extensive collection of testing resources

Automated Testing Specialists : great links to articles on test automation, SW testing sites, and tools.

Software testing related links page

JavaWorld.com’s Testing Article Listing page

Gurus

James Bach - Information about testing methodologies and more.

Scott Barber - Offers advising, consulting, and training services as well as resources to bring software testing expertise and thought-leadership to organizations.

Robert Binder - A different approach to object oriented development and testing.

Rex Black - testing training and consulting, also books.

Elizabeth Hendrickson - Consulting and training in software quality and testing. Her personal articles and blog entries are moved to this site.

Doug Hoffman - Consulting and training in strategy and tactics for software quality. Articles on software testing, quality engineering, and testing.

Karen Johnson - Testing services, training and defect tracking products.

Cem Kaner’s writings, courses, and links to his other sites. Articles on software testing and testing-related laws.

Mike Kelly - writes and speaks about topics in software quality and testing.

Brian Lawrence - Project Planning from Brian Lawrence and Bob Johnson.

Brian Marick’s testing site - interesting series of papers, articles and checklists.

Hung Nguyen and Hans Buwalda - testing services, training, and defect tracking products.

Brett Pettichord has put together a great list of links to articles and sites about SW Testing. Several interesting papers on test automation.

Johanna Rothman - Consulting in project management, risk management, and people management.

Robert Sabourin - Testing services to help companies, and individuals, succeed in developing quality software solutions in a timely, cost-effective and manner

Game of Life:

Website for downloadable versions of the Game of Life used in lab html

Website for patterns to run in the Game of Life program you downloaded. Start the Game of Life program, then double click on one of the patterns html

Search for more resources