In this episode Matthew and Federico discuss the differences between black box and white box testing. Based on their experience testing MVC and WebMatrix, they talk about problems that can happen if testers are too focused on code inspections as well as advantages if used properly.
In this episode Matthew talks with Scott Hunter about the new released WebMatrix, Microsoft's free tool for building websites on windows. They give you an inside look into how the team comes up with new features and products as they discuss the reasons for creating WebMatrix. They also discuss the short life cycle of the products and how that effected the way the team functioned and how the disciple lines where blurred.
In this episode Matthew talks with Scott Densmore about Test Driven Development for the iPhone. The Objective C world is slowly gaining popularity with platforms like iOS and producing High quality apps is a must. Test Driven Development is just starting to catch on in this world and Scott and Matthew discuss how Scott has been able to maintain a TDD lifestyle while developing for the iPhone.
iPhone TDD
How Test Driven Development (TDD) works on hardware
What are some of the tools
What frameworks are available (mocking, unit testing, etc.)
How does the standard TDD pattern change
Is the red/green pattern still followed
Can you still have things like Continuous Integration
Parts of the pattern that just don’t work
Personal Experiences developing iPhone applications
In this episode Matthew talks with Brad Wilson about Continuous Integration. The ASP.NET team has converted almost all of there projects to use some sort of continuous integration pattern. In this episode they discuss what CI is and how it helps the team produce a better product.
Continuous Integration
What is continuous integration
What is the idea behind it
What is the difference between an integration build and an incremental build
Details about the CI server
What is run for each build
How often builds are run
What technology is used
How does CI help the team produce a better product
In this episode Federico and Matthew talk about product metrics. Metrics are a common technique used by management to measure progress towards some goal, but they are not without risks. This show lays ground to understand what are product metrics and common pitfalls to avoid.
Attribute: measurable physical or abstract property of an entity.
Measurement: the assignment of numbers to objects or events according to a rule derived from a model or theory.
Metric: a measurement function.
Quality Factor: a type of attribute. A management-oriented attribute of software that contributes to its quality.
Software quality metric: a function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affect its quality.
Requirement statement
The most important aspect when designing a metric is to think about the intended goal.
"To [understand, evaluate, control, predict] the [attribute of the entity] in order to [goal]"
Metric design
Goal -> Questions -> Metric
Evaluation of a metric
How much do we need to know about an attribute before it is reasonable considering measuring it?
How do we know if we have really measured the attribute we wanted to measure?
Direct and indirect measurements.
Distortion: metric creates incentives for the employee to allocate his time so as to make the measurements look better rather than to optimize for achieving business goals.
Dysfunction: if optimizing for a measurement so distorts the employee's behavior that he provides less value to the organization than he would have provided in the absence of measurement.
Examples of bad metrics:
Bug counts as a metric of the work of testers and programmers.
In this episode Matt and Federico have fun talking about tips to survive the corporate jungle. After too many serious episodes, in this show we joke about our secret strategies to avoid getting thrown under the bus.
Surviving the Corporate Jungle:
What to do when somebody asks for volunteers in a meeting?
How to handle feedback from a review meeting?
What to do when your manager tells that “this is a very good opportunity for you” and will “increase your scope of impact”?
Teach, don’t answer.
Avoid offering a suggestion you are not willing to do.
In this episode Federico and Matthew talk about their experience testing installers. Taking from their experience testing the ASP.NET MVC installer, they share tips and advice for those faced with the challenge of testing the very first experience of many products: the installer.
Before somebody gets to use a product, they have to get it into their machines. It is the first experience with a product.
When building an installer test plan start by brainstorming 4 general areas:
Pre-requisites. How does the installer check for requirements? Does it provide enough information if requirements are not met? Does it fail as soon as possible?
Installation: What files are copies and where? Registry changes? GAC changes? Folders created? IIS changes? Does it need to check the web?
Uninstall: Is everything rolled back completely? Is anything is supposed to be left behind?
Machine state: OS, bitness, other applications running, hardware, disk space, network access, permissions.
Based on these variables, start building combinations of scenarios.
Use additional heuristics for identifying and discussing special cases:
Interruptions: Machine goes into hibernation, unplug network cable, user cancels the installer.
Full circle: Snapshot the state of the machine before installing, make sure it is the same after uninstalling.
Side by Side: Previous versions of the software installed, multiple versions installed.
Reinstall/Repair: Reinstall multiple times, break the application and repair it.
Temp folder: what is written in the temp folder.
Unintended changes: Additional files in placed installation directory.
Partial installs: some of files that are meant to be installed are already present, some of the files that are meant to be removed are not there.
In our experience, installer testing is closer to scripted testing than exploratory testing.
In this episode Matthew and Federico sit down with Damian Edwards to talk about his experience working for a company that writes functional tests as part of their development cycle. Damian is a new Program Manager in the Visual Web Developer team and used to work for Readify, a consulting company.
Interview with Damian Edwards
Introduction
Types of testing that an agile team can do.
Unit testing
Integration testing.
Functional testing
Performance/Load testing.
Working on a team that releases to web every week.
Who is responsible for what type of testing?
One week is too small of a time to do dev and test work.
Developers implement features in one week, testers test it the next.
What happens if a tester finds a bug?
Using functional automated tests.
Is there enough time for testers to write them?
What automation framework did you use?
How did you deal with fragile tests?
Making markup easier to automate vs. making markup minimal for client.
In this episode Matthew and Federico talk about the quality of test code. They share what to look for when reviewing test code and their experience dealing with automated test cases written by a large team.
News
One year anniversary of the Coding QA podcast!!!
Test Code Quality
A sense of pride on the code you write.
Reviewing tests is more effective when done early.
What to look for when reviewing tests?
How easy it is to figure out why a test failed from the log?
How good is the test at catching the intended bug?
How resilient is the test to other changes?
Over verifying and under verifying.
Does it align to .NET coding guidelines?
Common excuses by testers
I don't have time.
I don't understand what is the value.
Test code is not the same as product code.
Test engineers produce code, as professionals we must strive for the best code.
In this episode Federico and Matthew talk about test heuristics. Starting with what are they and why they are useful, they give a run down several heuristics that the ASP.NET QA team uses while exploring software.
Disclaimer: The opinions expressed herein are the personal opinions
of Matthew, Federico, and their guests and do not represent their employer's view
in any way.
Zune
iTunes
Other Options
