Fake it till you make it!

How IBM saved millions by NOT building a PROTOtype.

Objective: testing speech-to-text software

This is not an example from my personal experience, but certainly worth mentioning, since it explains very well how far back one can go in testing a new product on the general public.

Marketing research by IBM showed that many people would be very interested in the prospect of speech-to-text software.

IBM quickly learned that marketing research doesn't always render reliable results.

The initial reaction would be to develop the software for it, after which a test panel could try it out. But wait, isn't there another, easier way?

Chosen solution: pretend it's software

IBM wisely decided to not build the software, which would have taken a few years at least. Instead they chose to test speech-to-text out with regular test candidates and a real typist, hidden from view. This is called Pretotyping (from 'Pretending'). Beside the obvious time and cost savings, the great advantage was also that the typist was probably much better at understanding speech than any software at the time could have been.

Surprisingly, it turned out that people didn't really like speech-to-text software, even though many had indicated enthusiasm beforehand. This is understandable, a speech interface for a regular text document might be reasonably convenient, but controlling an application is another thing. For instance, telling the word processor to go two words back and correct a particular text can be cumbersome.