Grown Up Digital Research Methods

In Grown Up Digital, Don Tapscott makes some substantial claims about the Net Generation and its impact on business, education and society in general. He calls for major changes to our educational institutions and to how employers treat employees. His recommendations are based on the results of a $4 million research project. But how was that research conducted?

One of the ways we determine the quality of research is by examining the methods used. We want to be sure the research methodology is appropriate, that appropriate sampling and analysis techniques are used, that there is no bias and that the conclusions are supported by the evidence. But the research that informs Grown Up Digital is proprietary. In other words, it was conducted under contract for several businesses. As a result, only some of the "high level findings and main conclusions" can be shared publicly. This means the reader has little opportunity to assess the quality of the research.

While Tapscott doesn't provide the full details of his methodology that would allow for a proper assessment, he does share some of the methodology:

• Data was gathered from a sample of 7685, composed of randomly selected Internet users, stratifed to avoid gender or socioeconomic bias.
• Interviews were conducted using an online questionnaire.
• Facebook group was used to collect over 200 stories.
• Discussions on a global online network TakingITGlobal were conducted and analyzed

While the sample is large, and the sources varied, there is an obvious problem here. The data was gathered from people who are already engaged with digital technologies: Internet users, Facebook users and participants of an online discussion. This is a biased sample. One of the main goals of the research was to determine how engaged this generation is with digital technology and whether there are generational differences. Might the results have been different if people who aren't active users of the technology were sampled?

This highlights the value of academic research and why we need to be careful about how we use proprietary research. Academic research is subject to peer review and it requires transparency and openness of methods. With proprietary research, it is up to the sponsor to decide what to make publicly available.

An Informed Review of Grown Up Digital

In an earlier post, I took a somewhat skeptical view of Don Tapscott's latest book, Grown Up Digital. I said I was put off by the techno-utopic language but encouraged by the amount of data he had collected. At least one reader took me to task for appearing to pass judgment without having read the book. Fair enough.

Well I"m now making my way through the book so I'm in a positon to make a more informed review. Over the next few days I will post my observations beginning with the following:

According to Tapscott, there are eight "norms" that distinguish the Net Generation from other generations. One of them is what he calls "Scrutiny": "Net Geners are the new scrutinzers. Given the large number of information sources on the Web, not to mention unreliable information - spam, phishers, inaccuracies, hoaxes, scams, and misrepresentations - today's youth have the ability to distinguish fact from fiction...The Net Generation knows to be skeptical whenever they're online." (p. 80)

But he goes even further: "On the Net, children have to search for, rather than simply look at, information. This forces them to develop thinking and investigative skills – and much more. They must become critics. Which Web sites are good? How can I tell what is real and what is fictitious – whether in a data source or in the teenage movie star in a chat session.” (p. 21)

How do we reconcile these claims with the results of a substantial study conducted in the UK that found exactly the opposite:

• the information literacy of young people, has not improved with the widening access to technology: in fact, their apparent facility with computers disguises some worrying problems

• internet research shows that the speed of young people’s web searching means that little time is spent in evaluating information, either for relevance, accuracy or authority

• young people have a poor understanding of their information needs and thus find it difficult to develop effective search strategies as a result, they exhibit a strong preference for expressing themselves in natural language rather than analysing which key words might be more effective

• faced with a long list of search hits, young people find it difficult to assess the relevance of the materials presented and often print off pages with no more than a perfunctory glance at them

The problem is Tapscott's conclusion is not based on the right evidence. The evidence he uses is the Net Gen respondents self-reported online behavior when searching for product information: "Almost two-thirds of Net Geners tell us they search for information about products that interest them before purchase. They compare and contrast product information online; they read blogs, forums and reviews; and they consult friends." (p. 323). But this is hardly evidence of critical thinking skills and highly developed information literacy skills.

More in the coming days.