Methodology Matters in Digital Learner Research

The study by Hargittai et al. (2010) that I reported on in my previous post is and example of a methodologically sound piece of research. This is unlike much of the popular writing on this topic which tends to be speculative, anecdotal or research based on questionable methodology. Educators have tended to pay more attention to the latter than the former. The result is a widepread belief that all people born after 1982 are technologically sophisticated and digtially literate and that we need to make radical changes to our educational systems to accomodate this generation. As the growing body of sound research, like that of Hargittai et al. (2010) is showing, this popular view is not grounded in fact.

Here are some key methodological strengths of the Hargittai study:
1) they actually observed user behaviour rather than relying on self reports and they didn't restrict what Web sites participants could consult: "Our findings suggest that utilizing this more naturalistic method allows us to uncover user practices that have been hard to capture using earlier methods."
2) They linked trustworthiness and credibility to branding which has been neglected in earlier studies and they took into account the full search context in their investigation: "How users get to a Web site is often as much a part of their evaluation of the destination site as any particular features of the pages they visit."
3) They used a mixture of quantitative and qualitative methods.
4) They administered a paper/pencil survey, "to avoid biasing against people who feel less comfortable filling out Web forms or who spend less time online and thus may have less opportunity to participate."

Look at the popular Net Gen literature and you won't find this kind of methodological rigour. And you won't find this conclusion:
"Students rely greatly on search engine brands to guide them to what they then perceive as credible material simply due to the fact that the destination page rose to the top of the results listingsof their preferred search engine."

A British study released last year came to similar conclusions about the information seeking behaviour of what it called the "Google Generation", a slighter younger group born in 1993 and later.

Grown Up Digital Research Methods

In Grown Up Digital, Don Tapscott makes some substantial claims about the Net Generation and its impact on business, education and society in general. He calls for major changes to our educational institutions and to how employers treat employees. His recommendations are based on the results of a $4 million research project. But how was that research conducted?

One of the ways we determine the quality of research is by examining the methods used. We want to be sure the research methodology is appropriate, that appropriate sampling and analysis techniques are used, that there is no bias and that the conclusions are supported by the evidence. But the research that informs Grown Up Digital is proprietary. In other words, it was conducted under contract for several businesses. As a result, only some of the "high level findings and main conclusions" can be shared publicly. This means the reader has little opportunity to assess the quality of the research.

While Tapscott doesn't provide the full details of his methodology that would allow for a proper assessment, he does share some of the methodology:

• Data was gathered from a sample of 7685, composed of randomly selected Internet users, stratifed to avoid gender or socioeconomic bias.
• Interviews were conducted using an online questionnaire.
• Facebook group was used to collect over 200 stories.
• Discussions on a global online network TakingITGlobal were conducted and analyzed

While the sample is large, and the sources varied, there is an obvious problem here. The data was gathered from people who are already engaged with digital technologies: Internet users, Facebook users and participants of an online discussion. This is a biased sample. One of the main goals of the research was to determine how engaged this generation is with digital technology and whether there are generational differences. Might the results have been different if people who aren't active users of the technology were sampled?

This highlights the value of academic research and why we need to be careful about how we use proprietary research. Academic research is subject to peer review and it requires transparency and openness of methods. With proprietary research, it is up to the sponsor to decide what to make publicly available.