Grown Up Digital Research Methods

In Grown Up Digital, Don Tapscott makes some substantial claims about the Net Generation and its impact on business, education and society in general. He calls for major changes to our educational institutions and to how employers treat employees. His recommendations are based on the results of a $4 million research project. But how was that research conducted?

One of the ways we determine the quality of research is by examining the methods used. We want to be sure the research methodology is appropriate, that appropriate sampling and analysis techniques are used, that there is no bias and that the conclusions are supported by the evidence. But the research that informs Grown Up Digital is proprietary. In other words, it was conducted under contract for several businesses. As a result, only some of the "high level findings and main conclusions" can be shared publicly. This means the reader has little opportunity to assess the quality of the research.

While Tapscott doesn't provide the full details of his methodology that would allow for a proper assessment, he does share some of the methodology:

• Data was gathered from a sample of 7685, composed of randomly selected Internet users, stratifed to avoid gender or socioeconomic bias.
• Interviews were conducted using an online questionnaire.
• Facebook group was used to collect over 200 stories.
• Discussions on a global online network TakingITGlobal were conducted and analyzed

While the sample is large, and the sources varied, there is an obvious problem here. The data was gathered from people who are already engaged with digital technologies: Internet users, Facebook users and participants of an online discussion. This is a biased sample. One of the main goals of the research was to determine how engaged this generation is with digital technology and whether there are generational differences. Might the results have been different if people who aren't active users of the technology were sampled?

This highlights the value of academic research and why we need to be careful about how we use proprietary research. Academic research is subject to peer review and it requires transparency and openness of methods. With proprietary research, it is up to the sponsor to decide what to make publicly available.

An Informed Review of Grown Up Digital

In an earlier post, I took a somewhat skeptical view of Don Tapscott's latest book, Grown Up Digital. I said I was put off by the techno-utopic language but encouraged by the amount of data he had collected. At least one reader took me to task for appearing to pass judgment without having read the book. Fair enough.

Well I"m now making my way through the book so I'm in a positon to make a more informed review. Over the next few days I will post my observations beginning with the following:

According to Tapscott, there are eight "norms" that distinguish the Net Generation from other generations. One of them is what he calls "Scrutiny": "Net Geners are the new scrutinzers. Given the large number of information sources on the Web, not to mention unreliable information - spam, phishers, inaccuracies, hoaxes, scams, and misrepresentations - today's youth have the ability to distinguish fact from fiction...The Net Generation knows to be skeptical whenever they're online." (p. 80)

But he goes even further: "On the Net, children have to search for, rather than simply look at, information. This forces them to develop thinking and investigative skills – and much more. They must become critics. Which Web sites are good? How can I tell what is real and what is fictitious – whether in a data source or in the teenage movie star in a chat session.” (p. 21)

How do we reconcile these claims with the results of a substantial study conducted in the UK that found exactly the opposite:

• the information literacy of young people, has not improved with the widening access to technology: in fact, their apparent facility with computers disguises some worrying problems

• internet research shows that the speed of young people’s web searching means that little time is spent in evaluating information, either for relevance, accuracy or authority

• young people have a poor understanding of their information needs and thus find it difficult to develop effective search strategies as a result, they exhibit a strong preference for expressing themselves in natural language rather than analysing which key words might be more effective

• faced with a long list of search hits, young people find it difficult to assess the relevance of the materials presented and often print off pages with no more than a perfunctory glance at them

The problem is Tapscott's conclusion is not based on the right evidence. The evidence he uses is the Net Gen respondents self-reported online behavior when searching for product information: "Almost two-thirds of Net Geners tell us they search for information about products that interest them before purchase. They compare and contrast product information online; they read blogs, forums and reviews; and they consult friends." (p. 323). But this is hardly evidence of critical thinking skills and highly developed information literacy skills.

More in the coming days.

Skepticism from Down Under

Christopher Scanlon writes in The Australian:

"The question is, why is there such a divergence between claims about digital natives and the realities of the classroom?"

His answer echoes many of the concerns we have about this issue. He identifies class, commercial interest and confusion as contributing to the divergence:

"It's partly a matter of class because there are some students who perfectly fit the mould of digital natives. For example, it's not surprising that two Harvard law professors were among the first to write about such students...The digital natives theory is also partly driven by commercial interest. It's worth noting that Prensky, who was one of the first to popularise the idea of the digital native, is also the chief executive and founder of Games2Train, a company that specialises in creating computer learning games...Confusion also plays a part. It's telling that most of the accounts of digital natives come not from natives themselves but from middle-aged people observing young adults using computers. Those writing about digital natives confuse the ability to navigate around ready-made online environments or download content from the net for a general ease with technology."

Read the full article.

Generation is Not the Issue

Here is the presentation of the results of the research that looked at how students at the BC Institute of Technology are using information and communication technologies. The results clearly show that generational differences are not the issue. Contextual issues such as the nature of the program are more important considerations when making decisions about the integration of learning technologies.

Here's the SlideShare version which allows you to control the slides. Note, however, the audio does not sync up properly when you use the slide advance.

Net Gen P Dday09 Final

View more presentations from markbullen. (tags: learner millenial)

Communication Preferences of the Net Generation

Further to the previous post on the BCIT research, we also examined our students' communication preferences. We wanted to find out how they were using information and communication technologies (ICTs) to communicate with their peers and with their instructors. We were also interested in finding out if there were any differences in ICT use and age.

Our survey of a random sample of over 400 BCIT students did find some statistically significant differences but overall, we found that net generation and non-net gen students were not using ICTs more than face to face communication to interact with their peers.

We found that net gen students were more likely to use instant messaging, text messaging, Facebook/MySpace and phone to communicate with peers than non-net gen students.

However, when communicating with instructors, the only significant difference in use of ICTs between net gen and non net gen was with WebCT. Non-net gen students were more likely to use WebCT than net gen students.

By far the most common mode for communicating with instructors for both net gen and non net gen sudents is talking in person.

More Evidence Supports the Need for Caution

Results of research being conducted that the BC Institute of Technology cast serious doubt on many of the Net Generation claims.

We surveyed a random sample of over 400 students to determine the extent to which the students in the net generation category exhibited the characteristics that have been attributed to this generation by people like Don Taspscott, Marc Prensky, and Neil Howe & William Strauss, and others. These characteristics include:

1. Digitally literate
2. Preference for structure and experiential learning
3. Social
4. Goal oriented
5. Community minded
6. Connected
7. Multitaskers
8. Preference for group work
9. Aversion to reading and text

The results show that at BCIT there is no statistical difference between net gen and non-net gen students on items 1-5. But even for items 6-9, where there were statistically significant differences, the effect sizes were very small, representing only between 1.4% and 2.9% of the variance.

Google Generation Study Casts More Doubt

This comprehensive study of how young learners are searching for and researching content was commissioned by the British Library. While it focuses on the implications for libraries, it contains some valuable insights into how these learners use information technologies. The results tend to contradict much of the prevailing hype about the net generation. The study defines the "Google Generation" as anybody born after 1993.

Here are some of the claims about this group that the study refutes:

1. They have zero tolerance for delay and their information needs must be fulfilled immediately
2. They are the `cut-and-paste’ generation
3. The find their peers more credible as information sources than authority figures
4. They need to feel constantly connected to the web
5. They prefer quick information in the form of easily digested chunks, rather than full text
6. They are expert searchers

The study did find evidence to support the following claims:

1. They are more competent with technology
2. They have very high expectations of ICTs
3. They prefer interactive systems and are turning away from being passive consumers of information

With respect the claim that this generation multitasks in all areas of their lives, the study concluded there is no solid evidence to support this but "it is likely that being exposed to online media early in life may help to develop good parallel processing skills. The wider question is whether sequential processing abilities, necessary for ordinary reading, are being similarly developed."

Thanks to Agnes Bosanquet and the McQuarrie University Learning & Teaching Centre blog for drawing my attention to this study.

Digital Wisdom or Digital Cynicism?

I've never liked the digital native/digital immigrant terminology because it is simplistic and inaccurate. It divides people into two categories and attributes behaviours, attitudes and ways of learning to them, based solely on when they were born. The person who coined those terms, Marc Prensky, has now come up with a new one, digital wisdom.

"Digital wisdom is a twofold concept, referring both to wisdom arising from the use of digital technology to access cognitive power beyond our innate capacity and to wisdom in the prudent use of technology to enhance our capabilities...Leaders are digitally wise when they use available techniques to connect with their constituents for polling and to solicit contributions and encourage participation, as Barack Obama did so well in the 2008 U.S. presidential campaign. "

Well, I'm not sure about this term either. My conception of wisdom doesn't include using digtial technology to convince people to give you money and vote for you. What is the difference between digital wisdom and digital literacy? For me, wisdom must have a moral dimension. Would Coca Cola be considered digitally-wise because it uses demographic data from social networking applications to increase its sales and market share?

Read the full article and judge for yourself.

More on Generational Differences

Here's another review of the literature that raises serious questions about the conventional wisdom around generational differences. According to Thomas Reeves and Eujong Oh, "Generational differences are the subject of much popular speculation but relatively little substantive research. Among the speculations are suggestions that instructional designers should take generational differences into account when developing instruction and that games and simulations will be more effective learning environments with today's younger generation than they have been with earlier ones...Most of the popular literature on the subject...appears to rest on limited data, almost always conducted by survey methods characterized by a lack of reliability and validity data."

Read the full chapter from the Handbook of Research on Educational Communications and Technology (2007), edited by J. Michael Spector, M. David Merrill, Jeroen van Merrienboer, Marcy P. Driscoll.

Thanks to Norm Friesen for drawing my attention to this.

We Still Need to Change

One point I don't make frequently enough is that I am not anti-technology. My critique of the net gen, digital native discourse is not meant to support the status quo but rather it is motivated by a desire to ensure that our decisions are based on evidence, not hype. We do need to change and adapt to the new technologies but we need to be sure we are making appropriate changes and not changes that are driven by a naïve view of who our learners are and what their needs are.

Anoush and Littlejohn make the case for change:

"As students look to their lecturers for clues as to how to use technology tools for learning, many lecturers are unaware of the potential of these tools, since they themselves are not using emergent technologies for their own learning and work. While some lecturers recognise the educational value of some emergent technologies, others view these as ‘fads’. This situation could become exceedingly problematic as many social technologies such as blogs, wikis, and virtual worlds are progressively adopted by organisations, where employees are required to use them regularly for knowledge sharing and communication. This raises the question as to how well universities are preparing students for employment if they continue to dismiss these tools and more importantly the processes and philosophies of learning and collective knowledge creation underpinning these tools. " (pp. 22-23)