Friday, July 17, 2009

Born Digital Research Methods

One of the problems with the net generation discourse is that, for the most part, it is not being driven by issues that have been identified in academic research. Instead, educators are responding to the hype, speculation and murky research in the lay press and often accepting uncritically the claims that these writers are making. The popular literature that does claim to have a basis in research rarely reports the kind of methodological detail that would allow readers to make an informed judgement of its quality.

Elsewhere I highlighted the methodological problems with Grown Up Digital. Surprisingly, despite being the work of two academics, Born Digital provides us with even fewer methodological details. So, it may well be based on sound research, but all we are told about the research that informs the book is contained in two paragraphs:
  • They conducted a series of focus groups and interviews of young people.
  • They held 100 converstaions with young people from around the world about the technologies they use, their online identities and their views on privacy and safety.
  • They held conversations with about 150 informants.
They tell us nothing about how the data was analyzed, how the informants and interview subjects were chosen, what specific questions were asked, nor how their study is grounded in the existing literature. These, of course, are the requirements of academic research, not popular writing, but the problem is academics are citing the popular net gen literature as if it were academic research. I have stopped counting the number of articles that refer to claims made by Prensky, Tapscott and other as if they were based on conclusive evidence. Some like Danah Boyd even argue that academics worry too much about academic rigour and should be more willing to accept generalizations: "Academics tend to err on the side of nuance and precision, eschewing generalizations and coarse labels. This is great for documenting cultural dynamics, but not so great for making intervention." But isn't this precisely the problem? Interventions are being advocated, based on speculation and/or research that has not undergone the accepted process of scholarly review and publication. It is fine to raise the issues in the popular press but when the claims are accepted uncritically by educators and cloaked in an aura of research respectability, we have problems.

8 comments:

Mark said...

Good call, Mark. Some references that may be of interest (though you no doubt already have them) can be found at http://it.coe.uga.edu/itforum/paper114/Extending_Possibilities.pdf, section 5.1.3. In draft format, but largely robust! Keep up the great work.

Mark Bullen said...

Thanks Mark, I'll check out the document and references.

Mark.

Unknown said...

Mark, I appreciate your comments on our book. I was wondering how you can conclude that the book "does not contain a bibliography or reference list". If you take a look at pp. 295-344, you'll find many references to academic and other literature. And on pp. 353-363 we provide a "selected bibliography". I appreciate your "skeptic" reading of the book, which hopefully doesn't have an impact on how careful you read it.
Best,
-Urs

Mark Bullen said...
This comment has been removed by the author.
Mark Bullen said...

Urs:

My apologies for the error about the bibilography and reference list. I will correct that.

Mark

Chris said...

I spoke with a representative from Certiport this past week; I've been working with them to, at some point in the near future, use the new IC3 standard to assess our early college learners' computer and information literacy skills. The rep indicated they are encountering more higher education institutions which are making the general assumption that learners already possess computer and information literacy skills - to the point that they are dropping Introduction to Computers and/or Microcomputer Applications type courses from the basic requirements. I believe that's one of the negative policy implications of the over-generalizations made by net generation proponents.

Chris said...

@Urs...
Born Digital absolutely includes a bibliography, but one of the things I've noticed in several different pro net-generation pieces is that they cite each other, and to my knowledge, the ones most often cited - including Oblinger & Oblinger - are not empirically based research.

That's not to suggest the articles are not well informed; they are based on well informed experiences, wisdom of the authors, and general observations of trends and issues within the marketplace. Changes are needed within higher education.

However, as Mark points out in his blog entry, many academics are referencing the net-generation literature as though it is based on foundational, empirical research. As a result, there appears to be more than a few policy decisions being made based upon the net generation rhetoric; the rhetoric may serve a purpose and be useful, but the research behind it doesn't justify or support some of the decisions being made.

It's likely a problem more with the users of the information rather than the authors of it.

Mark Bullen said...

Thanks for your comments Chris. One of the most insidious aspects of the net gen discourse is they way in which unsupported claims get repeated so often that they become accepted.

And yes, many of the articles and books that I am critical of are based on the well-informed experiences of the authors and they raise important issues. The problem is they too often conclude with outlandish claims that aren't supported by their own evidence.

But as you say, it is the users of the information (e.g. educators, educational administrators and even some researchers) who are really at fault for not being more critical and perhaps I should emphasize that more strongly.

Mark