Replication & Open Scholarship

One of my favourite blog sites is Retraction Watch. It's a little like watching Border Security: Australia's Front Line, a show that

takes viewers behind the scenes of Australia's Customs, Immigration and Quarantine departments

 Watching folks trying to sneak past people intent on catching smuggled contraband has a morbid fascination, the driver of reality TV around the world. But when it intrudes on your professional world it starts to look a little bit different.

PubMed was featured recently in the NY Times article (Published: April 16, 2012) on the rise of academic fraud in the scientific publishing community. The 'currency' of academia sure isn't in denominations of fiscal exchange (with the minor exception of grant funding featuring ever more prominently in academic staff rankings as more meaningful metrics of contribution remain elusive). The real currency of the academy is reputation.

Reputation is established by peer-reviewed publication in journals that exert care and critical attention to the quality, process, and conclusions presented for publication to the scholarly community.  That process of carefully curated, critically reviewed sharing of research outcomes is under siege from both within and without. From within by the challenges of sustaining  quality procedures that insure the highest calibre of writing, methodology, and presentation of accurate, reproducible results. And from without by the rapidly changing means of communication, production, dissemination and sharing of the outputs from this process.

The troubling reality is publication fraudulent research reports is on the rise. That could be for a number of reasons - greater care in the review process catching things that before might have slipped through; increasing sloppiness in that process allowing things that shouldn't pass into 'print' (used loosely for distribution in various media formats); increasing pressure on researches to get "their data" published in the pursuit of increasingly scarce research dollars; and increasing numbers of researchers who are unethical (whether consciously aware of it or not) or just plain deceitful.

The latest chapter in this concerns the failure to replicate published findings.  One of the hypotheses is that the incentives to publish predispose researchers to take liberties that influence how they report what they did. The result of this is attempts to follow published methodologies result in different outcomes. Why? Because what really happened wasn't quite what was written up in the published paper.  That adds a five reason why retractions have gone up, and another category for the list of fraudulent practices.

The Reducibility Project is setting out to test this prospect in papers published in the psychological sciences.  The goal of the project is to

...estimate the reproducibility of a sample of studies from the scientific literature. The project is a large-scale, open collaboration involving dozens of scientists from around the world. The investigation is currently sampling from the 2008 issues of three prominent psychology journals - Journal of Personality and Social Psychology, Psychological Science, and Journal of Experimental Psychology: Learning, Memory, and Cognition

There are many reasons to follow this project. We assume that things making it through the peer review process by qualified reviewers reflect careful efforts to reveal how things in nature work. If the outcomes of this publishing process don't reliably accomplish this we're in a heap of trouble.  Not only is the primary set of metrics for career reward and advancement now suspect, but our understanding of the world around us must be, as well. And the system has a built in bias that may contribute significantly toward this concern - unwillingness to publish failures to reject the null hypothesis.

Granted not every experiment can be replicated. Large longitudinal clinical trials for one. They're too expensive, take too much time, and are conducted under conditions that just can't be reproduced exactly even if you had time and money. But many others can and aren't. Journals don't like them. They prefer the shiny new factoid or the innovative discovery.  Journals are run by people, too.

But the vast number of experiments do result in failures to reject the null hypotheses. And we're blind to them. That means not only are we doomed repeat the past  - how do we know it IS the past if we don't share this information? - but we're unable to see exactly the context for experiments that do produce rejected null hypotheses.

The early indications aren't very promising. Another project in the psychological sciences has been working at the replication of experiments for a year now. PsychFileDrawer has tried to replicate 9 studies and succeeded at three.  Not a promising start.

-- pdl --

Tools for Open Scholarship - Practicing to be a eResearcher

Today I gave a talk at Research Week on our campus (the University of Queensland) on the kinds of tools one uses as a practicing academic to manage the open, social, process that my colleague Tim Kastelle sums up nicely with "aggregate, filter, and connect."

I borrowed liberally, and attributed clearly, from a talk by Ismael Peña-López that he kindly shared on Prezi earlier.  His, like mine, is CC licensed so build on it please.  You can get it from the Prezi or watch it below. His is focused more on the social sciences, where my examples tend to be drawn from the natural science community. But the arc of the talk after my intro on the current NMC Horizon Report findings (about 1/3 to 1/2 of it) is drawn from his inspiration.  The folks at  the Open University of Catalonia are really good. I have the pleasure of being on the NMC board with one of their members and if Eva would just send me her thesis proposal then I'd be overjoyed .

His talk is really very nice and can be found at http://ictlogy.net/presentations/20120215_ismael_pena-lopez_ _eresearch_social_media_social_sciences.zip

It's part of a seminar he gave, the details for which are here

Cheers.

Graduate Student Week 2012 UQ on Prezi

Shadow Illusion

This is too cool to pass up. Our visual systems are not as good as we might like when it comes to interpreting the data that are really in the external world. The Shadow Illusion is a great example of this, and in its structure tells us something about how the visual system organises data and processes information.

The visual system uses a number of queues to help it decipher and attribute meaning to what it sees. One of them is relative brightness - comparison of adjacent spots normally helps determine if one is brighter than the other. However, in this case, a shadow cast on a light square compared to a dark square also in the shadow erroneously leads the brain to conclude that the lighter square is actually white - when it's not.

Prof. Adelson, at MIT discovered this illusion back in 1995. Thanks to the Open Culture site for resurfacing this fascinating play on perception and for Stephen Landry for posting it on Facebook to bring it to a wider audience. For a more detailed explanation go to Prof. Adelson's explanation on his website.

Here's an example of demonstrating the illusion's reality -

--pdl--

Crowdsourcing Venture Captial - Kickstarter & Mimal Design's TickTok/LunaTick Project

Kickstarter - VC for the crowds - Follow Up

I've been a supporter of the crowd sourcing venture capital model of Kickstarter for some time (see my last post on this blog site, for example). It's a wonderful way to contribute to projects that you think are valuable and see them come to fruition. One of the projects I've been an 'investor' in is the TikTok+LunaTIk Multi-Touch Watch Kit by Chicago-based MINIMAL design, led by among others Scott Wilson.

One of the things that's illuminating in this process is seeing how the idea is translated into a product. For engineering designers this is their bread and butter. It's what they do. But for the rest of us, we may have some insight into the concept development or the design planning, but seeing the actual steps that go from design drawings to a sturdy, reliable functioning product often happens out of sight, if it happens as it should at all.

Kickstarter and in particular Scott Wilson has done a marvelous job keeping us 'investors' up to date on the fabrication of the TickTok and LunaTik watch bands designed to provide a wristband for the new iPod Nano, allowing you to wear it as a watch. There have been several of these put out since the Nano was released. But this one is really sweet and the design is clever.

As part of the production process there is testing that must take place to insure that the end product had the durability to withstand the wearing environment into which it will be put. How do they do that? Here's a short video that describes part of the testing process and pays attention to making the assembly simple and easy enough that it hopefully will reduce assembly worker fatigue and therefore improve assembly quality.

Very simple but critical step towards taking an inventive idea and making it into a useful innovative product. Nice job Scott.

-- pdl --

TikTok & Kickstarter: Crowdsourcing investmenting

TikTok+LunaTik Multi-Touch Watch Kits

This post is about two related phenomena. The first is crowdsourcing, the second is just about a

really cool new product!

Crowdsourcing is hardly new. It has been employed in the business world as a means of presenting challenges in design, drug development and related issues and associating creative and useful solutions with financial gain. InnoCentive, for example, posts challenges on its website for people to solve with a price tag for the best solution. In today's list of challenges the top four are:

• Irreversible Low Temperature Indicator
• Selective Ascorbate Scavenger
• Oligodendrocyte Precursor Immortalized Cell Line
• Shelf Ready Display Cases (Packaging)

As you can see, these tend to be quite technical but the solution rewards are comparable, $40,000, $30,000, and the last two $20,000 USD. Perhaps that's not a lot in terms of the value to the companies posting these challenges, but it's not a bad day's work either.

Kickstarter brings people with ideas and products forward seeking investors to help them fabricate and bring them to market. So it's a marketplace of people with ideas needing investors, and rather than go to the VC world they're asking you to help. Investments can be as little as $1 USD or higher. The product that I fell in love with is the TikTok which until Dec. 15 has a minimum investment of $25 USD (after that the fundraising is closed).

The TicTok and it's more permanent companion the LunaTik are thoughtfully designed wristwatch holders for Apple's new iPod Nano. When Jobs announced the product he mentioned that one of the member's of Apple's board of directors was planning to use it as a wristwatch. Scott Wilson, the 'father' of a small design firm called Minimal, posted their design idea for making a wristwatch holder for the iPod Nano on the Kickstarter website. They hoped to raise $15k to get the process going. As of Sunday, Dec. 5th, they have gotten investments from 8,192 people (at the minimum of $25 but some have pledged $500 or more), raising $582, 221 so far! That's only 38 times more than their original goal.

Watch the video and see what you think. This is an exciting way for the public to get involved in bringing to market products that we want. I remember when Netscape was going public. I was sailing up the Long Island Sound at the time and Maryann and I were coming into Greenwich Yacht Club to pick up a mooring for the night. This was back in the early 90's (well Aug. 9, 1995, to be exact). I like anyone who was deeply involved in technology back then knew that this was going to be a huge deal. We were walking up through town and I saw the ticker on the bottom of the screen of CNN tracking the market and watched in amazement. The offering price initially was set at $28 a share. It rocketed up as high as $75 that day and closed at $58. This might not seem like much in light of the more recent Internet bubble/bust cycle, but in 1995 it shook markets and investors to the core, a wake-up call that something fundamentally different was happening. But the average Jane couldn't really participate. Places like Kickstarter give all of us a chance to support and help bring to market things that we really think are valuable, useful, and sometimes just plain cool. In this case, TikTok ticks all three boxes.

-- pdl --

Resistance or experience?

I had the pleasure of spending some time recently with the engineering academic staff from the University of New South Wales (UNSW) at their end of academic year senior staff retreat (dean, associate deans, heads of school). My role was novel in that I was the first outsider to have ever been invited to these executive retreats. Either this was going to open up such possibilities for the future of forever close them down. Time will be the judge.

The topic was two fold. Talk about the emerging learning environments that are being built around the world in response to the growing recognition that didactic lecture is often misaligned with learning objectives. The corollary conversation that inevitably accompanies this discussion is about the technologies that enable or inhibit the learning processes for which they are intended.

Secondly, what are the learning activities that academic staff want their students to experience? In reality this should precede the discussion of learning spaces as they ought to serve as their design guidelines. Sadly this happens all too infrequently. It usually is the case that the design, technology, and related affordances of new spaces triggers rather than follows discussions about learning activities as the unrest rises with the unveiling of new teaching environments. "We have theses new collaboration rooms. But what do we really do with them?"

What followed was an active discussion, rather than 90 minutes of 'death by PowerPoint'. What struck me through the course of this discussion is something that I've experienced before but it seems to be occurring with both greater frequency of late and certainly with greater emphasis & depth of feeling. That is, the forceful "knowing conviction" that introductory courses need, indeed must be designed to provide the experienced academic the opportunity to explain and, in the framework of this argument, motivate the young (naive?) students to appreciate why they are going to have several semesters of maths, chemistry physics, and discipline specific engineering courses in the coming years. Only by virtue of this careful and well orchestrated overview by the knowledgable and experienced academic will the student have the context to appreciate the connection between the foundation courses and the engineering challenges that lie ahead. They need to be prepared to confront the development and production difficulties that, for example, the chemical process engineer will encounter and have to solve.

This sounds like the voice of experience from someone who's "been there and done that", and wants to offer their guidance and scaffolded insights to these impressionable young minds. It certainly comes from a deep conviction that it is necessary and really irresponsible to take another path. A gentle question asking if perhaps the student needs to encounter these issues on their one terms to build this framework themselves is quickly countered with the certainty from years in the field dealing with unprepared and otherwise unequipped students who simply wouldn't understand what to do or think without this scaffolding hammered home. How would they ask the right questions? Why would they ask anything at all from their foundation-free, construct thin frame of reference? These aren't MIT students, after all. Don't think that what works at such an elite institution will work here.

And this may be right - for vast majority of students who find themselves in middle of the class distribution, and who are just following along in their career path that is as much laid out for them as it is chosen by them.

But it resonates with the same discussion and arguments that emerged during the development and piloting of MIT's Technology Enabled Active Learning (TEAL) redesign of first year physics. As Yogi was want to say, "it's deja vu all over again."

Lori Breslow, the Director of the MIT Teaching and Learning Laboratory wrote a recent article in Change magazine (Breslow, L.. (2010). WRESTLING WITH PEDAGOGICAL CHANGE: THE TEAL INITIATIVE AT MIT. Change, 42(5), 23-29. Retrieved November 12, 2010, from Academic Research Library. (Document ID: 2140404421) describing the history of educational innovation in a research intensive university through the lens of TEAL In it she wrote, "While TEAL had its supporters among the faculty, it also had its detractors who sought-with various degrees of effort-to undermine it. Some faculty simply believed TEAL was bad pedagogy. One well-respected faculty member who had extensive experience teaching Physics I wrote a long critique of TEAL that began, "What I don't like about the TEAL format is that it seems to be effectively based on the premise that lectures are obsolete."

She went on to write something that sounded like it had come directly from the conversations I'd just had over the past few days in discussing alternatives to broadcasting content through lecturing in search of more valuable uses of the precious "face time" between students and academic staff. Breslow wrote "Lecturing, in this faculty member's view, allowed the instructor to "lay out the logic of physics -the beautiful way in which just about everything that we teach in the freshman year can be seen as the logical consequence of a few fundamentally simple ideas." Deja vu all over again, again.

The conversations are separated by years, tens of thousands of miles and vast differences in cultural history. But the story they tell about the difficulties of accomplishing major changes in teaching in higher education are remarkably similar. This speaks to the very heart of the challenge that confronts the institution of higher education around the world. As Breslow noted, everything was aligned to support a major change in pedagogy represented by the development and introduction of TEAL at MIT:
- the reform was centered in the department
- external pressure existed to make changes in the first year physics program because of perceived higher than appropriate failure rates
- support was strong from the department head, associate dean for education, the dean for undergraduate education and the provost. When th times were tough, they defended the reform and committed to 'stay the course'.
- it had a faculty champion with an established record of research (that matters more than data or video evidence of merit)
- finally, the reform took place in the midst of a major investment in pedagogical change and technology development for teaching ($35 million dollars from a donor alum and Microsoft Research).

It's now nearly eight years on and the reform continues, but it remains a work in progress. And therein is a message about how difficult it is to achieve substantive reform even when everything is in place. That is an important recognition and at the same time deeply worrying. Breslow ends with this unsettling question:

"What TEAL demonstrates is that successful educational innovation requires an enormous amount of effort and a good deal of luck. For TEAL, the stars were in alignment-the ingredients required for major pedagogical reform were there. But the question that higher education needs to ask itself is, why does this have to be so hard?"

Seymour Papert wrote decades ago about how it is essentially impossible to reform or make fundamental changes in organisational structures as complex as schools. The complexity of forces all pushing to sustain homeostasis is overpowering. Perhaps that's both the realisation and the pathway out of the dilemma. John Maeda's Laws of Simplicity, transformed by MobiusView's to "simplify, learn, connect", and channeled by my colleague Tim Kastelle into "aggregate, filter, and connect". The bottom line: simplicity has a more value that we acknowledge.

-- pdl --








- Posted using BlogPress from my iPad

Location:Manly Beach Q-Station

Another Airport

After 14 hours in a tube with wings I'm again in another hemisphere. The attempt to maintain a bi-hemispheric professional life is bumping against the physical constants of the universe. While time passes inexorably in its linear narrative, the astronomical phenomenon of planetary orbits and angles of incidence make these 14 hours pass differently to an animal carrying evolutionary baggage. Thirty words that sum up to, "I'm tired."

Re-entry into the "States" this time is shaded with differences. The lines were long, giving me time to reflect as well as read a few more pages from Straight Man (Richard Russo). I had been in a prior line, which was long and congealed, going no where. A helpful gate agent opened up the elastic cattle cordon and pointed to the far end of the entrance hallway saying there was shorter line down there. I was at queue 31. Queue 1 was just being opened. Between queue 24 and 1 the cubicles where custom's agents sit were all empty. At the far end there appeared a small climb of humanity. I walked in that direction, but approaching saw it was a line of people in wheelchairs. Surreal and unsure I asked if this was the line to which I was directed but received only a silent nod, suggesting yes, despite the absence of wheels transporting me along the queue.

When I finally walked up to the custom's agent she looked at me, book in one hand, passport and entry form in the other, and said "you're a professor, right?" Was it written all over the tiredness in my face? I wondered if she'd glanced down at the custom's form where I had written that in the space associated with work, but if she had, she was very stealthy. At 7:00 am I suspect that was not likely, even for her. I acknowledged the identify, caught before I could elaborate, and she swiftly found a free page in the passport, stamped my arrival and waved me on.

I had packed light. A roller bag, only half full, and a shoulder messenger bag, overstuffed with paper, cables, electronics, and thing formerly called 'computers'. I don't know quite what to call them now. It's getting more difficult to discern what that means. The only certainty is that going through airport security I have to separate them from the messenger bag, making the number of trays to manage reminiscent of an Australian road train - one engine and five or six cargo trailers, sliding down the belt. But I've argued before when challenged to put all computers in trays and responding by only pulling out the laptop, leaving the rest. Lights flash, guards retrieve the offending bag and I'm admonished to pull out all computers. Of course I say I have, what's remaining isn't a computer really. It's an iPad, or my Kindle2, or the bluetooth keyboard, or the Airport Express. None of these are 'computers' per se. But if I protest too much I'll get the full search on the other side and that's not only a pain, but it's led to loss as after dissembling my entire electronics kit, satisfied there isn't hidden C4 or liquid fertilizer cleverly concealed in the components, I've failed to re-pack one thing or another only to recognize the loss later. That happened with a par of Bose noise canceling headphones. The thought shuts my lips tight and I grabbed another set of trays.

Not having to wait for luggage is a blessing. Despite the customs checks and I'm through much faster than those waiting for the ground crew to unload the bags of 300+ passengers ferried across the Pacific on this particular journey. Through the last check I'm given the "Welcome Home" greeting which is somehow bitter sweet as I don't really feel all that welcome, and home isn't the sense that I'm getting. Familiarity. Yes. Home... No.

Out of the arrival terminal in search of the next departure terminal, Bradley International to Terminal 2 (or so I thought), I follow the signs for Virgin America. I'm fleetingly distracted by the peculiar juxtaposition of words, but snap out of my brief reflection as the signage points unequivocally to the left. It's cool. Typical of moist, fog shrouded California mornings. I'm quickly in front of further way finding messages saying Virgin America is here, up the stairs. In a few seconds I'm ticketed and preparing to once again disassemble the gear from their places in these shoulder bag. Through the line the only diversion was my failure to remember the shoe bomber and the consequent removal of footware that is required uniquely of American air transportation. I curse the jerk that tried to light and eternalize his soles forcing the rest of us to be forever in socks through security checkpoints.

Through and into the new terminal I'm at once in familiar territory and yet strangely distant. I come across a BestBuy electronics vending machine - Sony headphones, cameras, USB sticks, Bose noise canceling headphones (is this is message?), and rows of iPods and related accessories. It's a massive case, requiring a credit card to dislodge any of its inhabitants. Looking around I notice the 'texting walk' all around me. People moving forward, heads down, punching at tiny keypads on various mobile devices, remarkably avoiding collisions as they thread their way to the food court. As I'm gazing I start to notice the majority of those upright are in distributed engagement. They may be walking together but they are not in the same place. Those seated are disproportionately tapping away at full size keyboards on laptops. Here where couples are seated together there is a surprisingly higher proportion of people talking to each other, spinning around their laptops to share what's on their screens. Many are solitary and getting in a bit more work in the few minutes they have to savor the fast food dollop that they've purchased as they examine an ruminate on quarterly sales or bond returns. Who knew what VisiCalc would reap?

The familiarity filtered through the separation is new. This must be what Margaret Mead described as the 'participant observer' experience encountered in anthropological research. Yet this is anything but intellectual. I feel it in my gut. I'm in but not of this place.

The other weekend we went to a Writer's Festival event that featured a panel of literary folks were there to reflect on our overindulgence with technology. One was a very articulate and witty woman author, Susan Maushart, who recently published The Winter of Our Disconnect (apologies no doubt to Dickens). Her adventure was into a land of banished network connections. No internet at home or in the car for six months. It was a deal she struck with her family as part of the research she was undertaking for a new book. As a professor of media and communications, this was both 'work' and a personal exploration of social dynamics among her and her offspring.

The others on the panel, writers all as befitting of the setting, were voicing various stages of technology concern, from the Nicholas Carr quasi scientific exhortations of caution bolstered by neuroscientific research, to a more adamant condemnation of the capitalist conspiracy to use the net as a vehicle for consumption and industrial hegemony. I thought Marxism had passed away but the panel reminded me that it's alive and well in circles I obviously don't inhabit. Susan's take was different, measured, thoughtful, and funny, but ultimately nuanced in a way the others weren't.

Where this comes together, if it does, is the recognition that we need the the stimulation and augmentation that digital technology affords to us as social organisms. At the same time, the sense of distance and the observations I shared above only come from having a chance to reflect and process what otherwise is just the air around us, or water to a fish. We need that time and space to integrate and process the rich information that is increasingly swirling around us, brought to us, projected on us.

The data is emerging that despite what we think about when we're in 'the flow' the reality of multitasking is that it does detract from some kinds of cognitive engagement. Despite what interviews of otherwise high performing students say, and they're experience generally reflects their self-perception is that they are performing 'better' in this multitasking world. Yet the data shows they aren't. A Stanford study last year published in PNAS is among the many coming out now who are demonstrating the costs of this hyperconnected multitasking environment.

"Using several standard psychological benchmark tests of focus, the study showed that college students who routinely juggle many flows of information, bouncing from e-mail to web text to video to chat to phone calls, fared significantly worse than their low-multitasking peers. "

(Commenting on a PNAS study Cognitive Control in Media Multitaskers -- InfoQ)

Being in the flow requires knowing when to seek out the eddies - gain the time and perspective that creative integration requires. My passage through the airport this time around is with new eyes. That perspective is gained by intense engagement juxtaposed with periods of being outside, in reflective contemplation. Who knows what will stand out the next time around? I'm not sure - but I'm looking forward to it.

--pdl--

- Posted using BlogPress from my iPad

Location:Los Angeles,United States