BICEP2 Redux: How the Sausage is Made

An ongoing problem with communicating science to the general public is the existence of widely-held misconceptions among the public regarding how science actually works. A case in point is the March 17 announcement by the BICEP2 Collaboration regarding the detection of B-Mode polarization in the Cosmic Microwave Background and the events which have unfolded since then.

All too often, news stories and blog posts will trumpet some announcement with sensational headlines like “Scientists Say Cheap, Efficient Solar Cells Just Around the Corner”, or “Scientists Close in on Cure for Cancer.” Many people take such announcements at face value and consider the case closed. The work has been done.  The reality of the situation, however, is that the initial announcement of a discovery or breakthrough is just the beginning of the hard work, breathlessly hyped headlines notwithstanding.

How Science Actually Works (or at least how it is supposed to work)

Once a researcher or a team of researchers do their initial work, they write up the results of their work, including any experimental details needed to reproduce the work, in a paper and submit it for publication in a scientific journal. It is at this point that the “peer  review” process kicks in. Peer review really is the “special sauce” of the scientific method, and one of the least generally understood aspects of science among the general public. It is one of the key self-correction mechanisms of science, a way to keep errors from creeping in.  When a journal receives a submission for publication, they pass along copies of the proposed article to subject matter experts in the specific area of research covered by the paper.  These are the peer reviewers. (Ostensibly, the reviewers are anonymous. However, in practice, the number of specialists in a given area of research tends to be so small that they all know each other, so it isn’t hard to tell who is doing what in terms of peer review.) The reviewers are then supposed to go over the proposed paper with a fine-toothed comb, looking for errors or inconsistencies.  They will either recommend that the paper not be published, recommend that it be published as-is, or offer recommendations for changes or corrections to make the paper suitable for publication.

So, once an article passes muster in the peer-review process, it gets published. In the case at hand, the actual paper describing the BICEP2 results announced in March has now been published in Physical Review Letters:

P. A. R. Ade et al. (BICEP2 Collaboration), “Detection of B-Mode Polarization at Degree Angular Scales by BICEP2“, Phys. Rev. Lett. 112, 241101 – Published 19 June 2014 DOI: http://dx.doi.org/10.1103/PhysRevLett.112.241101

BICEP2_snip

This is where the next step in the process kicks in: replication of results. It isn’t enough for a paper to say, “Hey, we’ve observed Whatsits decaying to a new particle, which we’ve dubbed Plaitons.”  Someone else has to try to reproduce the results (putting their own papers through the peer review process). Another team might give it a try and say, “Nah, we’re not seeing it.  Did you calibrate your framistasis properly?” Or, perhaps, “Yeah, we see something like that as well. Plaitons may be a real thing.” And just one or two follow-up papers like this don’t necessarily settle the issue. It can take years, or decades even, for the evidence to be strong enough or unambiguous enough for a consensus to build among the community of subject matter experts. (It took decades for particle physicists to accept the idea that the deficit of solar electron anti-neutrinos detected in the late sixties at the Homestake Gold Mine was due to neutrinos having a tiny amount of mass, allowing them to transform into different neutrino types. But that is a story for another day.) Again, like peer review, this is part and parcel of the self-correcting mechanism of the scientific method. It is all about keeping us from fooling ourselves.

For replication or refutation of the BICEP2 results, the scientific world eagerly awaits results from either the Planck Collaboration (which may come this fall), or from SPTpol (an instrument at the other end of the building that houses the BICEP2 instrument). It is only through validation from independent teams that the BICEP2 results can go from “Ooh, that is interesting” to physicists nodding sagely and saying “Yeah, we’ve got this” and placing serious wagers on Nobel Prizes.

I should note here that it is not essential that the replication phase come sequentially after publication of initial results. Sometimes, independent teams working in parallel come to the same conclusions at the same time.  This was the case with the Higgs boson discovery, in which the Atlas and CMS teams were working independently with separate detectors, not sharing data with one another until after the fact (in order to keep from contaminating each other’s work with errors or mistaken assumptions). Ditto for the discovery of the acceleration of cosmological expansion due to dark energy.

I should also point out that press conferences and press releases aren’t really part of this process.  They happen, but they aren’t part of the process.  When a researcher or team makes a big breakthrough, it isn’t uncommon for the associated institution to hold a press conference or issue a press release (with the latter often botching the story as badly as the press tends to mangle science stories, but that is a rant for another time). However, these things have little to do with advancing the process, and tend to be done for some combination of the following purposes:

  • Promoting the prestige of the institution or institutions under which the research is performed. (“Hey, see what cool stuff our faculty are doing. Send your kids and money to our university, and they can do cool stuff like this too.”) Sometimes, this is tied to announcing a result before a competing team announces similar results.
  • Public outreach.  Research institutions have a legitimate interest in communicating what they do to the public, even if the message gets mangled by the media.  (Sorry, I should still save that rant.)
  • Funding/politics. (“Hey, Congress, we are getting results. Please don’t cut our funding.”)  BICEP2, by the way, is funded by a grant from the National Science Foundation.
  • Rapid communication to the scientific community. Remember the big LHC meeting where the Higgs announcement was made on July 4, 2012?  It was pretty technical in content and aimed at scientific peers.  They had embargoed their results for about as long as they could and didn’t want to wait for the next big science conference or for their papers to get published, and wanted to get the actual results out before the rumor mill took over. However, due to the hype that had been built up around it in the press and the blogosphere, LOTS of lay-people watched the live-stream, even going so far as to endure a PowerPoint presentation using Comic Sans. (Really, Fabiola? Oh, well.)

Of course, sometimes science by press conference backfires, as in the case of the claimed discovery by Fleischmann and Pons of “cold fusion” back in 1988. That “breakthrough” turned out to be a bunch of hot air. Well, hot water, anyway, not to mention poor calorimetry. I recall eagerly following the ensuing debates on USENET, hoping that there was something to it, but, alas, it was not meant to be. Failure to replicate put a nail into that coffin.  The self-correcting mechanism of science asserted itself.  Evidence is the final arbitrator of reality, trumping any degree of wishful thinking we might hold.

Crowd-Sourcing Peer Review

While the scientific process generally follows the form outlined above, the BICEP2 case has included an added element that is somewhat novel. With the availability of a preprint on Cornell’s ArXiv preprint server since the initial announcement of the results, the preliminary peer review process has effectively extended well beyond the official reviewers. Analysis and criticism of the work has come in from numerous quarters, to such an extent that the final published form of the paper incorporates feedback beyond that provided by the official review process.  In this case, the peer review process was essentially crowd-sourced.

It isn’t as if such a phenomenon couldn’t have happened before. The ArXiv preprint service has been available in one form or another since the early nineties. However, a tipping point seems to have been reached in this instance, perhaps due to the hype and media attention which accompanied the original announcement. Researchers well beyond the circle of scientists who would typically be involved in the formal process have taken an interest, and they have dug in, looking for, and potentially finding, flaws in the original work. The possible flaws that have been identified (which I’ll go into shortly) have been insufficient to invalidate the conclusions, but they do weaken the confidence in the results somewhat, rendering the results of replication efforts something even more eagerly awaited. The important point, though, is that more eyeballs than ever have been scrutinizing the results, something which can only bode well for the process.

Piling On

It did not take long after the initial announcement for critiques to start appearing.  One of the earliest was regarding the possibility that some of the observed B-mode polarization observed might be a foreground effect due to something called galactic radio loops , a phenomenon caused by magnetized dust interacting with our own galaxy’s magnetic field.

Then on May 12, physicist and blogger Adam Falkowski (a.k.a. “Jester”) revealed on his Résonaances blog a rumor of an even bigger potential problem with the BICEP2 analysis.  One of the biggest challenges for the BICEP2 team has been to filter out B-mode polarization effects due to foreground dust. One component of the procedure that was employed was to use data on foreground contributions collected by the Planck team. However, Planck hasn’t yet released the raw data for this, and the BICEP2 team had to effectively “scrape” the data from a graph on page 6 of this presentation PDF. (This is not an ideal scenario, and efforts are reportedly underway to make the raw Planck data available to the BICEP2 team to improve their analysis.)

planck_pol

The rumor revealed by Jester is that the BICEP2 team had misinterpreted the content of this graph, thinking that it represents polarization contribution only from dust, when in fact it represents ALL foreground contributions. (Note where the slide says “Not CIB subtracted.”) If this is the case, some or all of the effect reported might not be present.

The net result of these (and other) criticisms can best be summed up by the following lines added to the abstract of the final paper:

However, these models are not sufficiently constrained to exclude the possibility of dust emission bright enough to explain the entire excess signal…  Accounting for the contribution of foreground dust will shift this value downward by an amount which will be better constrained with upcoming data sets.

Ultimately, more data and observations by independent teams will be needed to settle this once and for all. And if the critiques sometimes seem harsh, don’t worry about it.  That is how the process works. Science is to some degree an adversarial process, although the slings and arrows aren’t really directed at the researchers themselves, but their results. And it is the results that survive such harsh scrutiny that move us forward.

As an aside, I note that this entire kerfuffle has been transpiring amidst the 50th anniversary of the initial discovery of the Cosmic Microwave Background.

For some useful commentary on these latest results and where we go from here, have a look at these articles:

 

 

 

Advertisements

About Glen Mark Martin

MCSE-Messaging. Exchange Administrator at the University of Texas at Austin. Unrepentant armchair physicist.
This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

One Response to BICEP2 Redux: How the Sausage is Made

  1. telescoper says:

    Reblogged this on In the Dark and commented:
    I came across this (rather lengthy) discussion of the BICEP2 story so far so thought I would share it here. There’s a particularly useful collection of articles at the end for those who would like to read more.

    I’ll also take this opportunity to refer you to a recent BBC News story which states that the BICEP2 and Planck teams are now in discussions about sharing data. About time, if you ask me. Still, it will take a considerable time to work out the ordering of the authors if they ever do write a paper!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s