BICEP2 Redux: How the Sausage is Made

An ongoing problem with communicating science to the general public is the existence of widely-held misconceptions among the public regarding how science actually works. A case in point is the March 17 announcement by the BICEP2 Collaboration regarding the detection of B-Mode polarization in the Cosmic Microwave Background and the events which have unfolded since then.

All too often, news stories and blog posts will trumpet some announcement with sensational headlines like “Scientists Say Cheap, Efficient Solar Cells Just Around the Corner”, or “Scientists Close in on Cure for Cancer.” Many people take such announcements at face value and consider the case closed. The work has been done.  The reality of the situation, however, is that the initial announcement of a discovery or breakthrough is just the beginning of the hard work, breathlessly hyped headlines notwithstanding.

How Science Actually Works (or at least how it is supposed to work)

Once a researcher or a team of researchers do their initial work, they write up the results of their work, including any experimental details needed to reproduce the work, in a paper and submit it for publication in a scientific journal. It is at this point that the “peer  review” process kicks in. Peer review really is the “special sauce” of the scientific method, and one of the least generally understood aspects of science among the general public. It is one of the key self-correction mechanisms of science, a way to keep errors from creeping in.  When a journal receives a submission for publication, they pass along copies of the proposed article to subject matter experts in the specific area of research covered by the paper.  These are the peer reviewers. (Ostensibly, the reviewers are anonymous. However, in practice, the number of specialists in a given area of research tends to be so small that they all know each other, so it isn’t hard to tell who is doing what in terms of peer review.) The reviewers are then supposed to go over the proposed paper with a fine-toothed comb, looking for errors or inconsistencies.  They will either recommend that the paper not be published, recommend that it be published as-is, or offer recommendations for changes or corrections to make the paper suitable for publication.

So, once an article passes muster in the peer-review process, it gets published. In the case at hand, the actual paper describing the BICEP2 results announced in March has now been published in Physical Review Letters:

P. A. R. Ade et al. (BICEP2 Collaboration), “Detection of B-Mode Polarization at Degree Angular Scales by BICEP2“, Phys. Rev. Lett. 112, 241101 – Published 19 June 2014 DOI: http://dx.doi.org/10.1103/PhysRevLett.112.241101

This is where the next step in the process kicks in: replication of results. It isn’t enough for a paper to say, “Hey, we’ve observed Whatsits decaying to a new particle, which we’ve dubbed Plaitons.”  Someone else has to try to reproduce the results (putting their own papers through the peer review process). Another team might give it a try and say, “Nah, we’re not seeing it.  Did you calibrate your framistasis properly?” Or, perhaps, “Yeah, we see something like that as well. Plaitons may be a real thing.” And just one or two follow-up papers like this don’t necessarily settle the issue. It can take years, or decades even, for the evidence to be strong enough or unambiguous enough for a consensus to build among the community of subject matter experts. (It took decades for particle physicists to accept the idea that the deficit of solar electron anti-neutrinos detected in the late sixties at the Homestake Gold Mine was due to neutrinos having a tiny amount of mass, allowing them to transform into different neutrino types. But that is a story for another day.) Again, like peer review, this is part and parcel of the self-correcting mechanism of the scientific method. It is all about keeping us from fooling ourselves.

For replication or refutation of the BICEP2 results, the scientific world eagerly awaits results from either the Planck Collaboration (which may come this fall), or from SPTpol (an instrument at the other end of the building that houses the BICEP2 instrument). It is only through validation from independent teams that the BICEP2 results can go from “Ooh, that is interesting” to physicists nodding sagely and saying “Yeah, we’ve got this” and placing serious wagers on Nobel Prizes.

I should note here that it is not essential that the replication phase come sequentially after publication of initial results. Sometimes, independent teams working in parallel come to the same conclusions at the same time.  This was the case with the Higgs boson discovery, in which the Atlas and CMS teams were working independently with separate detectors, not sharing data with one another until after the fact (in order to keep from contaminating each other’s work with errors or mistaken assumptions). Ditto for the discovery of the acceleration of cosmological expansion due to dark energy.

I should also point out that press conferences and press releases aren’t really part of this process.  They happen, but they aren’t part of the process.  When a researcher or team makes a big breakthrough, it isn’t uncommon for the associated institution to hold a press conference or issue a press release (with the latter often botching the story as badly as the press tends to mangle science stories, but that is a rant for another time). However, these things have little to do with advancing the process, and tend to be done for some combination of the following purposes:

• Promoting the prestige of the institution or institutions under which the research is performed. (“Hey, see what cool stuff our faculty are doing. Send your kids and money to our university, and they can do cool stuff like this too.”) Sometimes, this is tied to announcing a result before a competing team announces similar results.
• Public outreach.  Research institutions have a legitimate interest in communicating what they do to the public, even if the message gets mangled by the media.  (Sorry, I should still save that rant.)
• Funding/politics. (“Hey, Congress, we are getting results. Please don’t cut our funding.”)  BICEP2, by the way, is funded by a grant from the National Science Foundation.
• Rapid communication to the scientific community. Remember the big LHC meeting where the Higgs announcement was made on July 4, 2012?  It was pretty technical in content and aimed at scientific peers.  They had embargoed their results for about as long as they could and didn’t want to wait for the next big science conference or for their papers to get published, and wanted to get the actual results out before the rumor mill took over. However, due to the hype that had been built up around it in the press and the blogosphere, LOTS of lay-people watched the live-stream, even going so far as to endure a PowerPoint presentation using Comic Sans. (Really, Fabiola? Oh, well.)

Of course, sometimes science by press conference backfires, as in the case of the claimed discovery by Fleischmann and Pons of “cold fusion” back in 1988. That “breakthrough” turned out to be a bunch of hot air. Well, hot water, anyway, not to mention poor calorimetry. I recall eagerly following the ensuing debates on USENET, hoping that there was something to it, but, alas, it was not meant to be. Failure to replicate put a nail into that coffin.  The self-correcting mechanism of science asserted itself.  Evidence is the final arbitrator of reality, trumping any degree of wishful thinking we might hold.

Crowd-Sourcing Peer Review

While the scientific process generally follows the form outlined above, the BICEP2 case has included an added element that is somewhat novel. With the availability of a preprint on Cornell’s ArXiv preprint server since the initial announcement of the results, the preliminary peer review process has effectively extended well beyond the official reviewers. Analysis and criticism of the work has come in from numerous quarters, to such an extent that the final published form of the paper incorporates feedback beyond that provided by the official review process.  In this case, the peer review process was essentially crowd-sourced.

It isn’t as if such a phenomenon couldn’t have happened before. The ArXiv preprint service has been available in one form or another since the early nineties. However, a tipping point seems to have been reached in this instance, perhaps due to the hype and media attention which accompanied the original announcement. Researchers well beyond the circle of scientists who would typically be involved in the formal process have taken an interest, and they have dug in, looking for, and potentially finding, flaws in the original work. The possible flaws that have been identified (which I’ll go into shortly) have been insufficient to invalidate the conclusions, but they do weaken the confidence in the results somewhat, rendering the results of replication efforts something even more eagerly awaited. The important point, though, is that more eyeballs than ever have been scrutinizing the results, something which can only bode well for the process.

Piling On

It did not take long after the initial announcement for critiques to start appearing.  One of the earliest was regarding the possibility that some of the observed B-mode polarization observed might be a foreground effect due to something called galactic radio loops , a phenomenon caused by magnetized dust interacting with our own galaxy’s magnetic field.

Then on May 12, physicist and blogger Adam Falkowski (a.k.a. “Jester”) revealed on his Résonaances blog a rumor of an even bigger potential problem with the BICEP2 analysis.  One of the biggest challenges for the BICEP2 team has been to filter out B-mode polarization effects due to foreground dust. One component of the procedure that was employed was to use data on foreground contributions collected by the Planck team. However, Planck hasn’t yet released the raw data for this, and the BICEP2 team had to effectively “scrape” the data from a graph on page 6 of this presentation PDF. (This is not an ideal scenario, and efforts are reportedly underway to make the raw Planck data available to the BICEP2 team to improve their analysis.)

The rumor revealed by Jester is that the BICEP2 team had misinterpreted the content of this graph, thinking that it represents polarization contribution only from dust, when in fact it represents ALL foreground contributions. (Note where the slide says “Not CIB subtracted.”) If this is the case, some or all of the effect reported might not be present.

The net result of these (and other) criticisms can best be summed up by the following lines added to the abstract of the final paper:

However, these models are not sufficiently constrained to exclude the possibility of dust emission bright enough to explain the entire excess signal…  Accounting for the contribution of foreground dust will shift this value downward by an amount which will be better constrained with upcoming data sets.

Ultimately, more data and observations by independent teams will be needed to settle this once and for all. And if the critiques sometimes seem harsh, don’t worry about it.  That is how the process works. Science is to some degree an adversarial process, although the slings and arrows aren’t really directed at the researchers themselves, but their results. And it is the results that survive such harsh scrutiny that move us forward.

As an aside, I note that this entire kerfuffle has been transpiring amidst the 50th anniversary of the initial discovery of the Cosmic Microwave Background.

For some useful commentary on these latest results and where we go from here, have a look at these articles:

• “Notes from the Editors: BICEP2 Images the Past, Cosmology Looks to the Future“, Physics 7, 66 (2014) DOI: 10.1103/Physics.7.66
• Lawrence M. Krauss, “Viewpoint: Peering Back to the Beginning of Time“, Physics 7, 64 (2014)  DOI: 10.1103/Physics.7.64
• “Focus: Theorists Weigh in on BICEP2“, Physics 7, 65 (2014) DOI: 10.1103/Physics.7.65
• Perimeter Institute presentations on BICEP2.
• Elizabeth Gibney, “Physicists defend Big Bang wave announcement“, Nature News Blog, July 9, 2014.
• Chad Orzel, “What Scientists Should Learn from Economists“, Uncertain Principles, July 7, 2014.
• Peter Byrne, “Early-Universe Explorer Looks for Answers“, Quanta Magazine, July 3, 2014.
• Peter Byrne, “A Bold Critic of the Big Bang’s ‘Smoking Gun’“, Quanta Magazine, July 3, 2014.
• Jonathan Amos, “Cosmic inflation: BICEP2 and Planck to share data“, BBC News, July 3, 2014.
• Marcelo Gleiser, “Cosmic Confusion: It’s How Science Gets Done“, NPR, June 25, 2014.
• Steven Novella, “Inflation Evidence Questioned“, Neurologica Blog, June 20, 2014.
• Matt Strassler, “BICEP2’s Cosmic Polarization: Published, Reduced in Strength“, Of Particular Significance, June 20, 2014.
• Ron Cowen, “Gravitational-wave team admits findings could amount to dust“, Nature, June 20, 2014.
• Peter Coles, “Published BICEP2 paper admits ‘Unquantifiable Uncertainty’…“, In The Dark, June 20, 2014.
• “BICEP2 researchers publish nuanced account of stunning patters in the microwave sky“, Nanowerk News, June 19, 2014.
• Peter Woit, “Smoking Gun No Longer Smoking“, Not Even Wrong, June 19, 2014.
• Dennis Overbye, “Astronomers Hedge on Big Bang Detection Claim“, The New York Times, June 19, 2014.
• Peter Coles, “Has BICEP2 bitten the dust?“, In The Dark, June 5, 2014.
• Paul Steinhardt, “Big Bang blunder bursts the multiverse bubble“, Nature News, June 3, 2014.
• Matt Strassler, “The BICEP2 Dust-Up Continues“, Of Particular Significance, May 30, 2014.
• Ron Cowen, “No evidence for or against gravitational waves“, Nature News, May 29, 2014.
• Peter Coles, “BICEP2: The Dust Thickens…“, In The Dark, May 29, 2014.
• Adam Falkowski, “Weekend plot: BICEP limits on tensor modes“, Résonaances, May 25, 2014.
• Michael J. Mortonson, Uroˇs Seljak, “A joint analysis of Planck and BICEP2 B modes including dust polarization uncertainty“, May 22, 2014. arXiv:1405.5857 [astro-ph.CO]
• Michael D. Lemonick, “Backlash to Big Bang Discovery Gathers Steam“, Scientific American, May 21, 2014.
• Sabine Hossenfelder, “What is direct evidence and does the BICEP2 measurement prove that gravity must be quantized?“, BackRe(Action), May 21, 2014.
• Richard, Easther, “BICEP2: Two Months Later (and the Morning After)“, Excursionset.com, May 19, 2014.
• Matt Strassler, “Will BICEP2 Lose Some of Its Muscle?“, Of Particular Significance, May 19, 2014.
• Brian Koberlein, “Are the BICEP2 Results Invalid? Probably Not.“, Universe Today, May 19, 2014.
• Seth Zenz, “In Science, it’s OK to be Wrong“, Quantum Diaries, May 19, 2014.
• Amy Ellis Nutt, “50 years later: celebrating the cosmic discovery that earned two New Jersey physicists the Nobel Prize“, NJ.com, May 18, 2014.
• Ron Cowen, “Gravitational wave discovery faces scrutiny“, Nature News, May 16, 2014.
• Adam Falkowski, “Follow up on BICEP“,  Résonaances, May 16, 2014.
• Joanne Baker, “Cosmology: First light“, Nature, May 14, 2014.
• Dan Vergano, “Big Bang Discovery Comes Under Fire“, National Geographic Daily News, May 14, 2014.
• Ethan Siegal, “Don’t believe in cosmic inflation? You’re not alone, but you’re probably wrong.” Starts With A Bang, May 14, 2014.
• Peter Coles, “That BICEP Rumour…“, In The Dark, May 14, 2014.
• Sesh Nadathur, “New BICEP rumours: nothing to see here“, Blank On The Map, May 14, 2014.
• Sean Carroll, “Arrrgh Rumors“, Preposterous Universe, May 14, 2014.
• Peter Woit, “BICEP2 News“, Not Even Wrong, May 13, 2014.
• Lisa Grossman, “Rumours swirl over credibility of big bang ripple find“, New Scientist, May 13, 2014.
• Adam Falkowski, “Is BICEP wrong?“, Résonaances, May 12, 2014.
• Kathryn Jepsen, “Planck reveals galactic fingerprint“, Symmetry Magazine, May 12, 2014.
• Peter Coles, “Planck versus BICEP2: Round One!“, In The Dark, May 6, 2014.
• Matt Strassler, “Did BICEP2 Detect Gravitational Waves Directly or Indirectly?“, Of Particular Significance, April 30, 2014.
• “A ‘crack in the cosmic egg’“, Symmetry Magazine, April 23, 2014.
• Richard Easther, “BICEP2: A Month Later“, Excursionset.com, April 17, 2014.
• Brian Koberlein, “Loop de Loop“, One Universe at a Time“, April 16, 2014.
• Maggie McKee, “Star dust casts doubt on recent big bang wave result“, New Scientist, April 15, 2014.
• Tushna Commissariat, “Have galactic ‘radio loops’ been mistaken for B-mode polarization?“, Physics World, April 10, 2014.
• Peter Coles, “Galactic Loops as Sources of Polarized Emission“, In The Dark, April 8, 2014.
• Hao Liu, Philipp Mertsch, and Subir Sarkar, “Fingerprints of Galactic Loop I on the Cosmic Microwave Background“, Astrophys. J. 789 (2014) L29, DOI: 10.1088/2041-8205/789/2/L29
• George Musser, “Gravitational Waves Reveal the Universe before the Big Bang: An Interview with Physicist Gabriele Veneziano“, Scientific American Blogs, April 3, 2014.
• Clara Moskowitz, “Multiverse Controversy Heats Up over Gravitational Waves“, Scientific American, March 31, 2014.
• “Cosmologists Say Last Week’s Announcement About Gravitational Waves and Inflation May Be Wrong“, The Physics ArXiv Blog, March 24, 2014.
• John Preskill, “Inflation on the back of an envelope“, Quantum Frontiers, March 23, 2014.
• Ron Cowen, “Gravitational-wave finding causes ‘spring cleaning’ in physics“, Nature, March 21, 2014.
• Tabitha M. Powledge, “INFLATION and the origin of the Universe(s)“, PLOS Blogs, March 21, 2014.
• James B. Dent, Lawrence M. Krauss, Harsh Mathur, “Killing the Straw Man: Does BICEP Prove Inflation at the GUT Scale?“, March 20, 2014. arXiv:1403.5166 [astro-ph.CO]
• Matt Strassler, “If It Holds Up, What Might BICEP2’s Discovery Mean?“, Of Particular Significance, March 18, 2014.
• Max Tegmark, “BICEP2 Makes Waves in Cosmology: Now What?“, Scientific American Blogs, March 18, 2014.
• Matthew Francis, “Detection of primordial gravitational waves announced (Updated)“, Ars Technica, March 17, 2014.
• Dan Vergano, “Big Bang’s ‘Smoking Gun’ Confirms Early Universe’s Exponential Growth“, National Geographic, March 17, 2014.