Double-blind Experiments for Dummies: Wine - Part 1

This is part 1 of my attempt to carry out a double-blind wine-tasting experiment.

I was studying a case involving Penfolds Wines at uni the other week, and it made me want to get some on the way home. But I thought it would be good to buy a really good bottle, rather than relatively cheap wine, like I'd normally have. Then I thought about what a waste of money that might be, as there would be no guarantee that the more expensive wine would be any better. That made me want to find out, as scientifically as I could, whether the more expensive wine would be any better. So here's what I did:

Introduction

I bought 3 bottles of wine:

1. Penfolds Rawson's Retreat Cabernet Sauvignon 2010 - About $7
2. Penfolds Koonunga Hill Cabernet Sauvignon 2009 - About $12
3. Penfolds Bin 407 Cabernet Sauvignon 2008 - About $52

I know that the Penfolds range goes far higher than $52, but my budget doesn't, and I figured it was a high enough spread over which a clear differentiation of quality should be apparent.

I wanted this experiment to be a double-blind experiment, as close as I could get it, so that meant that not only could the people to whom I was giving the wine not know which wines they were drinking at any time (despite knowing which wines were involved in the experiment), but I couldn't know which wines I was giving them. For this reason, I could also be included in the study, as I was participating in the test on the same footing.

The participants in the experiment were:

1. Me: I know nothing formally about wine
2. My wife: She knows nothing formally about wine
3. Friend B: He has done a wine-tasting course and can speak fairly technically about wine
4. Friend K: She has done the same wine-tasting course 

Issue #1: My wife and I had recently shared a bottle of Koonunga Hill, so we could potentially be able to identify it easier.

Experimental Setup

When I was setting up for the experiment, one of the first things I noticed was that the bottles were all different heights. That wouldn't do, as we would have been able to tell which was which via observation. I found two other wine bottles that were the same height and shape as the Koonunga Hill bottle, washed them out and used a funnel to pour the Rawson's Retreat and Bin 407 into them.

Issue #2: The act of pouring, not pouring or decanting the wine could change its qualities

Issue #3: Even washing out the bottles and getting almost all water out of them could still change the qualities

Once the wine was in the bottles from which it would be poured, I marked which wine was in which bottle and attached new, clean shot measures to the bottles. I then wrapped most of each bottle in foil so that the bottles were obscured to the casual observer. While I wasn't looking, I got Friend K to do the same, in case I recognised how I'd wrapped the foil at the top. With Friend K not knowing which wine was in each bottle when doing her wrapping, neither of us should have been able to identify any wine after we'd both wrapped the bottles.

Issue #4: Friend K could have seen, from the bottom end of the bottle, which ones she was wrapping. I don't believe this happened.

Issue #5: Any of us could have identified the foil on the bottles between rounds of testing, to at least link each bottle in one tasting round to the same bottle later on. I don't believe this happened either, as I was the only person near the bottles, and I was trying not to look closely at them, and after a bit of wine, my memory wouldn't have been great with regard to small differences between scrunched up foil.

I then got 3 sets of 3 brown paper bags. On each set, I wrote, in a light yellow, fluoro marker, A, B or C. With the letters face down, I shuffled them out of site of Friend B, then I closed my eyes while Friend B shuffled them, still face down. I then lay them out face down on a bench. I repeated this for the other 2 sets of paper bags, so that by the end, there were 3 piles of 3 bags each, representing 3 randomised rounds for the experiment.

Leaving the bags face down, I put the top bag of each pile into the next bag and then put those two bags into the bottom bag. I then put each set of bags into a final, unmarked bag. I got someone (can't remember who) to shuffle the bottles and then, without looking at the bottles, I put each bottle into a bag-set.

If you can think of any factors that I haven't mentioned that reduce the credibility of the experiment so far, or think that the issues already mentioned somehow invalidate any results that it produces, let me know in the comments.

More to come later today or tomorrow regarding the experimental procedure.