[We tend to take medals and ratings for granted without considering the context of the evaluation. This article was written by Dan Berger and appeared in the California Grapevine. Dan may take his time getting to his point, but the journey is well worth the effort. I have taken extensive editing liberties for brevity and emphasis.]

He arrived in the Napa Valley one spring day to taste samples of newly or soon-to-be released Chardonnays. The reviewer arrived at 9 a.m. and was taken to the room where 310 Chardonnays were opened and ready to be sampled. The organizer of the event left the reviewer to try the wines, expecting it would take three or more hours. The reviewer appeared at the organizer’s office at 10:40 a.m., thanked him, and left. Do the math, and the reviewer spent only 19 seconds tasting and recording each wine. For perspective, a four-person panel asked to judge 252 Chardonnays at a competition some years ago took seven hours (100 seconds per wine).

I have a shirt that is half green and half blue. I use it to illustrate how we see wines in different ways. I will wear the shirt with only half of it uncovered and ask students in my wine class to describe the color. “Blue,” they will say. We then evaluate some wines that I tell them are from an undistinguished wine district. I later admit that the wines had in fact come from a far more prestigious location and ask them to reevaluate the wines which are often seen as far better. I then uncover the other half of my shirt and ask what color it is. Most get the message: seeing the whole picture is a necessary part of making a successful judgment about something.

Such assumptions (like assuming a wine is one thing only to see it later in a different light) can be truly mystifying. One is certain a thing exists only to realize later that the original “reality” was merely an impression and that the truth lay elsewhere. It is like those optical illusions that amused us as children, like the drawing of various lines and the question, “Which one is longer?” or drawings by Maurits Escher, notably in his well-known waterfall building sketch.

With wine, the simplest of these misreadings relates to the glassware used for evaluation. A classic Pinot Noir or Barolo served in a tiny glass cannot show its depth, character, or potential. But our perceptions are also affected by other factors such as our expectations based on our knowledge of what’s in the glass in front of us. Preconception is one of the best (or worst) elements in wine evaluation since it allows us to compare this wine with others of its type.

Preconception is one of the best (or worst) elements in wine evaluation since it allows us to compare a wine with others of its type. If it’s a Napa Valley Cabernet Sauvignon, does it show the regional character that we have come to expect? Does it show Cabernet character? Most importantly, does it show how that grape variety typically grows in that region? And how does this wine compare to prior wines from this same winery? It is not only a matter of how we like or dislike a wine, but also a function of our effort to objectively determine whether a wine conforms to what it is supposed to be. Some people love delicious Cabernets that have no particular Cabernet character, no regional characteristics worth discussing, and otherwise fail the objectivity tests, but which pass the subjectivity test (“I like it”) with flying colors.

I was thusly concerned recently when a winery owner took me to task for the results of a wine competition I coordinate, arguing that the event was flawed because the judges did not give his wines marks as high as they get when judged by some of the esteemed individuals who rank wines by numbers. His complaint was a natural outgrowth of what he saw as an incongruity in the wine evaluation process. How, he wondered, could his Syrah, judged to be worth 90+ points by one powerful wine critic, only receive a bronze medal from a wine competition panel? This man takes pride in his wines and believes they are great. Surely the competition panels that routinely give his wines bronze medals (and, on occasion, not even that!) must be wrong.

What this winery owner does not understand is the multifaceted nature of wine evaluation which is radically different from the sort of criticism one finds for art, music, or drama. A classic case in point: if a drama critic dislikes the introspective and occasionally depressingly realistic dramas of Lillian Hellman and then is asked to review one of that dramatists’ works, chances are the acting, staging, lighting, etc. will color the critic’s final view of the play’s performance. Likewise, it would probably be foolhardy to ask one of today’s best rap reviewers to critique a performance of works by Buxtehude or Pergolessi. Sure, it’s all music, but unless trained to understand the latter composers, a rap reviewer would be lost in Gregorian chant land.

So I would ask the winery owner, what evidence is there that the single reviewer of his wines is right and the judging panel is wrong? Indeed, simply posing the question brings up another: If the single reviewer always gives his wine similar high scores, and the panels (which change from judging to judging) always judge them lower, is there not a vital and valuable lesson here? Perhaps another style of wine would get a high score with both the number scorer as well as the wine competition. I can name a number of wineries that do well in both.

One such lesson could well be this interpretation: the single reviewer, who knows a lot about the wine before pulling the cork, likes this particular style, which is big, chewy, high in alcohol (in the 14.5% range), and has soft acidity with pH levels in the 3.8 range. And that various panels of wine judges (acting without sight of the label, but with discussion among them) find that style of wine overbearing, ungainly, and without the finesse or acidity to work with food.

But, one could counter, the single reviewer is consistent in giving this wine high scores, and the single reviewer can, if he chooses, look back to what he gave this wine in the past and give a similar score to the newly released wine, even if one such wine is noticeably of lesser quality.  

When evaluating wine, the ability to look back at scores on prior vintages is a nice insurance policy against looking slipshod - but it can backfire. This can occur for the single reviewer, but not when the evaluations are made by members of a judging panel who know nothing about the wine before evaluating it. Indeed, many competitions do award high medals to big, chewy wines, so if your wines did not fare well at one or two competitions, must you assume that the judges are flawed? In the cases you cited, the judges are some of the world?s finest, each with a long history of wine evaluation. You would argue that so are the credentials of the evaluators for the two publications you cite as being ?better? at understanding your wines.

The single reviewers do have the advantage of knowing what to expect when tasting your wines; moreover, your wines seem to be a product of the “bigger is better,” terroir-is-unimportant system that was attacked by Jonathan Nossiter in his film Mondovino. The panels judging your wines are not filled with preconceptions. They are made up of people who are intolerant of flabby wines made to impress the taster who makes a quick “evaluation” and then moves on to the next blockbuster. I know for a fact that most reviewers for the “rate-by-the-numbers” publications do not spend as much time making evaluations as do most wine competition judges or as much time as I do. The top reviewers are so fast in evaluation that they spend no more than a few seconds with most wines before making a numerical evaluation.

Not only are panels of judges composed of more than one sort of palate, but they are also made up of people who come from different areas of wine perception: merchants who see things they can offer retail buyers; wine makers who know infinitely more about wine making (and such other issues vital to proper wine evaluation as pH, acid, technical flaws, oak, tannin, and other topics) than do the single reviewers. There are also savvy, informed consumers (those who actually buy wine and? make their decisions carefully), restaurateurs, wholesalers, and importers.

In short, these are people who do not look at a portion of one shirt and proclaim that the entire shirt is blue; they look at issues from more than one angle. Moreover, they know NOTHING about what’s in the glasses in front of them and are free to be bored to tears with overbearing, richly endowed, but rather simplistic flavors that are designed for the fast hit as opposed to long-savoring and food-matching.

Randall Grahm of Bonny Doon Vineyards, the sage of vinedom, said it best in a Wines and Vines editorial. Referring to such oafish wines, Grahm said, “I am bothered by the rampant, erroneous belief that unnatural concentration, triple C-cup overripeness, lashings of new oak, in sum, fruit bombasity, aka the International Wine Style, is now considered to be the only viable wine style. Viable because it is commercial, commercial because it panders to the simplest and least- tutored palates.... I am saddened that the most influential critics, who truly believe that they are shining the bright light of enlightenment on blighted wine-making...are bringing out the very worst in winemakers, in perfect synchrony with our self-absorbed culture...”

He refers to and describes some wines that often get high scores as “the sort of generic, white bread, airbrushed vinonymity, wines that could have come from literally anywhere, ones that are all surface and no depth, a triumph of style over substance, wine as an invention or confection, sort of like a vinous, vacuous Britney Spears...”

The fact that these people largely review European or Australian wine should not have any bearing on this issue. Take my word for it, I have spoken at length with most of them, and we have shared wines and meals. The overwhelming consensus is that the number mongers are right only for those people who have untutored palates: those for whom the big, fast hit is the only worthwhile thing about wine; those for whom subtlety is a vice, not a virtue.     

I firmly believe that the results of judgings far better reflect the state of the art for the vast majority of consumers, not only those prepared to spend $50 for a bottle of an oafish, port-like confection with enough tannin to remove pine tar from a baseball bat.

In my aforementioned article entitled “Mondovino and Simplism,” I took the position that balance was far more important than weight in almost all wines. That drew a comment from a Sonoma County wine maker who makes a strong effort to make balanced wines. In his tasting room, said this wine maker, “we preach to the ‘unwashed’ about the benefits of table wine, aka, 10%- 14% alcohol. Some get it, some don’t. California wines, and to some extent other regions, have gone beyond a beverage of moderation with distinctive appellation characteristics.

“I was served a fancy [Napa Valley] red wine last night, which I had trouble defining. I guessed Syrah. (Turned out it was Syrah and Merlot.) The point is: I just took a wild guess based on process of elimination because there was little, if any, varietal character. [I knew it was] picked somewhere in the 27-28 brix range since the alcohol was 15%+! I believe one should apologize for that kind of wine, not brag about it.”

Those who have chosen the paths of later picking and new-oak flavorings with low acids are aiming their wines at the number pushers. Wines of balance and structure are more appreciated by wine judges who discuss wine style in a wine-by-wine manner as they evaluate.

A final benefit of judging panels is that they are their own best check-and-balance system. No judge wants to be reproved by his or her compatriots if he or she should choose a style of wine that has little or no viability with the other judges. One phrase I used to hear at wine competitions, but which I no longer hear, is a wine merchant pleading a case with a phrase, “Sure, it’s massive, but I can sell it. Gold medal.” Such people are terrible judges and ought not to be judging wine. And few quality judgings allow such thinking any more. Of course the operative word here is “quality,” and the winery owner who questioned the capability of the judges at one of my events has not spent any time sitting in on a judging panel to see what’s involved. I invite any winery owner or wine maker to see the process up close. The result could be enlightenment of a type that could lead to a more moderate approach in winemaking styles.