What happened at USUDC 2018

After two errors in the team tab were reported to me, I spent a weekend reviewing all 445 scoresheets from the 2018 United States Universities Debating Championships. I discovered six further errors, making eight in total. Four of these errors affected the team tab, including two affecting the order of breaking teams. Of the other four, two affected the speaker tab, and two only affected speaker positions.

A quick primer on procedure: Every scoresheet must be seen by two different people before being confirmed in the tab system. The first person types in the scoresheet; the second person checks it. Tabbycat enforces this process — the confirmer cannot edit the data (only confirm or reject) and must be on a different account to the enterer. Tab assistants are also instructed to reject any scoresheet with any ambiguity or inconsistency.

The details of data entry errors can get a little dry, so if you start getting bored, skip down to the “Thoughts on data entry procedures” section, where I add a few more thoughts to what’s over at this accompanying post on the quiz that these errors inspired.

Details of data entry errors

Errors affecting breaking teams

Round 2, Stanford EL vs Seattle GS vs Rochester LO vs Cornell KW

The 79 for the opposition whip of Cornell KW was entered as a 74. This didn’t affect their rank in the debate, but it did deprive them of five speaker points. In this case, ballot math wouldn’t have helped resolve the ambiguity, because the 159 also looks like a 154. Nonetheless, tab assistants should have rejected the ballot as ambiguous and asked the chair to clarify.

Round 7, Penn AW vs Harvard AD vs Morehouse BP vs Harvard BH

The 76 for the prime minister of Penn AW was entered as a 75. Although the handwriting was ambiguous, tab assistants should have picked up on the inconsistency in the total, since 75 + 75 ≠ 151. This didn’t affect their rank in the debate.

These two errors affect teams in the same part of the break, so it’s easiest just to compare the original order, and the order with both corrections:

Errors affecting non-breaking teams

Round 1, Alaska KM vs Middlebury BW vs Harvard BH vs La Verne CD

The 75 for the opposition whip of La Verne CD was entered as a 79. Although the handwriting was ambiguous, tab assistants should have picked up on the inconsistency in the total, since 76 + 79 ≠ 151. This didn’t affect their rank in the debate. As a result of this error, La Verne CD originally appeared as 180th in the team tab; they should have been 181st.

Round 5, Pacific Lutheran AG vs Clemson MS vs Wheaton KV vs Wheaton KP

The 79 for the prime minister of Pacific Lutheran AG was entered as a 72. Although the handwriting was ambiguous, tab assistants should have picked up on the inconsistency in the total, because 72 + 80 is neither 159, 151 nor 157 (all three possible readings of that total); in fact, they should have rejected the ballot as ambiguous before checking for consistency. As a result of this error, Pacific Lutheran AG should have come second in the debate rather than fourth; tab assistants should also have noticed this inconsistency between the scoresheet and the tab system’s calculations. Because there were three problems that two different people failed to notice, this error is particularly disappointing.

Assuming all their other results would have held, this means Pacific Lutheran AG should have been 114th in the team tab, not the 163rd where they were originally reported. However, the draws for rounds 6 through 8 would have had Pacific Lutheran AG in lower rooms than they should have been, giving them a theoretically easier draw. It’s difficult to predict how they would have fared against teams in their rightful brackets.

Errors not affecting team rankings

In Round 3, George Fox CT vs Regis AJ vs Northeastern PS vs CUNY Brooklyn MM, the names of the prime minister and deputy prime minister for George Fox CT were entered the wrong way round. Because their scores differed by two, this affected speaker rankings.

In Round 7, La Verne CD vs Corban HW vs Claremont CW vs Loyola Marymount EL, the names of the member for the government and government whip for Claremont CW were entered the wrong way round. Because their scores differed by two, this affected speaker rankings.

In Round 1, Berkeley BS vs Denver BF vs Chicago KL vs Rochester ST, the names of the leader of the opposition and deputy leader of the opposition for Denver BF were entered the wrong way round. However, their scores were the same, so only speaker position data was affected.

In Round 4, USC HR vs Alaska CH vs Northeastern LS vs Northeastern MS, the names of the leader of the opposition and deputy leader of the opposition for Alaska CH were entered the wrong way round. However, their scores were the same, so only speaker position data was affected.

It should be said here that Tabbycat, unlike Tabbie2 and Tournaman, collects speaker position data separately from speaker scores. This means that scoresheets reflect speaking order, and also allows statistical analysis by speaker position after the tournament.

Thoughts on data entry procedures

Most of my thoughts on procedural changes to make are over at this post, summarizing results from the scoresheet handwriting quiz that these errors inspired. There are just a handful of additional notes here:

• My suspicion that confirmation bias (in the second person, who checks the data entered by the first person) played a role, and consequent views on complete double-blind entry, was partly informed by discussions with tab assistants after I discovered the errors.
• I don’t believe that adding more redundancy will help. With some of these errors, two people needed both to miss all of several triggers for rejecting a ballot. That’s at least six or seven things that have to go wrong. If a six-point system fails at every stage, the problem isn’t with how much redundancy we have, it’s with the efficacy of individual checks.
• Obviously, I wish I had emphasized better to tab assistants that they should be overzealous in rejecting ambiguous ballots. Actually, some assistants were surprised at how low I set the bar for summoning a chair, so simply telling them to do so isn’t really enough. This is another reason to keep an example bank of awry ballots from the past: to persuade assistants that such pedantry is in fact warranted.
• In case you were wondering, it wasn’t just tab assistants without prior experience who made mistakes. All three experienced tab staff were represented in at least one of the above errors. (Tabbycat tracks who enters and checks each ballot, and tab staff and assistants write their initials on ballots after they’ve dealt with them.) This is just data entry, and I continue to believe that anyone who is competent in general can do a fine job of it. Mistakes occur because we’re human, not because we’re inexperienced. The salient principle is that tab systems should be tolerant of ordinary human error.
• As discussed in that post, more effective checking also means longer delays. If we want to reduce one without exacerbating the other, we have to reduce handwriting ambiguity at its source. The other post discusses some ways to do that.

Finally: while this is the first data entry error I’ve known to have happened under my watch, I’ve also never undertaken a full review where I personally type in every ballot into a duplicate tab afterwards, on top of the routine double-checks. Some tabbers do full visual checks of this nature (I don’t), but I don’t know of anyone who personally types every ballot in again, at least not for tournaments of this size. (This is for good reason — this review actually took the entire weekend.) So while I like to think this isn’t the norm, I don’t really know whether this error rate is an anomaly, or par for the course.