The Mysterious Bug in qbreader's Answer Checker, and How I Fixed It

For the longest time, qbreader.org would mysteriously crash once every week or so¹ for reasons that I could not explain. Every time I checked the logs, I saw something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

TypeError: tokens[i].endsWith is not a function
    at tokenize (file:///app/node_modules/qb-answer-checker/lib/tokenize.js:170:19)
    at checkAnswer (file:///app/node_modules/qb-answer-checker/lib/check-answer.js:34:29)
    at file:///app/routes/api/check-answer.js:8:41
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
    at next (/app/node_modules/express/lib/router/route.js:149:13)
    at Route.dispatch (/app/node_modules/express/lib/router/route.js:119:3)
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
    at /app/node_modules/express/lib/router/index.js:284:15
    at Function.process_params (/app/node_modules/express/lib/router/index.js:346:12)
    at next (/app/node_modules/express/lib/router/index.js:280:10)

You can see here that the checkAnswer function, which I spun off into its own npm module, is the culprit here. The first thing you may ask is what inputs were passed into this function to cause the error. Great question! I wish I logged this - it would have saved me so much time - but more on that later.

You may think, “this doesn’t seem that bad, clearly tokens[i] is supposed to be a string but isn’t, so just follow the stack trace”. I thought so too, but I was very confused, since before this, I had recently overhauled the code (see commit) in the hopes that (among other things) I could annotate and track the types of everything, causing this bug to disappear.

But, the bug still persisted, so I decided to dig in. In the qb-answer-checker Github repository where the source code lives, I have a bunch of tests that test for correctness. I decided to run my code on those tests, printing everywhere along my code to see where mysterious output could occur.² I couldn’t find anything, so I tried the next thing:

I ran the checkAnswer function on all 200,000 tossup answerlines in the database. I wrote a small program to do so, which I’ve reproduced in full:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

import "dotenv/config";

import { MongoClient } from "mongodb";
import checkAnswer from "../lib/check-answer.js";

const uri = `mongodb+srv://${process.env.MONGODB_USERNAME || "geoffreywu42"}:${
  process.env.MONGODB_PASSWORD || "password"
}@qbreader.0i7oej9.mongodb.net/?retryWrites=true&w=majority`;
export const mongoClient = new MongoClient(uri);

await mongoClient.connect();
console.log("connected to mongodb");

export const qbreader = mongoClient.db("qbreader");

const total = await qbreader.collection("tossups").countDocuments();
let count = 0;
const cursor = qbreader.collection("tossups").find({});
let current = await cursor.next();
while (current) {
  count++;
  if (count % 1000 === 0) {
    console.log(`${count} / ${total}`);
  }
  const { answer, answer_sanitized: answerSanitized } = current;
  try {
    checkAnswer(answer, answerSanitized);
    current = await cursor.next();
  } catch (e) {
    console.log({ answer, answerSanitized });
    console.error(e);
    break;
  }
}

The idea is to create a cursor over the entire collection, checking each answer against its sanitized version (the one with no HTML tags), which should hopefully cover every case of the checkAnswer code. It ran, and it ran, and as it approached the 92% mark, I was getting nervous that this wouldn’t work and I’d have to try something else, until it finally crashed on this answerline from 2024 ACF Winter:

1

Soviet Union [or USSR or SSSR or Soyuz Sovietskikh Sotsialichetskhikh Respublik; accept Russia or Russian Empire or Rossiya or Rossiyskaya Imperiya] (Pavel Tchelitchew is the subject of Davenport’s essay“Tchelitchew” from The Geography of the Imagination. The second sentence refers to El Lissitzky’s The Constructor. Vladimir Tatlin created Corner Counter-Relief and designed the Monument to the Third International.)

Do you see what’s wrong with it? Yeah, I didn’t either. But, I repeated my procedure from above, inserting console.log statements to figure out the line that failed, until I eventually stumbled upon it:

1
2
3

if (tokens[i] in ordinalConversions) {
  tokens[i] = ordinalConversions[tokens[i]];
}

ordinalConversions, which is supposed to be a dictionary, also is an Object, and as such, it comes with some properties that it inherits from its prototype. In this case, when a string, such as our answerline from above, contains the word “constructor”, instead of skipping over the if block because ordinalConversions doesn’t contain that as a key, it instead returns the constructor of ordinalConversions, which is a function! This explains why we’re getting the type error from earlier - functions don’t have .endsWith defined on them.

Luckily, the fix is simple: only check for properties that we have defined and excluding ones that we inherit:

1
2
3
4

- if (tokens[i] in ordinalConversions) {
+ if (Object.prototype.hasOwnProperty.call(ordinalConversions, tokens[i])) {
  tokens[i] = ordinalConversions[tokens[i]];
}

…and repeat it everywhere else I make a similar check. You can view the full commit on Github: 9f2d32b.