Superforecasting Challenge: Update 2024

Do you believe that some people are gifted with the ability to make really good predictions about the future? According to theory, such people exist and are called “superforecasters”. Superforecasters don’t have to have extra education or specializations— they can come from any walk of life. But they do have certain habits and ways of thinking that enable them to consistently predict the outcomes of future events with significant accuracy. Again, according to theory. And some evidence.

Each year, we at the Tandy Center run a Superforecasting Challenge. The future events that we are trying to predict are related to the environment and climate in our home state of Maine.

Here is a summary of the group’s performance for the 2024 Superforecasting Challenge questions. Note that a 50-50 guess would give a score of 0.25, and lower scores are better.

There were eight questions/events:

Event 1. GMRI to report summer 2024 as among the three warmest summers on record in the Gulf of Maine. The summer of 2024 clocked in at the seventh warmest on record, well below the group’s expectations. The group average score was 0.50, significantly worse than a 50-50 guess.

Event 2. DMR to report lobster landings for 2023 as at or above 80 million pounds. Landings were over 90 million pounds. The group average score was 0.10, significantly better than a 50-50 guess.

Event 3. A single weather or storm event to knock out power to at least 200,000 Maine customers in 2024. In early April, CMP reported over 259,000 customers out of power. The group average was 0.04 — highly certain of this kind of power outage, and way better than a 50-50 guess.

Event 4. Ice out date for Damariscotta Lake in 2024 to be reported as on or after April 1. No April Fools ice out here; open water came in mid March. The group average score was 0.13, significantly better than a 50-50 guess.

Event 5. More than 4,000 deer ticks (Ixodes scapularis) reported in the UMaine Tick Surveillance Program Annual Report for 2023. The count came in at a bit over 3,000, and perhaps our collective fears of ticks inflated our guess. The group average score was 0.37, significantly worse than a 50-50 guess.

Event 6. Popham Beach State Park to issue a closure to water access due to a confirmed presence of a white shark between February 1 and October 1, 2024 (including those dates). Again, perhaps fear won out here? There was no closure, and the group average score was 0.41, significantly worse than a 50-50 guess.

Event 7. Confirmed sighting of a Steller’s Sea Eagle in Maine reported in the Press Herald between February 1 and November 1, 2024 (including those dates). The Sea Eagle seems to have settled in in Newfoundland for now. Realism won slightly over hope here, and the group average score was 0.17, slightly better than a 50-50 guess.

Event 8. Five or more North Atlantic right whale births in January-March 2024. This was a quick win for the right whales, but only a slight win for the aspiring superforecasters. The group average score was 0.20, ever so slightly better than a 50-50 guess.

Some final notes:

• We had 40+ perticipants. With eight questions, there are 2^8 = 256 possible outcomes. At the outset, every single participant had a potential path to victory.
• The winning score among the official participants was 0.11. (One unofficial ten-year-old participant scored 0.03!!)
• The overall group average score was 0.24 (median 0.26). Again, for context, guessing 50% on everything would have given a score of 0.25.
• There were also some anonymous unofficial participants (a middle schooler, and a professional forecaster), who scored a little better.
• We also had an AI participant (Chat GPT), who came in at slighly worse… 0.26.

All in all, things clustered pretty tightly around that 0.25 number. Can we do better than a coin toss? One test of superforecasters is how they learn from their previous performances.

See you in 2025!