Katja's Institution Design Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== How to make valuable prediction markets? ======

Prediction markets don't seem to add as much value in reality as they promise in theory. This is notes on how to do better.

===== General impediments to prediction markets being valuable =====
  * Explicit public records of many important inputs to decision-making are embarrassing or stressful for people involved. E.g. ‘will Alice regret hiring Bob by February 28?’ or quantitative estimate of particular people’s expected research outputs (like if we had markets on each employee’s value this year in some sense, they might not like it)
  * Surprisingly hard to find things people actually want to know (in Katja’s opinion)
  * To make forecasts in general, you want precise resolution criteria. But those are far from what people use to make decisions. E.g. maybe I want to run an event if I think it will be broadly awesome. So I make a prediction market about whether it will be awesome, but that’s vague, so I specify it as ‘will it have at least fifty guests, and will the first five I ask give it a 4 or 5 out of 5 rating?’ Then I have a clear answer on that, but that wasn’t the same as ‘awesome’ and it is less clear how to use this answer.
  * Often things that are valuable to know about are fairly unlikely, and prediction markets aren’t great at distinguishing 0.01% and 1%


===== Ideas for how to make prediction markets more valuable =====
  * Somehow make predictions about vague things directly. For instance, via human judgment: e.g. ‘How good will it be if we run another Vignettes Workshop?’ => resolved to 0-10 how good Katja says it was


===== Ideas for classes of valuable prediction market =====
  * **How much funding will org X get?** Seems maybe valuable because funders thinking about what is going to get funded anyway seems like a bit of a mess, though ideally you probably want some kind of coordination between the different parties who might fund a thing vs. everyone playing chicken and hoping someone else will fund something, or whatever happens; for making decisions in orgs, would often be really nice to know if you will have money, and there is much uncertainty. This also seems like a good use case for prediction markets because the info about it is spread among people, and you dont' know who has it really
  * **Effectiveness-related metrics for charities** e.g. 'when this two-year trial is done, how many people will have turned out to get malaria in these villages?'
  * **funding of projects that haven't yet been funded** e.g. 'will I get a $50k grant from SFF or LTFF or OP to build ----'. People often fail to ask for money for projects, when they could get the money pretty easily, because they don't know what it looks like inside the funding orgs.
  * **‘Change my mind’ markets on research questions**
  * **Markets on outcome of research questions** => get answer earlier, but still have to do research to resolve. Or conditional thing, where if it looks like the answer will be uninteresting, you don’t do the research.
  * **If we do [research project], will we [have desired kind of output]** E.g. If I try to get data on price performance for GPUs in the past five years, will I end up with less than an order of magnitude of uncertainty in my estimate of the slope of the curve? [not great, too easy to alter it]
  * **How long will project take?** E.g. If I try to get data on price performance for GPUs in the past five years will I put up a page about it before April 1?
  * **Other costs** E.g. If I try to get data on price performance for GPUs in the past five years, will I cry more than 20 times?
  * **Career decisions** E.g. ‘If I work at MIRI, will I write a paper worthy of being mentioned in the alignment newsletter in 2023?’ + ‘if I work at ARC, will I…’
  * **Recommendations for important problems** E.g. ‘What intervention will Alice do and consider responsible for causing her to work for >20h in some week?’ or ‘Who will Alice marry? (self-nominations only)’ [to avoid people being involved who don’t consent]
  * **What will happen with EA hubs in the coming years** E.g. ‘Will at least 100 people say they live in Austin in the 2025 EA survey?’
  * **What will be the outcome of a trial?** RCTs for instance can be expensive and time consuming, so this is good for both getting results earlier, and in cases where the study isn't done. Apparently superforecasters were great at guessing what will happen in trials, in some research.

===== More places where straightforward forecasts are used in decisionmaking =====

  * What ads will be clicked?
  * 

===== Types of uses for prediction market style forecasts =====

  * Causing visceral updates, e.g. having to put a number on particular AI trait in 2028 makes it feel real in a different way.
  * Info the 

===== Why are prediction markets not as useful as they seem? =====

  * Just takes time for new things to be used (Katja doesn't buy that this is much of it)
  * Questions need to either be formed as decision markets and thus conditional (and those markets are kind of awkward) or they need to be about a factual claim that would imply a decision directly.
  * Prediction markets don't interface with people's normal decisionmaking
===== Ideas for specific valuable markets =====
  * How much will AI Impacts raise?
  * ‘Change my mind’ markets on AI Impacts questions
  * Will Katja think in a year that AI system values are likely to land within the region of human values
  * Will Katja think AI system values are likely to land within the region of human values, after reading the various counter-counter-argument posts that people have written?
  * Same for each of the other counterarguments
  * Will we find an example of $100bn being forgone for ethical/risk reasons?
  * Is there a discontinuity, per our definition, in <metric>?
  * Will we find another ten discontinuities by the end of next year? (people can then tell us them—a better system for this than our bounties?)
  * All of the ‘concrete tasks for AI forecasting’ questions
  * Make up more narrow/near AI questions


===== Examples of potentially valuable prediction markets =====
  * [[https://manifold.markets/NathanpmYoung/how-much-will-rethink-priorities-ra-0c3690316d79]] 
  * [[https://manifold.markets/NathanpmYoung/will-eagx-nordics-happen-as-we-expe]] 
  * [[https://manifold.markets/NathanpmYoung/will-eag-london-happen-as-we-expect]] 
  * [[https://manifold.markets/NathanpmYoung/will-eagx-latin-america-happen-as-w]] 
  * [[https://manifold.markets/NathanpmYoung/will-eag-bay-area-happen-as-we-expe]]