Frequently Asked Questions

PURPOSE

PARTICIPANT BRIEFING

METHOD


dotPURPOSE

What is Vision Prize?

Vision Prize® is a polling platform for capturing meta-knowledge — knowledge about what people know. In addition to assessing the views of scientists, Vision Prize asks its expert participants to predict the views of their scientific colleagues. This approach gives new insight into the level of scientific consensus on various issues, which in some cases may be just as important as knowing the majority view. Charity gift cards — our way of thanking panelists for their participation — are awarded as prizes for exceptional meta-knowledge.

Three-year data collection for our Poll of Climate Scientists ended December 2014, and was run in collaboration with IOP Publishing’s scientific community website, environmentalresearchweb. Vision Prize also is affiliated with researchers at Carnegie Mellon University. Additional data analysis and visualization is currently underway with research partners and collaborators, including IOP. Vision Prize is strictly nonpartisan — we are not an advocacy organization. See also: About.

What is its purpose?

Vision Prize aims to fill the gap between traditional scientific meta-analyses like the IPCC AR5 (which are robust but slow) and more timely sources like the Wall Street Journal editorial page (which may not accurately portray the views of experts). Our purpose: help leaders, managers and investors make better decisions. See also: Learn More. TOP

dotPARTICIPANT BRIEFING

Who is eligible to participate?

The Vision Prize poll is open to climate scientists, and other scientists or researchers with relevant expertise. This includes:

dot Members of the academic community with relevant expertise (including doctoral students in climate-related fields)

dot Research scientists in relevant industries, government agencies or laboratories, and non-governmental organizations

dot Others with advanced degrees and relevant scientific or technical credentials.


What is the h-index?

We compile h-index scores of our expert participants using Google Scholar for benchmarking, data analysis and quality assurance purposes. H-scores provide an approximate measure of a scientist’s cumulative research impact. On charts of poll results, for example, mean h-scores give readers of the poll an approximate metric for assessing the relative expertise of the participants who selected a given answer. As a point of reference, h ≈ 12 might be a typical value for advancement to tenure at major research universities; membership in the U.S. National Academy of Sciences may typically be associated with h ≈ 45 and higher (Hirsch, PNAS, 2005). See also:

dot Who is eligible to participate?
dot What is the principal criterion for participant selection?
dot Demographics of expert participants TOP

What is the principal criterion for participant selection?

Is the prospective participant generating original scientific research in a climate-related field?

NOTE: Requiring more restrictive criterion like X number of peer-reviewed papers or high h-index has been considered. We have not done so thus far mainly because it might introduce bias against younger researchers (e.g. Ph.D candidates in atmospheric science, some of whom are involved in extensive field work in the Arctic etc.), government or industry research scientists who are less likely to publish, and researchers from academic disciplines with different average publication rates. Since climate science is inherently interdisciplinary, publication-based measures like h-index may introduce such biases. However, we do compile h-index scores of our expert participants for benchmarking, data analysis and quality assurance purposes. See also: Are Internet polls inferior to standard survey? Do incentives create bias in the sample? TOP

What are the benefits of participating?

dot Help policymakers, managers and investors make better climate-related decisions

dot Compare your predictions to those of your peers. (See also: What is myVISION?)

dot Gain recognition among your colleagues as a Vision Prize charity winner and domain expert

dot Support your favorite charity with a cash grant — at no expense to you. TOP

Which charities can winners select?

Charities recommended by participants must be recognized by the IRS and be in good standing with U.S. federal and state regulators. Participants who earn charity grants search an online database of over one million eligible charities — the process is fast and easy.TOP

Who funds the prizes?

Charity gift cards — called “Good Cards” — are the sole source of prize funds, 100% of which benefit public charities. Contributions help expand our poll of scientific experts by incentivizing their participation. Top scorers receive Good Cards to support their favorite charity causes and institutions.

dot Anyone may buy Good Cards — participants, friends & family, foundations, charitable giving programs
dot Good Card purchases are tax deductible in the U.S.
dot Minimum purchase: $25. TOP

How frequently does the poll run?

Polls run at least twice per year. (They take about 5-10 minutes to complete.) Registered experts are encouraged to participate in all rounds, and thus will have opportunities to revise their predictions in follow-on polls based on new information and results from previous rounds. Suggestions for poll questions are welcome. TOP

When will the poll findings be released?

All participants will receive an announcement when poll findings are released. TOP

What is myVISION?

With the myVISION feature of Vision Prize, you can compare your predictions of the views of your colleagues to the true distribution among our sample of experts for the most recently completed polling round, as shown in the example below.

These personalized charts are made available for your viewing — we don’t share them with anyone else. TOP

How does the poll work?

Each Vision Prize question has two parts — the first part will ask which answer you believe to be most likely and the second part will ask you to predict how all participants will answer. Based on your answers and how accurately you predict the answers of the other participants, you will receive a Vision Score, which is a measure of meta-knowledge. The higher your Vision Score, the more you earn in charity gift cards to support the charity of your choice. Charity gift cards are our way of thanking you for your participation. Top scorers are listed in the Leaderboard, which is cumulative and includes all polling rounds.

Not all participants will earn charity donations in every poll, but to maximize your Vision Score and charity donation, you should answer both parts of each question in accordance with your best guess even if you are very uncertain. All questions in this poll ask for your best guess about what you expect — not what you hope — will be the true outcomes. Gift Card winners can choose any public charity to which to donate their prizes through the Give page on the Vision Prize website. TOP

What is the privacy policy?

By default, registered players are associated with a username of your choosing, your affiliation, position and research field(s). You are encouraged to share your real name with other participants, but may elect to remain anonymous. If you choose to remain anonymous, pick an anonymous username and select “hide” for your real name. Vision Prize safeguards your privacy and will never sell your information to a third party. You may close your account — or update your profile — at any time. See also: Privacy Policy (full text). TOP

Why was my registration not completed?

dot If you have not received an email request from us within five minutes requesting confirmation of your email address, please check your spam folder or contact your network administrator for a message entitled “Click link to activate your account for Expert Poll” from registrar@visionprize.com. (If this message was not delivered, we would be especially grateful if you could ask your network administrator that future emails from registrar@visionprize.com be “whitelisted”.)

dot You may need to contact the Registrar to complete your registration if you have not confirmed your email address within a hour. (For security reasons, links need to be used promptly.)

dot To help improve the user experience for you and your colleagues, you are also encouraged to post comments about any website issues on our Forum for Expert Participants. Your feedback is much appreciated. TOP

Why am I unable to log in?

In most cases, this is because the password being entered is incorrect.

dot You may reset your password.

dot Alternatively, in your email records, find “Email account verified” message from registrar@visionprize.com. This email message contains your username and, for recent registrants, your password hint. Use this to confirm that you are using the correct information to log in.

dot If these steps do not resolve your issue, please email Registrar.

dot To help improve the user experience for you and your colleagues, you are also encouraged to post comments about any website issues on our Forum for Expert Participants. Your feedback is much appreciated. TOP

Who are the current expert participants?

Click here to view profile and number of expert participants. This visualization — shows participants by position, affiliation (for those who registered with a professional vs. personal email account), and approximate location of their primary affiliation. See also: Leaderboard.

How do I update my profile information?

Click here to share with us any changes you would like to make to your profile, including email address, “hide” or “show” your real name preference, or link to your public professional page. TOP

dotMETHOD

What does “incentivized” mean?

Incentives both to participate and to carefully consider one’s responses are integral to the design of Vision Prize. Though interest in communicating scientific opinion may be the key motivator, we also make use of a scoring algorithm designed to encourage thoughtful answers, especially with regard to predicting the beliefs of others. The algorithm is published in the journal Science (Prelec, 2004) and described in the next two sections. Vision Scores reward both “surprisingly common” answers and accurate prediction of the distribution of answers. The important thing to know is that answering carefully and honestly will tend to maximize the amount you win in charity gift cards. TOP

How does Vision Prize differ from consensus-based polls?

The Vision Prize incentivized scoring method mitigates the impact of herding and other behavioral biases of conventional aggregation procedures like calculating the mean, median or modal response. It does this by rewarding the answers that are more common than expected, not necessarily the most common overall. This approach means that there is no bias against atypical views and no special incentive to pick the answer you think most others will pick. TOP

Surprisingly Common Example

How are polls scored?

The Vision Prize incentivized scoring system is designed to reward answers that represent your best guess of the true answer (which, for some questions, may not be known for many years). For the first part of each question, it does this by rewarding answers that are more common than the group expects them to be. (You are free to think strategically when giving your answer, but answering truthfully will earn you just as high a Vision Score.) For the second part of each question, the scoring algorithm rewards accurate prediction of the group’s responses to the first part. Your Vision Score is a combination of these rewards. Refer to the example below, which illustrates high scores in a hypothetical case. TOP
Scoring Example

Because of how the Vision Score is calculated, as shown above, Vision Prize can distribute charity gift card prizes without waiting to find out the true answer to the prediction question. See also:

dot Vision Prize builds on history of innovation in scientific opinion polling
dot How does Vision Prize differ from consensus-based polls?
dot How does Vision Prize differ from betting markets?
dot What does “experimental” mean?
dot TOP

How does Vision Prize differ from betting markets?

Betting markets — sometimes called “prediction markets” or “information markets” — allow you to make predictions by buying and selling shares in the outcome of real-world events. Such markets are typically short-term oriented, due to the need to settle bets and pay out winners based on known outcomes. Like crowd-based approaches, betting markets also are susceptible to herding and other misinformative social behavior, and are not selective in terms of whom is eligible to play. Not surprisingly, betting markets tend to focus on politics and popular culture — not scientific and technical issues where expertise matters — and are rarely selective in terms of whom is eligible to play. TOP

Are Internet polls inferior to standard opinion surveys?

Nonprobability “opt-in” polls like Vision Prize have shortcomings, which we have carefully considered from the outset. In particular, we expect our nonprobability sample to be biased towards those with interest and expertise in the topic of the survey. For the case of polling expert opinion, however, our view is that our opt-in approach — with screening for relevant expertise — offers a reasonable way forward, especially in a field like climate science where the population of interest transcends multiple domains and affiliations. See also:

dot What is the principal criterion for participant selection?
dot Do incentives create selection bias?

We also believe that our self-referential “surprisingly common” method is less susceptible to the shortcomings of nonprobability sampling. Provided that the relevant demographics of the participants are clearly communicated, meaningful results can be generated. That said, the larger and more representative our sample of climate scientists, the better. We actively encourage additional scientists to participate.TOP

Do incentives create selection bias in the sample?

To help mitigate significant selection bias induced by monetary incentives, participants only earn rewards for the charity of their choice, not themselves personally. TOP

What does “experimental” mean?

In addition to collecting and publicizing the distribution of beliefs of experts, we are also interested in prediction methods that might be more accurate than taking the mean or modal response of the group. In particular, the same research that provides our incentivized scoring system (Prelec, Science, 2004) also proposes an algorithm that harnesses the relationship between knowledge and meta-knowledge (accurate beliefs about the beliefs of others). Vision Prize is designed to provide an ongoing test of this approach.

Vision Prize is affiliated with researchers at Carnegie Mellon University. We believe better prediction technology is possible through experimentation and a feedback loop between academic research and real-world problems. TOP