Innovation in How We Choose and Evaluate

I sometimes serve as a judge or evaluation committee member for grants, awards, and the like. What I have come to feel is that the way we choose may also need innovation.

We pay little attention to "choosing." In particular, when it comes to the "process of choosing" and the "method of choosing," we tend to adopt approaches such as majority voting, or averaging and summing each person's scores to produce a ranking.

However, doubts have recently been raised about whether this way of choosing is really adequate, and various proposals have been made¹. In elections, for example, introducing lotteries and Quadratic Voting are well known. In science funding, random funding, which awards grants at random, is also being tried. I have also heard of grants that select recipients through something like a baseball draft, on the premise that the person in charge sees them through to the end.

Likewise, I feel that by trying out these new methods of choosing in the review of R&D subsidies and startup grants, we may be able to allocate funds more appropriately.

Example: J-Startup

Let's take J-Startup as an example.

J-Startup uses a nomination system by reviewers. Each nominating committee member nominates five companies, and each nomination is weighted². The list is then narrowed down while taking factors such as past track records into account.

This system has the following drawbacks.

• Because many of the nominating members are investors, a startup that receives small investments from many investors can easily collect many votes, which gives it an advantage.
• On the other hand, a startup that receives large investments from a small number of investors, or a startup that has not raised funds, will have fewer nominators and is at a disadvantage.

There is also the question of whether it is fine for the 3rd place of one nominator, A, and the 3rd place of another nominator, B, to carry the same score. Suppose A is very knowledgeable and knows 200 startups, and picked a startup as 3rd among them. Suppose B, on the other hand, is new to the industry and picked a 3rd place while knowing only about 10 startups. The meaning of each 3rd place seems very different, yet the score is the same.

There are also problems after the scores are assigned. When this kind of ranking method is adopted, what usually becomes an issue is what to do with the "middle ranks." The top and bottom are mostly settled, but discussion time tends to concentrate on whether to pass or reject those in the middle, whose evaluations are close to the cutoff line. And the outcome is decided by a slight difference.

Furthermore, nominators do not know all the startups. They have to make nominations while probably knowing less than 5% of them, and they even have to rank them. Yet collecting all the candidate startups from the nominators first (probably more than 200 companies) and then having each nominator rank all of them would impose too high a cognitive load to be feasible.

This method has several limitations. What approaches could change this situation?

Example: Using Best-Worst Scaling or Pairwise Comparison

For example, I think one option is to use Best-Worst Scaling (BWS).

Best-Worst Scaling is a method in which people are shown about 4 to 5 options and asked to select only the "best" and the "worst."

An even simpler version is a technique called pairwise comparison. In it, you simply show two options and choose which one is better, and repeat this.

Let's think about how J-Startup's review would work if pairwise comparison were used.

1. Nominators propose five candidate startups without ranking them
2. The secretariat compiles them into a list and imports it into a pairwise comparison tool
3. Evaluators (who may also be nominators) choose, from two startups shown on the screen, the one they think is better suited to J-Startup
4. This is repeated about 100 times
5. The secretariat aggregates the evaluators' scores in the system and produces a ranking

This method tends to involve a more complex model than majority voting or scoring, and it would have been quite difficult to carry out in the days before computers. With the power of computers, however, the calculation can be done instantly.

In fact, reviews of this kind are conducted at hackathons and similar events. An OSS tool called Gavel, developed for HackMIT, where more than 100 judges evaluate more than 200 projects, does this almost automatically. Here is a demo video of Gavel. The underlying ideas are introduced in this article and elsewhere.

This method also has the drawback that it is harder to give applicants feedback on what to improve than with a rubric or other approach that has clear evaluation criteria. It cannot be used for every kind of review or evaluation. In addition, the range in which this method can be applied is still being researched, and I may be wrong. (If my understanding is mistaken, please let me know.)

Under other conditions, the current scoring by each evaluation item may be the right fit, and in some cases Quadratic Voting, or Quadratic Funding, which applies it, may be the right fit.

Still, just by changing the rules of choosing, the companies that are selected should change significantly.

The Review Process

So far the discussion has been about the method of choosing itself, but the review process can also be improved.

Under current grants, everyone prepares a large amount of documentation and applies. Those who are accepted are fine, but for those who are rejected, all of that effort goes to waste.

For example, the NSF's SBIR divides the review into two stages, asking for a short pitch in the first stage and a full proposal in the second, thereby reducing the burden on applicants. FoundX also uses a similar mechanism to keep the burden as light as possible. It would also be possible to make the first stage consist only of "sending existing materials" and to ask in more detail from the second stage onward.

The drawbacks are that it takes more effort on the reviewers' side, that the review takes longer, and that some things cannot be determined from a short pitch alone. Still, if the benefit of reducing the burden on applicants and increasing the number of applications outweighs those drawbacks, I think adopting such a process is a possible option.

That said, too many stages are also a burden on applicants, so I believe two to three stages would be the limit.

How to Choose the Reviewers

How reviewers are chosen also has a major impact on those being selected. How committee members are chosen depends almost entirely on the secretariat, and a large part of the review outcome is determined by the selection of the committee.

Consider grants for startups. The amounts are small, but they greatly shape the direction of the companies that receive them. The earlier the grant being decided, the more important the reviewers' ability to spot promising companies actually is, yet because the sums are small, it seems that people with that ability are not often assigned. I also sometimes wonder how many reviewers actually understand the purpose of the grant they are reviewing.

If you think that small amounts are not important and that it is enough to be able to support a large number of recipients, then I think that, after applying some threshold cut, funding at random would be easier and fairer.

Also, considering that VCs tend to invest by looking at the people, the earlier the stage, a draft system might be better than majority voting or scoring: it would allow selection from various perspectives and make the selectors responsible even after the selection. That seems like an option as well.

Summary

Grants at the early stage of founding a company can be a matter of life and death for the company. For startups involving hardware, the business often cannot move forward unless they first obtain a grant. Moreover, subsidies are funded by tax money, and as an individual I feel frustrated when subsidies are given to companies that clearly do not fit the purpose of the grant.

There is still room for improvement within the current rules, but I think changing the way of choosing itself could change things greatly, so I am summarizing examples in this article.

Also, regarding this kind of "way of choosing," although proposals such as Quadratic Voting have been made, the current situation is that it is difficult to greatly change electoral systems. So I think it may be worthwhile to start by changing how grants and the like are chosen, and to test the effectiveness of such methods.

I also feel that there are various possible improvements to what happens "after choosing," that is, to the operation of grants. Moreover, choosing and what follows are two sides of the same coin. For example, if the stage gates after selection can be made stricter, it becomes possible to relax the review.

I would like to discuss this "after choosing" on another occasion.

Footnotes

1. See also Questioning Majority Rule. ↩

2. Disclosure: The author has participated as a J-Startup nominating committee member since fiscal 2022. In that role, the author has given feedback on the way of choosing to the secretariat. ↩