[EM] Voter satisfaction measure in a general case?

[Magosányi Árpád]

Hi,

I am looking for established methods to measure voter satisfaction for
Condorcet votes, where the result is available, but no data on the utility
function.

I understand that it can be measured by assigning an utility function to
each voter, and see how close the result came to that utility function.
However I could not find anything on how to construct that utility function.

My thoughts:
- I would use a "satisfaction index"
- The best choice have 100% satisfaction index (I am okay with
disregarding the fact that even that might not be 100%)
- If there is a dummy choice (e.g. "I disagree with anything I rank below
this"), than that one is 0%, and the worst ranked is -100%
- If there is no dummy choice, then the worst ranked is 0%

Now what should the function look like for the choices in between?
- Linear (as it is easy and middle ground)?
- Logarithmic/exponential (as choices might mostly near good solutions or
mostly bad)?
- Gaussian (as we are talking about Bayesian regret)?
- Result of some clever computation based on vote statistics?

[Juho Laatu]

I think you can't really have any good rating style satisfaction measures in methods that measure rankings only.

Hi,
I am looking for established methods to measure voter satisfaction for Condorcet votes, where the result is available, but no data on the utility function.
I understand that it can be measured by assigning an utility function to each voter, and see how close the result came to that utility function. However I could not find anything on how to construct that utility function.
- I would use a "satisfaction index"
- The best choice have 100% satisfaction index (I am okay with disregarding the fact that even that might not be 100%)

It seems you want to measure normalised satisfaction, not absolute satisfaction. I mean that even if my true preferences would be a=52 b=51, you will count my preferences as a=100, b=0. That's ok, although it doesn't reflect voter satisfaction accurately (to me all candidates were about equal).

- If there is a dummy choice (e.g. "I disagree with anything I rank below this"), than that one is 0%, and the worst ranked is -100%

Here you actually have "data on the utility function". You ask voters to tell where their preferences change from "ok" to "not ok". You might get quite sincere results. But that applies also to asking voters to rate all the candidates in range 100% to -100%. Rankings can then be derived from those ratings.

The dummy choice might however be used for some other reason (not just to collect ratings). One could for example have an election that would be inconclusive if the winner loses to the dummy choice (acceptability limit). In that situation you would get the (0%) ratings sort of free, without really asking for ratings :-) .

- If there is no dummy choice, then the worst ranked is 0%
Now what should the function look like for the choices in between?
- Linear (as it is easy and middle ground)?
- Logarithmic/exponential (as choices might mostly near good solutions or mostly bad)?
- Gaussian (as we are talking about Bayesian regret)?
- Result of some clever computation based on vote statistics?

I'll try to address the problem of measuring in between candidate satisfaction measures using some simple examples.

49: a=100, b=99, c=0
02: b=100, a=1, c=0
49: c=100, b=99, a=0

It is not possible to see from these rankings that voters like b very much.

The ratings could be as well:
49: a=100, b=1, c=0
02: b=100, a=1, c=0
49: c=100, b=1, a=0

We might have also some problems with clones.

50: a1 > a2 > a3 > a4 > b
50: b > a1 > a2 > a3 > a4

Most functions will give a1 a high score although b voters might dislike a1 about as much as a1 voters dislike b.

In summary, if you want to estimate satisfaction based on the given rankings (and maybe the acceptability threshold), you must make some assumptions like having an even distribution of candidates (in the voter opinion space). Collecting ratings (and deriving rankings from them) would be one step simpler, but not be problem free either.

If I'd assume even distribution of candidates, maybe the linear approach (out of the ones that you proposed) would be sufficient. Or maybe it is typical that voters like their own candidate and hate most of the others. In order to find out what this function looks like (it could be quite different in different voting societies), one could make a small query and ask people how they sincerely rate different candidates. One could approximate the function (curve between favourite 1000% and least favourite 0%) based on those answers.

BR, Juho


----
Election-Methods mailing lis

[Kristofer Munsterhjelm]

Post by Juho Laatu
I think you can't really have any good rating style satisfaction
measures in methods that measure rankings only.

The only thing I can think of, statistically speaking, is to set up some
kind of utility generator that produces ratings style votes, and then
randomly sample elections from this utility generator. Throw away every
sample that doesn't have the same ranked reduction as the actual data
you have, and then calculate the mean utility for each candidate across
the samples that remain.

In other words, if you have an election that is

3: A>B>C
2: B>C>A
1: C>A>B

(say)

then you run your utility generator and save every generated scenario where:

3 voters rate A higher than B and both higher than C
2 other voters rate B higher than C and both of those above A
1 other voter rate C higher than A and both of those above B
there are no other voters

and then you take the mean utility for A, B, and C across all those
generated scenarios.

In practice, this method becomes completely impractical whenever the
number of voters is greater than say, 10 or so. It might be possible to
use statistical cleverness to speed up the sampling, but it would
probably take a long time to find out just how to be clever in such a way.

It would not be an established method. The utility generator would also
have tunable parameters (spatial model? how many dimensions? degree of
correlation, etc), and those would have to be set depending on the
political context. Introducing noise (n voters vote however they want)
would also make it significantly harder.

Apart from that, the only way I think you can do it is to dissolve the
problem by asking for ratings information.
----
Election-Methods mailing list - see http://electorama.com/em for list info

[Juho Laatu]

Post by Juho Laatu
I think you can't really have any good rating style satisfaction
measures in methods that measure rankings only.

The only thing I can think of, statistically speaking, is to set up some kind of utility generator that produces ratings style votes, and then randomly sample elections from this utility generator. Throw away every sample that doesn't have the same ranked reduction as the actual data you have, and then calculate the mean utility for each candidate across the samples that remain.

Makes sense. That way you can get a guesstimate on what the rated opinions might have been. In this method you probably assume some level of even distribution of candidates in the sense that if votes are 50:A>B>C, 50:C>B>A, the algorithm assumes that B has equal distance to both A and C. Also vote normalisation will be assumed.

In other words, if you have an election that is
3: A>B>C
2: B>C>A
1: C>A>B
(say)
3 voters rate A higher than B and both higher than C
2 other voters rate B higher than C and both of those above A
1 other voter rate C higher than A and both of those above B
there are no other voters
and then you take the mean utility for A, B, and C across all those generated scenarios.
In practice, this method becomes completely impractical whenever the number of voters is greater than say, 10 or so. It might be possible to use statistical cleverness to speed up the sampling, but it would probably take a long time to find out just how to be clever in such a way.

Maybe you could use also samples that are close enough to the actual vote set (with some weight that decreases when the distance to actual votes grows).

It would not be an established method. The utility generator would also have tunable parameters (spatial model? how many dimensions? degree of correlation, etc), and those would have to be set depending on the political context. Introducing noise (n voters vote however they want) would also make it significantly harder.

Old rating based polls could be used as a base. The rankings of this election could be compared to the rankings of those old polls where we have both rating and (derived) ranking information available. Then we would tweak the old ratings a bit into the apparent direction to get our current estimate.

Apart from that, the only way I think you can do it is to dissolve the problem by asking for ratings information.

Ratings would be more accurate. But also they would have some problems, like interest to push the worst opponents down in ratings in order to give an impression that nobody likes them.

Vote normalisation is also an interesting problem. Maybe the ratings would not be just on a numeric scale, but on a scale that has some named points like "unacceptable", "neutral", "very good".

Juho


----
Election-Methods mailing list - see http://electorama.com/em for list info

[Magosányi Árpád]

Thanks for the good inputs.I try to summarize here

proposed methods:

- set up some kind of utility generator that produces ratings style votes,
and then randomly sample elections from this utility generator.
- use previous rated votes similarly to the utility generator above
- ask for ratings

I also thought about asking about perceived closeness of alternatives

The reason for asking was to figure out a way to show voter satisfaction
measure with vote results, in a general voting application.
Directions towards solutions I see:
- utility generators and previous similar votes seems to be too complex for
the task
- optionally asking rates from voters (maybe asking them to put the choices
on a two-dimensional field, along with their own position) and use that as
the basis of the utility function is an interesting idea maybe worth
exploring
- leaving the choice of an appropriate utility function form to the user
when voter satisfaction is measured is the easiest

Post by Kristofer Munsterhjelm

Post by Kristofer Munsterhjelm

Post by Juho Laatu
I think you can't really have any good rating style satisfaction
measures in methods that measure rankings only.

The only thing I can think of, statistically speaking, is to set up some

kind of utility generator that produces ratings style votes, and then
randomly sample elections from this utility generator. Throw away every
sample that doesn't have the same ranked reduction as the actual data you
have, and then calculate the mean utility for each candidate across the
samples that remain.
Makes sense. That way you can get a guesstimate on what the rated opinions
might have been. In this method you probably assume some level of even
distribution of candidates in the sense that if votes are 50:A>B>C,
50:C>B>A, the algorithm assumes that B has equal distance to both A and C.
Also vote normalisation will be assumed.

Post by Kristofer Munsterhjelm
In other words, if you have an election that is
3: A>B>C
2: B>C>A
1: C>A>B
(say)
then you run your utility generator and save every generated scenario
3 voters rate A higher than B and both higher than C
2 other voters rate B higher than C and both of those above A
1 other voter rate C higher than A and both of those above B
there are no other voters
and then you take the mean utility for A, B, and C across all those

generated scenarios.

Post by Kristofer Munsterhjelm
In practice, this method becomes completely impractical whenever the

number of voters is greater than say, 10 or so. It might be possible to use
statistical cleverness to speed up the sampling, but it would probably take
a long time to find out just how to be clever in such a way.
Maybe you could use also samples that are close enough to the actual vote
set (with some weight that decreases when the distance to actual votes
grows).

Post by Kristofer Munsterhjelm
It would not be an established method. The utility generator would also

have tunable parameters (spatial model? how many dimensions? degree of
correlation, etc), and those would have to be set depending on the
political context. Introducing noise (n voters vote however they want)
would also make it significantly harder.
Old rating based polls could be used as a base. The rankings of this
election could be compared to the rankings of those old polls where we have
both rating and (derived) ranking information available. Then we would
tweak the old ratings a bit into the apparent direction to get our current
estimate.

Post by Kristofer Munsterhjelm
Apart from that, the only way I think you can do it is to dissolve the

problem by asking for ratings information.
Ratings would be more accurate. But also they would have some problems,
like interest to push the worst opponents down in ratings in order to give
an impression that nobody likes them.
Vote normalisation is also an interesting problem. Maybe the ratings would
not be just on a numeric scale, but on a scale that has some named points
like "unacceptable", "neutral", "very good".
Juho
----
Election-Methods mailing list - see http://electorama.com/em for list info

[Juho Laatu]

Different elections could need different kind of ballots. For example, if the number of candidates is high, it could be easier to rate the candidates than rank them all. The ballot could e.g. have one row for each candidate, and twenty boxes to the right of the name of each candidate. The voter would tick one box for each candidate, or not tick any boxes, which would indicate the lowest possible value. This could be a practical approach in some elections, and it would give you also some sort of rating values. Collecting ratings this way (as a by-product) could be a viable approach if you want to keep it simple.

If you have those boxes in the ballot, or some other kind of scale, there are two options. The scale can be either abstract numbers / positions only, or it could contain some guidance, like three lowest boxes having different colour and a descriptive word "not acceptable", etc. Such descriptions would give you some guidance on how strong the feelings of the voters are, and it would make all the voters use the scale in the same way.

One question about voter satisfaction. My ratings are A=40, B=30, C=10. B wins the election. What is the level of my satisfaction? Is it 30 (that's how much I like B)? Or is it 75 (based on normalising my vote to range 0..100)? Or is the level of my dissatisfaction 10 (40-30, that's how much I would have been happier if A had won)? If there is also candidate D that I didn't rate at all (maybe 0 by default), that might or might not change the (above mentioned) normalised value.

The results of Condorcet methods can be quite difficult to interpret to normal voters if the outcome of the election appears to be a pairwise matrix or a complex graph that points out the results of each pairwise comparison. It would be simpler if one could give just one number that indicates how strong the victory was (one way to measure the overall stability of the election, and thereby also some form of satisfiability of the result). Most notably minmax(margins) decides the winner and measures the strength of the victory by counting the number of required extra votes (that would be needed) to win the election (instead of the current winner). This measure is based on number of voters only, not on ratings. All candidates will get a number that tells how far they were from winning the election. That distance could tell something also on the satisfaction level of the supporters of the competing candidates. This was just a side note, but maybe interesting when trying to find numerical measures describing the outcome of the election.

Juho

Post by Magosányi Árpád
Thanks for the good inputs.I try to summarize here
- set up some kind of utility generator that produces ratings style votes, and then randomly sample elections from this utility generator.
- use previous rated votes similarly to the utility generator above
- ask for ratings
I also thought about asking about perceived closeness of alternatives
The reason for asking was to figure out a way to show voter satisfaction measure with vote results, in a general voting application.
- utility generators and previous similar votes seems to be too complex for the task
- optionally asking rates from voters (maybe asking them to put the choices on a two-dimensional field, along with their own position) and use that as the basis of the utility function is an interesting idea maybe worth exploring
- leaving the choice of an appropriate utility function form to the user when voter satisfaction is measured is the easiest

Post by Juho Laatu
I think you can't really have any good rating style satisfaction
measures in methods that measure rankings only.

The only thing I can think of, statistically speaking, is to set up some kind of utility generator that produces ratings style votes, and then randomly sample elections from this utility generator. Throw away every sample that doesn't have the same ranked reduction as the actual data you have, and then calculate the mean utility for each candidate across the samples that remain.

Makes sense. That way you can get a guesstimate on what the rated opinions might have been. In this method you probably assume some level of even distribution of candidates in the sense that if votes are 50:A>B>C, 50:C>B>A, the algorithm assumes that B has equal distance to both A and C. Also vote normalisation will be assumed.

In other words, if you have an election that is
3: A>B>C
2: B>C>A
1: C>A>B
(say)
3 voters rate A higher than B and both higher than C
2 other voters rate B higher than C and both of those above A
1 other voter rate C higher than A and both of those above B
there are no other voters
and then you take the mean utility for A, B, and C across all those generated scenarios.
In practice, this method becomes completely impractical whenever the number of voters is greater than say, 10 or so. It might be possible to use statistical cleverness to speed up the sampling, but it would probably take a long time to find out just how to be clever in such a way.

Maybe you could use also samples that are close enough to the actual vote set (with some weight that decreases when the distance to actual votes grows).

It would not be an established method. The utility generator would also have tunable parameters (spatial model? how many dimensions? degree of correlation, etc), and those would have to be set depending on the political context. Introducing noise (n voters vote however they want) would also make it significantly harder.

Old rating based polls could be used as a base. The rankings of this election could be compared to the rankings of those old polls where we have both rating and (derived) ranking information available. Then we would tweak the old ratings a bit into the apparent direction to get our current estimate.

Apart from that, the only way I think you can do it is to dissolve the problem by asking for ratings information.

Ratings would be more accurate. But also they would have some problems, like interest to push the worst opponents down in ratings in order to give an impression that nobody likes them.
Vote normalisation is also an interesting problem. Maybe the ratings would not be just on a numeric scale, but on a scale that has some named points like "unacceptable", "neutral", "very good".
Juho
----
Election-Methods mailing list - see http://electorama.com/em <http://electorama.com/em> for list info