Activity 2

Observations on problem

An issue of primary interest is whether different judges are in
some sense 'fair' in their treatment of these appeals.

We could test whether judges' findings are consistent with those
of raters. However, this would require assuming that the raters
set the standard for fairness.  The judges might have far more
experience and expertise than the raters.

Another possible test would be to investigate whether judges
have similar rates of postive and negative decisions.  This would
be a test of fairness if the cases considered by different
judges are similar, for example if they were completely
randomly assigned.  We have no assurance that this is the case.
In fact there different judges have very different proportions
of cases by language, which may be highly related to 
nation of origin which would, in turn, be related to 
the degree of justification for refugee status.

If we had a way of identifying groups of cases that are
likely to be similar in merit, then we could assess whether
judges are consistent in their decisions for these cases.

Some general comments:

- Don't just cut and paste from examples on the web. You might produce a really good figure for
  the problem from which you did the cutting and pasting. It's like stealing a really
  fancy expensive suit from a wealthy celebrity. It will probably look terrible on you
  because it won't fit and won't be appropriate for the occasion.

- Really complicated figures might impress people who have little interest in the
  subject matter but they won't fool an audience that is seeking to understand
  what you have done.  

- It's tempting to make things complicated to conceal one's own lack of clear understanding.   
  The best proof you really understand something is to explain it in a way others can understand.

- Start by sketching what the figure should look like. Look at it as if you knew little
  about the topic. Would it make sense? Would it be informative? Could you explain it to your
  brother who is a literature major? Would he say "hey, I see, that's cool!" or would
  he say "you're giving me a migraine!"
 
- Once you know what you want, only then do the programming to achieve your goals. 
  You might need to compromise but try to achieve the main goals of your graph.

- ALWAYS ask yourself: Would this be clear and interesting to my intended reader?

- NEVER add elements that are uninformative and distracting.

- ALWAYS have informative detailed captions in tables and figures for 'final' reports

- The validity of the statistical tests used in Greene and Shaffer are based on
  random assignment to judges and a lack of clustering in the assignment to
  judges. Neither of these conditions would seem to be strictly valid, the former
  because of relationships between location of appeal and reviewing judge and the
  latter because of assignments in weekly batches.