HOW-TO: Meta-Reviewer Compares Papers
OVERVIEW
When enabled by the Chair, the Meta-Reviewer will have access to a page that will allow comparisons of papers to be made - which paper is better than another, or which paper is equal to another. It uses Paper IDs to distinguish the papers.
Note
The Chair must also enable the Meta-Review Submission Activity and set the deadline to a date in the future in order for Meta-Reviewers to be able to compare papers.
Important
Papers from different tracks cannot be compared.
META-REVIEWER CONSOLE
- From the Meta-Reviewer Console click 'Actions>Paper Comparison.' The 'Paper Comparison' page appears.
PAPER COMPARISON PAGE
Here you can add any number of non-contradicting comparisons between papers. For instance,
- A > B You prefer paper A over B, i.e. given a choice, you would accept paper A instead of B.
- A ~ B You consider paper A and B to be comparable and about the same.
- Click 'Add Comparison +'
The action row appears.
In this case we have 6 papers. Lets' say Paper ID 5 is about the same as Paper ID 9.
- Click the 'Left paper' dropdown. Pick Paper ID 5.
- Click the 'Operatore' dropdown, select the tilde
- Click the 'Right paper' dropdown, select Paper ID 9
- Click 'Save'
It shows the comparison equation that Paper ID 5 is approximately equal to Paper ID 9.
If a comparison is deemed invalid then an alert 'Comparison violation detected' will appear at the bottom of the page.
- For example we add a comparison that paper 18 is better than 12. Click 'Save'. The alert appears.
- Now input the following by clicking 'Add comparison +' and saving after each equation:
2 ~ 9
3 > 5
4 > 9
4 > 3
- Click 'Get Summary'
It shows a hierarchal result. What can be derived from the summary is that Paper 3 is better papers than 2, 5, 9; while Paper 4 is beter than all.
Important
Papers from different tracks cannot be compared.
If a Meta-Reviewer (assigned papers in different tracks of a multi-track conference with the comparison enabled in both tracks) tries to compare papers from different tracks, a 'Bad Request' alert error will appear at the bottom of the page.