CHAIR HOW-TO: Create and Manage Meta-Review Questions
Note
Create meta-review questions in the meta-review form for meta-reviewers to answer about their assigned papers.
OVERVIEW
Meta-Review questions are unique to each conference.
This document will show how to configure the Meta-Review Form with generic questions/tasks/descriptions/calls-to-action.
There are four types of Meta-Review Questions:
- Comment - The Meta-Reviewer enters text to comment on the paper
- Agreement - The Meta-Reviewer agrees to the posted statement/question
- Options - The Meta-Reviewer has a choice of answers. It can be restricted to a single answer or be multiple choice.
- Options with Value - The Meta-Reviewer has a choice of answers (usually restricted to a single answer), each with its own number value. This allows the Chair to quickly see the Meta-Reviewer's opinion of the quality of the paper and aids in the decision-making.
The end of this document will show how to copy questions from one track to another in a multi-track conference.
Note
Everything with a red asterisk is required.
META-REVIEW FORM
Note
If a Meta-Reviewer gets a 'No Questions Configured' error, it means Chairs have not created any meta-review questions yet.
- Go to Settings>Forms>Meta-Review and click ‘+Add new question.’
CREATE NEW QUESTION (COMMENT)
The Create New Question dialog box appears. This incorporates the text field, the checkboxes and the type of question. The ‘Note’ field is ancillary.
Text
- The required ‘Text’ field is where the question/task/description/call-to-action goes. In this case we enter ‘Comments to Reviewers.’
Checkboxes
- Required: Check this if the question is required to be answered by the Meta-Reviewer.
- Locked for Edit: If the Meta-Reviewer gave a response, yet more Meta-Reviewing is to be done, this will lock the question so it cannot be edited by the Meta-Reviewer.
- Visible to authors during author feedback: Authors will be able to view Meta-Reviewers’ answers to this question in the Author Feedback phase.
- Visible to authors after author notification: Enabling this will show the Meta-Reviewers’ answers to this question after the Authors are notified of the decision.
- Visible to reviewers: This makes this question/answer visible to Reviewers of the paper.
- Visible to meta-reviewers: This enables Meta-Reviewers to see the Meta-Reviewer’s answer to this question.
- Visible to senior meta-reviewers: This enables Senior Meta-Reviewers to see the Reviewer's answer to this question.
Note
If all the checkboxes (except 'Required') are deselected, then only the Chairs can view the answers. This can be useful for confidential comments from the Meta-Reviewer to be seen by only the Chairs.
Type
- Since this is a comment the Meta-Reviewer would need to type in, click the dropdown next to ‘Type’ and select ‘Comment.’
- Click the ‘Maximum Length’ dropdown to set the maximum length (in characters) of the Meta-Reviewer’s comment. Then click ‘Save changes.’
META-REVIEW QUESTIONS PAGE
It is now saved on the ‘Review Questions’ page.
CREATE NEW QUESTION (AGREEMENT)
Text
- From the ‘Create New Question’ dialog, enter an agreement-type text. For example, we entered ‘While performing my duties as a meta-reviewer, I have and will continue to abide by the xxx code of conduct.’
Note
The checkboxes were covered in the previous section.
Type
- Select ‘Agreement’ from the ‘Type’ dropdown and click ‘Save changes.’
META-REVIEW QUESTIONS PAGE
It is now saved on the ‘Review Questions’ page.
CREATE NEW QUESTION (OPTIONS)
Text
- From the ‘Create New Question’ dialog, enter an option-type text. For example, we entered ‘Do you agree with the Reviewer's assessment?’
Type
- Select ‘Options’ from the ‘Type’ dropdown.
Show as
- Click the ‘Show as’ dropdown and there will be 4 choices:
- List with single choice (radio button)
- List with multiple choice (checkbox list)
- Drop down list with single choice (drop down box)
- List box with multiple choice (use Ctrl/Shift to select multiple items) In this case we will use the ‘List with single choice (radio button)’ selection
Choices
- Click ‘+Add New Choice.’
- In the text box, enter the first-choice answer. In this case, ‘Yes.’
- Click ‘+Add New Choice’ again. Then enter the second-choice answer. In this case, ‘No.’ Click ‘Save changes.’
META-REVIEW QUESTIONS PAGE
It is now saved on the ‘Review Questions’ page.
CREATE NEW QUESTION (OPTIONS WITH VALUE)
This is perhaps the most important question type. From the ‘Create New Question’ dialog, enter an option-type text. Each answer will have a value.
For example, there will be 5 answers to this question: ‘Overall score’.
Each answer will have a unique value so the Chairs can make a more informed decision on the paper. The answers and values will be:
- Strong Accept 4
- Accept 3
- Weak Accept 2
- Weak Reject 1
- Reject 0
Text
- Enter ‘Overall score’ in the text box.
Type
- In the ‘Type’ dropdown select ‘Options with value.’
Show as
- For this example we do not want multiple answers for this question. Select ‘List with a single choice (radio button list).
Choices
- Click ‘+Add New Choice.’
- Enter ‘Strong Accept’ and change the value to ‘4.’
- Click ‘+Add New Choice’ 4 more times.
- Enter the text and values for each as outlined above. Click ‘Save changes.’
META-REVIEW QUESTIONS PAGE
It is now saved on the ‘Meta-Review Questions’ page.
- Click the ‘Preview’ button.
META-REVIEW QUESTIONS PREVIEW
The ‘Meta-Review Questions Preview’ page is what the Meta-Reviewers will see when they go over the Reviews of their assigned papers.
Note
Everything in blue italics are the results of the enabled checkboxes in each question.
EDIT QUESTION
Chairs may find the need to edit a question. In this example we will edit Question #3 to allow visibility to only the Chair. No one else will be able to view this question/answer.
- From the ‘Review Questions page, click on the edit icon for Question #3.
Edit Question Page
- The ‘Edit Question’ page appears. Deselect the checkboxes except for ‘Required.’ Then click ‘Save changes.’
META-REVIEW QUESTIONS PAGE
- Click the ‘Preview’ button.
META-REVIEW QUESTIONS PREVIEW
- Verify there is no visibility to anyone but the Chair (by virtue of no blue italics).
COPY QUESTIONS TO ANOTHER TRACK
Note
In a Multi-Track Conference, the Chairs can copy questions from one track to another making it a much less time-consuming task.
Caution
If the target track already has questions, copying questions from the source track to that track will not override those pre-existing questions. The new questions will be added to the target track Form after the last pre-existing question.
In this case we will copy the questions from the 'MAIN' track to the 'Supp' track.
The 'Supp' track is showing no questions.
- After the questions for the 'MAIN' track have been created, click 'Copy questions to other track.'
- In the 'Copy Questions to Other Track' pop-up, click the 'Supp' track and then click the 'Copy' button once. To highlight more, simply [CTRL] click on the tracks desired.
Important
Do not click 'Copy' more than once or the questions will be added to the new track again, thus doubling them.
A green alert with 'Questions copied!' will appear at the bottom of the page.
- Go to the 'Supp' track and verify the questions appear.
CHANGING QUESTIONS
Once the Meta-Review phase has started we strongly recommend no changes are made to Meta-Reviewer questions.
However, if aggregate columns are wanted, the Meta-Reviewer questions can be changed to 'Options with Values' if the original question type is 'Options.'
Changing ‘Options’ to ‘Options with Values’ = OK. Any other changes in question tpye will not work and remove whatever the meta-review is for that question.