Reviewer guidelines

Technical Guidelines

Scheduling

As soon as you are notified of your assigned paper, please check all of them to make sure that:

• there is no obvious conflict of interest (see section on Conflict of Interest below)
• the number of assigned papers and the deadline allow you enough time to complete your reviews (see section on Commitment and Respect below)
• you are qualified to review the papers assigned
• none of the papers assigned to you violates any of the paper submission guidelines

Please notify the Program Committee immediately if any issues arise regarding these points. Adhering to the deadline is essential, since the Program Committee and Chairs still have a huge amount of work to do after the review process has been completed. It is advisable to read papers well in advance before the deadline in order to have time to think about them over a sufficiently long timespan before writing your reviews. This is essential in order for you to make thoughtful decisions about your assigned papers, and to provide helpful suggestions for the authors.

Evaluation Criteria

You will be asked to provide an evaluation of the papers assigned to you according to the following criteria:

• Scholarly/scientific quality
• Novelty of the paper
• Reusable insights
• Pioneering proposals
• Readability and paper organization
• Potential to generate discourse
• Relevance of the topic to ISMIR

Keep in mind that minor flaws can be corrected, and should not be a reason to reject a paper. However, accepted papers must be technically sound and make an original and substantial contribution to the field. Also, please familiarize yourself with the information in the Call for Papers regarding the topics of the conference.

When deciding your recommendation for a paper, do not be shy. Use the whole spectrum of evaluation scores: if you think a paper is outstanding, give it the highest score and no less than that. Similarly, if you think a paper is really bad (and can convincingly support your opinion), then give it the lowest score and no more than that. Very often reviewers tend to use intermediate scores, because they are not entirely confident on their judgment, and/or because they did not have time to read their papers thoroughly. This attitude causes really good papers not to stand out, and very bad papers not to be “caught”.

Ensure that your scores are consistent with your comments to the authors. In particular, receiving good comments and a poor score is frustrating, and often causes the authors to request clarifications or rebuttals.

Comments for the Authors

Your comments for the authors are probably the most important part of your reviews. They will be returned to the authors, so you should include any specific feedback which can help improve the papers. Thorough comments also help the Program Committee decide which papers to accept, sometimes more than your score. Remember that your reviews are evaluated by the Program Committee. Moreover, after the completion of the review process, they will be available to other reviewers of the same papers. Therefore your good work will help generating a positive trend in the research community. Short reviews are not helpful to both the authors and the Program Committee. Please be as specific and detailed as you can. When discussing related work and references, simply saying “this is well known” or “this has been common practice for years” is not appropriate. You should cite publications, or other public disclosures of techniques, which can support your statements. Be specific also when you suggest improvements in the writing. If there is a particular passage in the text that is unclear, point it out and give suggestions for improvements. On the other hand, if the whole paper is poorly written, you are not expected to rewrite it for the authors.

Be generous about providing new ideas for improvement. You may suggest different techniques or tools to be used in the applications presented in a paper. You may also suggest the authors a new application area that might benefit from their work. You may suggest them a generalization of their concept, which they have not considered. If you think that the paper has merits but does not exactly match the topics of the ISMIR conference, please do not simply reject the paper but communicate this to the Program Committee. These papers will then be looked at and discussed with specific care. Suggestions for alternative publication options that are more appropriate in your opinion (journals, conferences, workshops) are welcome.

ISMIR Review Examples

You can check out Tom Collins’ ISMIR Review Examples notes for some helpful advice and review samples.

Ethical Guidelines

Commitment and Respect

Remember that academic careers and reputations rely on scientific publications. Therefore you have to be seriously committed to your work as a reviewer. In the past there have been complaints about some reviews being too sketchy and superficial, so that it looked like the reviewer did not take the time to read the paper carefully. A sketchy or casual review is a lack of respect to the authors who have seriously submitted their paper, and in the long run damages the ISMIR conference. If you have agreed to review a paper, you should devote enough time to write a thoughtful and detailed review. If you think you cannot review properly your assigned papers because you are too busy, you should not commit to your assignment. However, please do communicate this to the Program Committee as soon as possible so that the papers can be reassigned in time. Acting that way you are helping much more than doing a superficial review.

Keep in mind that belittling or sarcastic comments are not appropriate. Even if you think that a paper is really bad, you should be constructive and still provide feedback to the authors. If you give a paper a low score, it is essential that you justify the reason for that score in detail. Just saying “I do not like this approach because I am a guru in this area” is not constructive. Also keep in mind that directly talking about the authors can be sometimes perceived as being confrontational, even though you do not mean it this way. For this reason, you may want to avoid referring to the authors by using the phrase “you” or “the authors”, and use instead “the paper”.

Maintaining Double-Blind Review Process

In order to maintain the legitimacy for our double-blind review process, we discourage authors from posting near duplicate manuscripts on public archives (technical reports, arXiv, etc.). If you detect the identity of the authors because of a preprint of the work on arXiv or elsewhere or because of activities on social media, blogs, mailing-lists, etc., please immediately inform your meta-reviewer. The Program Chairs will try to find another reviewer (hoping this new reviewer will not find the preprint as well), but due to the time constraint of the reviewing process this might not be possible and hence the paper would not have time to be reviewed according to the double-blind policy of ISMIR. In the same spirit, to protect our double-blind reviewing process, we ask the reviewers to not actively trying to find out about the identity of authors.

Confidentiality and Anonymity

As a reviewer you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. Submissions to the ISMIR conference have not (or should not have) been published before. Although the authors’ ultimate goal is to publish and disseminate their work, a percentage of submitted papers will not be accepted and published in the ISMIR conference proceedings, and will most likely be submitted to some other journal or conference. Sometimes a submitted paper is still considered confidential by the author’s employers or funding sources. In order to comply with confidentiality requirements:

• you should not show your assigned papers (or their accompanying material) to anyone else, including colleagues or students, unless you have asked them to help with your review.
• you should not use ideas from your assigned papers to develop new ones until the paper has been made public
• after completing your reviews you should keep all copies of your assigned papers and accompanying material strictly confidential; also you should not use implementations you may have written and results you may have obtained to evaluate the ideas in the papers until they have been published and are properly citable.

Although some reviewers like to disclose their identity to authors, it is advisable not to do so. One of the most common ways of inadvertently disclosing your identity is asking the authors to cite your past work and several of your own papers. This should be avoided. Besides, this attitude may have a negative effect on your review: it may be seen as if you just want to gain more citations, and may ultimately result in the authors just ignoring your review (and possibly the Program Committee too).

Conflicts of Interest

Even though you would judge impartially any paper assigned to you, there has to be no doubt about the impartiality of your reviews. Therefore, if there is a potential conflict of interest with one of your assigned papers, you should inform the Program Committee. Although in general you should use your judgment, examples of situations of potential conflicts of interest are the following:

• you work in the same research group as one of the authors;
• you have been involved in the work and will be credited in some way (e.g. you have hosted one of the authors in your lab, to carry out work related to the paper);
• you have formally collaborated (e.g., written a paper together, or been awarded a joint grant) with one of the authors in the past three years (more or less);
• you were the M.Sc./Ph.D. advisor (or advisee) of one of the authors: this is often considered to be a lifetime conflict of interest;
• you have reasons to believe that others might see a conflict of interest, even though there is none (e.g., you and one of the authors work for the same multinational corporation, although you work in different departments on different continents and have never met before).

In case you have any doubt about a potential conflict of interest, then rather decide that there is such a conflict and contact the Program Committee.

Acknowledgements

This document closely follows the reviewer guidelines of the ISMIR 2018 and 2019 conferences.