This is a tough and thankless job, but science depends on it! Here are a few principles I keep in mind when suggesting (to journals that ask for them) or soliciting (when I'm an editor) peer reviewers.
When identifying reviewers for a particular paper, I try to find a balance of:
Where to find reviewers?
(Along those lines- if you’re publishing make sure you’re giving back to the community by serving as a peer reviewer and/or editor yourself! Read my guide to writing a solid peer review or how to get started, and register as a potential reviewer with journals in your field).
See guidelines for picking reviewers: https://methodsblog.wordpress.com/2015/10/15/preferred-reviewers/
For the PNAS guidelines see here: http://m.pnas.org/site/authors/coi.xhtml
Springer, Conflict of Interest: http://www.springer.com/authors/manuscript+guidelines?SGWID=0-40162-6-795522-0
Article on COI in medicine: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2246405/