Background--With the recent implementation of the Medicare Quality Payment Program, providers face increasing accountability for delivering high-quality care. Such pay-for-performance programs aim to leverage systematic data captured by electronic health record (EHR) systems to measure performance; however, the fidelity of EHR query for assessing performance has not been validated compared with manual chart review. We sought to determine whether our institution's methodology of EHR query could accurately identify cases in which providers failed to prescribe statins for eligible patients with coronary artery disease. Methods and Results--A total of 9459 patients with coronary artery disease were seen at least twice at the Emory Clinic between July 2014 and June 2015, of whom 1338 (14.1%, 95% confidence interval 13.5-14.9%) had no statin prescription or exemption per EHR query. A total of 120 patient cases were randomly selected and reviewed by 2 physicians for further adjudication. Of the 120 cases initially classified as statin prescription failures, only 21 (17.5%; 95% confidence interval, 11.7-25.3%) represented true failure following physician review. Conclusions--Sole reliance on EHR data query to measure quality metrics may lead to significant errors in assessing provider performance. Institutions should be cognizant of these potential sources of error, provide support to medical providers, and form collaborative data management teams to promote and improve meaningful use of EHRs. We propose actionable steps to improve the accuracy of EHR data query that require hypothesis testing and prospective validation in future studies.