Working Group on Blood Pressure Monitoring of the European Society of Hypertension International Protocol for validation of blood pressure measuring devices in adults.

Abstract

Introduction With the increasing marketing of automated and semi–automated devices for the measurement of blood pressure, potential purchasers need to be able to satisfy themselves that such devices have been evaluated according to agreed criteria [1]. With this in mind, the Association for the Advancement of Medical Instrumentation (AAMI) published a standard for electronic or aneroid sphygmomanometers in 1987 [2] that included a protocol for evaluating the accuracy of devices, this being followed in 1990 by the protocol of the British Hypertension Society (BHS) [3]. Both of these were revised in 1993 [4,5]. These protocols, which differed in detail, had a common objective, namely the standardization of validation procedures to establish minimum standards of accuracy and performance, and to facilitate the comparison of one device with another [6]. Since their introduction, a large number of blood pressure measuring devices have been evaluated according to one or both protocols. Experience has, however, demonstrated that the conditions demanded by the protocols are extremely difficult to fulfil. This is especially so because of the large number of subjects who have to be recruited and the ranges of blood pressure required. The time required to complete a validation study is such that it is difficult to recruit trained staff for the duration of an investigation. These factors have made validation studies difficult to perform and very costly, with the result that fewer centers are prepared to undertake them. This is particularly unfortunate as more devices than ever before are in need of independent validation. When the BHS dissolved its Working Party on Blood Pressure Measurement, the Working Group on Blood Pressure Monitoring of the European Society of Hypertension (ESH) undertook to produce an updated protocol, which it has named the International Protocol. The ESH Working Group on Blood Pressure Monitoring is composed of experts in blood pressure measurement, many of whom have considerable experience in validating blood pressure measuring devices. In setting about its objective, the ESH Working Group recognized the urgent imperative to provide a simplified protocol that does not sacrifice the integrity of the earlier protocols. When the AAMI and BHS protocols [2–5] were published, the relevant committees did not have evidence from previous studies on which to base their recommendations. The ESH Working Group has had the advantage of being able to examine and analyze the data from 19 validation studies performed according to the AAMI and BHS protocols at the Blood Pressure Unit in Dublin [7–23]. A critical assessment of this database of evidence has permitted a rationalization and simplification of validation procedures without losing the merits of the much more complicated earlier protocols. The basic recommendations of the simplified International Protocol have been presented at meetings of the ESH Working Group, and the proceedings of these meetings have been published in order to invite comment and discussion [24–27]. The International Protocol has been drafted in such a way as to be applicable to the majority of blood pressure measuring devices on the market. The validation procedure is therefore confined to adults over the age of 30 years (as these will constitute the majority of subjects with hypertension), and does not make recommendations for special groups, such as children, pregnant women and the elderly, or for special circumstances, for example exercise. It is anticipated that the relative ease of performance of the International Protocol will encourage manufacturers to submit blood pressure measuring devices for validation in order to obtain the minimum approval necessary for a device to be used in clinical medicine, and that, in time, most devices on the market will be assessed according to the protocol for basic accuracy. It does not preclude manufacturers of devices from subjecting their products to more rigorous assessment and validation. Validation procedure Summary The validation team should consist of four persons experienced in blood pressure measurement: two observers and a supervisor (generally nurses), and an ‘expert’ (a doctor overseeing the entire procedure). If the doctor can be present throughout the entire validation procedure, he/she can take over the role of supervisor, thereby reducing the number of personnel to three. The validation procedure consists of the following steps Observer training and assessment. Two observers are trained in accurate blood pressure measurement. Familiarization session. The validation team becomes familiar with the workings of the device and the accompanying software. Validation measurements. Observer and device measurements are recorded on subjects in two phases. In the first phase, 15 subjects are recruited; devices passing this primary phase proceed to the secondary phase, for which a further 18 subjects are recruited. Analysis. An analysis of the recorded measurements is carried out after each phase. Reporting. The results are presented in tabular and graphical forms. Observer training and assessment Consideration must first be given to the technique of blood pressure measurement, which should be as follows throughout the validation procedure. Blood pressure measurement technique A standard mercury sphygmomanometer, the components of which have been carefully checked before the study, is used as a reference standard. It is appreciated that terminal digit preference is a problem with conventional mercury sphygmomanometry, and care should be taken to reduce this in the observer training session. The Hawksley random-zero sphygmomanometer only disguises digit preference, and its accuracy has been questioned [7,28]; therefore, its use is not recommended in validation studies. All blood pressures should be recorded to the nearest 2 mmHg. Blood pressure should be measured with the arm supported at heart level [29]; the level of the manometer does not affect the accuracy of measurement, but it should be at eye level and within 1 m of the observer. The quality of the stethoscope is also crucial to performing the evaluation procedure. Stethoscopes with badly fitting earpieces and poor-quality diaphragms preclude precise auscultation of the Korotkoff sounds. A well-maintained quality stethoscope is recommended. Observer training The first prerequisite for this validation test is to ensure that the observers have adequate auditory and visual acuity, and that they have achieved the required accuracy as laid out below. It is, however, possible that observers who fulfil these criteria at the outset of the study will not do so at the end, and if this happens the observers must be re-assessed for accuracy. To avoid this, analysis should be performed as the study proceeds to detect any drift in agreement between the observers. Observers may be trained in the following ways: By fulfilling the test requirements of the CD-ROMs produced by the BHS or the Société Française d'Hypertension Artérielle as described in Appendix B [30,31]. By formal training and assessment is described in Appendix B [32,33]. By using an audio-visual method for validation, such as the Sphygmocorder [34,35] as discussed in Appendix B. Familiarization session As automated devices for blood pressure measurement may be complex, it is important that the personnel performing a validation study are fully conversant with the equipment. The observers, having satisfied the training criteria, should next be instructed in the use of the device to be validated and any accompanying computer software. For uncomplicated devices designed to provide a straightforward blood pressure measurement, the familiarization session should consist of performing a series of practice measurements on volunteers. A more formal session should, however, be applied to complex devices such as systems for measuring 24-h blood pressure. This session has two functions: first, it serves as a familiarization period for the personnel performing the validation study, and second, any technical peculiarities of the device being tested, which might influence the validation procedure, may be identified. Validation measurements General considerations Device validation should be performed at room temperature without disturbing influences such as telephones and bleeps in the area. Some automated devices have more than one method of measuring blood pressure. It may be claimed for a particular device, for example, that electrocardiogram gating may be used when more accurate measurement is required. In these circumstances, validation must be performed with and without electrocardiogram gating. Similarly, some Korotkoff sound-detecting devices provide an oscillometric back-up when sound detection fails. When both systems generate simultaneous readings, only one comparative validation is required, but when the oscillometric method is a back-up to the auscultatory method and provides a separate measurement, both systems of measurement must undergo individual validation. Arm circumference and bladder dimensions The circumference of the arms should be measured to ensure that the bladder being used is adequate for the subject. Measurements made with the test device should use the appropriate bladder according to the manufacturers' instructions. Standard mercury manometer measurements must be taken with a bladder of sufficient length to encircle 80% of the arm circumference [29]. If a test device recommends different cuff sizes, the appropriate cuff/bladder should be used, but no other part of the apparatus should be changed. It is important to ensure, when assessing auscultatory devices, that the same microphone(s) are used throughout the validation test. Devices for measuring blood pressure at the wrist The International Protocol may be used to validate devices that measure blood pressure at the wrist. There is little literature regarding the accuracy of devices for wrist measurement, and most studies have shown these devices to be inaccurate [1]. Measurements of blood pressure at the wrist using oscillometric devices generally overestimate blood pressure compared with conventional sphygmomanometry on the upper arm, and the differences can be substantial [36–38]. It must, however, be emphasized that although a device designed for measuring blood pressure at the wrist may be accurate when tested according to the International Protocol, it may be inaccurate for the self-measurement of blood pressure if the instructions to have the wrist at heart level during measurement are not strictly followed. Devices for self-measurement that measure blood pressure at the finger are not recommended because vasoconstriction of the digital arteries can introduce substantial errors. Subject selection In selecting 33 subjects (15 for the phase 1, and a further 18 for phase 2) with a wide range of blood pressure it is probable that there will be a representative range of arm circumference, and subjects should not be selected on the basis of their arm circumference. Subjects may be taking antihypertensive medication but must not present in atrial fibrillation or any sustained arrhythmia. Number Phase 1 Fifteen subjects Phase 2 Thirty-three subjects Sex Phase 1 At least five male and five female Phase 2 At least 10 male and 10 female Age range All subjects should be at least 30 years of age Arm circumference Distribution by chance Blood pressure range As in Table 1Table 1: Blood pressure ranges for entry blood pressure (BPA) There are three ranges for systolic (SBP) and three for diastolic (DBP) blood pressure, with 11 subjects in each range to provide 99 pairs of measurements. To optimize recruitment, it is recommended that subjects for the high-diastolic and low-systolic groups should be recruited first. The emphasis should then be placed on filling the remaining high-systolic and low-diastolic groups. Finally, the remaining gaps in the middle groups should be filled. The blood pressure used in this categorization is the entry blood pressure at the time of the validation procedure (BPA), rather than that at the time of recruitment for validation. Observer measurement Measurements can be either assessed live using two observers or recorded and later re-assessed using the Sphygmocorder [34,35]. Measurements made simultaneously by two observers must be checked by the validation supervisor. If the systolic and diastolic measurements are no more than 4 mmHg apart, the mean value of the two observer measurements for both systolic and diastolic blood pressures is used. Otherwise, the measurement must be taken again. When the Sphygmocorder is used, two observers should assess the recording separately. If their opinion differs, they should re-assess the values together until agreement is reached. Further references to ‘observer measurement’ indicate either the mean of two observer measurements or the agreed measurement using the Sphygmocorder. At least 30 s should be allowed between each measurement to avoid venous congestion, but not more than 60 s or variability may be increased. Procedure The subject is introduced to the observers, and the procedure is explained. Arm circumference, sex, date of birth and current date and time are noted. The subject is then asked to relax for 10–15 min (in order to minimize anxiety and any white-coat effect, which will increase variability). Nine sequential same-arm measurements using the test instrument and a standard mercury sphygmomanometer are recorded as follows: BPA Entry blood pressure, observers 1 and 2 each with the mercury standard. The mean values are used to categorize the subject into a low, medium or high range separately for SBP and DBP (Table 1). BPB Device detection blood pressure, observer 3. This blood pressure is measured to allow the test instrument to determine the blood pressure characteristics of the subject; more than one attempt may be needed with some devices; this measurement is not included in the analysis. If the device fails to record a measurement after three attempts, the subject is excused. BP1 Observers 1 and 2 with the mercury standard. BP2 Supervisor with the test instrument. BP3 Observers 1 and 2 with the mercury standard. BP4 Supervisor with the test instrument. BP5 Observers 1 and 2 with the mercury standard. BP6 Supervisor with the test instrument. BP7 Observers 1 and 2 with the mercury standard. Documentation must be provided for data omitted for legitimate technical reasons. Once a subject has been included, the data for that subject should not be excluded from the study if blood pressure values are obtainable; if blood pressure measurements using either the reference method or the test instrument are unavailable, data entry for that individual may be excluded, with an accompanying explanation. Additional individuals must then enter into the study to ensure a sample size of 33. Analysis For a detailed discussion on the statistical methods used in the protocol, see Appendix D. A software program has been designed specifically to analyze the data (Société Française d'Hypertension Artérielle, Paris). Accuracy criteria The BHS protocol introduced the concept of classifying the differences between test and control measurements according to whether these lay within 5, 10 or 15 mmHg, or were over 15 mmHg apart. The final grading was based on the number of differences falling into these categories. This protocol seeks to keep this concept but expand its flexibility. Differences are always calculated by subtracting the observer measurement from the device measurement. When comparing and categorizing differences, their absolute values are used. A difference is categorized into one of four bands according to its rounded absolute value for SBP and DBP: 0–5 mmHg These represent measurements considered to be very accurate (no error of clinical relevance). 6–10 mmHg These represent measurements considered to be slightly inaccurate. 11–15 mmHg These represent measurements considered to be moderately inaccurate. >15 mmHg These represent measurements considered to be very inaccurate. The analysis is based on how values in these bands fall cumulatively into three zones: Within 5 mmHg This zone represents all values falling in the 0–5 mmHg band. Within 10 mmHg This zone represents all values falling in the 0–5 and 6–10 mmHg bands. Within 15 mmHg This zone represents all values falling in the 0–5, 6–10 and 11–15 mmHg bands. Subject measurements For assessment of accuracy, only measurements BP1 to BP7 are used. The mean of each pair of observer measurements is calculated; this is denoted as observer measurement BP1, BP3, BP5 or BP7. Each device measurement is flanked by two of these observer measurements, and one of these must be selected as the comparative measurement. From these, further measurements are derived as follows. The differences BP2 – BP1, BP2 – BP3, BP4 – BP3, BP4 – BP5, BP6 – BP5 and BP6 – BP7 are calculated. The absolute values of the differences are calculated (i.e. the signs are ignored). These are paired according to the device reading. If the values in a pair are unequal, the observer measurement corresponding to the smaller difference is used. If the values in a pair are equal, the first of the two observer measurements is used. When this has been completed, there are three device readings for SBP and three for DBP for each subject. Each of these six readings has a single corresponding observer measurement, a difference between the two and a band for that difference as described above. Experience with existing protocols has demonstrated that the overall outcome of a device can be apparent from a very early stage. This is particularly so with poor devices and is in accordance with statistical expectancy – the larger the error, the smaller the sample size required to prove it. To persist with the validation of a device that is clearly going to fail is an unnecessary waste of time and money, and an inconvenience to participating subjects. A mechanism for eliminating poor devices at an appropriate stage is therefore introduced by dividing the validation process into two phases. In the primary phase, three pairs of measurements are performed on 15 subjects in the pressure ranges given in Table 1, any device failing this phase (Table 2a) being eliminated from further testing. Devices passing this proceed to a secondary phase in which a further 18 subjects (giving a total of 33) are recruited (Table 2b).Table 2a: Requirements to pass phase 1Table 2b: Requirements to pass phase 2.1Assessment of phase 1 Once there are five subjects in each of the six blood pressure ranges (Table 1), recruitment should be stopped and an assessment performed. Data from only the first five subjects in each range are used. (In filling these ranges, some ranges may be over-subscribed because of subjects having different SBP and DBP ranges.) This will yield 45 sets of measurements for both SBP and DBP. The number of differences in each zone is calculated as described above. A continue/fail grade is determined according to Table 2a (see also Table 3 for an example). If the device fails, the validation is complete; if the device passes, it proceeds to phase 2. Table 3: Example of device validation table in reportAssessment of phase 2 This phase determines how accurate the device will be for individual measurements (Phase 2.1) and for individual subjects (Phase 2.2) by determining the number of differences within 5, 10, and 15 mmHg, and then determining the accuracy. After all ranges have been filled, there will be 99 sets of measurements for both SBP and DBP. The number of differences in each zone as described above is calculated. A pass/fail grade for phase 2.1 is determined according to Table 2b (see Table 3 for example). For each of the 33 subjects, the number of measurements falling within 5 mmHg is determined. A pass/fail recommendation for phase 2.2 is determined according to Table 2c (see Table 3 for example). If the device passes both phase 2.1 and phase 2.2, it passes the validation and can be recommended for clinical use. If it does not, it fails and is not recommended for clinical use. Table 2c: Requirements to pass phase 2.2Reporting Statistical report The report should be prefaced with subject data in order to describe the key characteristics of the subjects in the study. An example of a device validation is shown in Table 3. Sex distribution. The number of males and females. Age distribution. The mean, standard deviation and range of the subjects' ages. Arm circumference distribution. The mean, standard deviation and range of the subjects' arm circumferences and, when different cuff sizes are used, the number of subjects on which each size was used. Blood pressure. The mean, standard deviation and range of the subjects' entry SBP and DBP (BPA). The report should then give the results of the validation. Phase 1 The number of differences falling in the Within 5 mmHg, Within 10 mmHg and Within 15 mmHg zones (Table 2a), together with the requirements, should be reported in text and tabular form as in Table 3. The basis on which the decision to continue or stop at this stage should be stated. Phase 2 The number of differences falling in the Within 5 mmHg, Within 10 mmHg and Within 15 mmHg zones (Table 2b), together with the requirements, should be reported in text and tabular form as in Table 3. The number of subjects with all three differences, at least two differences and no differences falling within 5 mmHg (Table 2c) should be reported in text and tabular form as in Table 3. The mean and standard deviation of the observer and device measurements and the differences should be stated. The basis on which the decision to pass or fail the device should be stated. Graphical representation Difference-against-mean plots should be presented for the data at the phase at which the study ceased. Phase 1 data should be plotted for devices failing at that stage, and phase 2 data for those passing. The x-axis of these plots represents blood pressures in the systolic range 80–190 mmHg and the diastolic range 30–140 mmHg, and the y-axis values from −30 to +30 mmHg. Horizontal reference lines are drawn at 5 mmHg intervals from +15 to –15 mmHg. The mean of each device pressure and its corresponding observer pressure is plotted against their difference using a point. Differences greater than 30 mmHg are plotted at 30 mmHg. Differences less than –30 mmHg are plotted at –30 mmHg. The same scales should be used for both SBP and DBP plots. An example is shown in Fig. 1 [39].Fig. 1: Devices passing and failing phase 2.1 The x–axis represents the mean of the device and observer measurements. Both systolic blood pressure (upper plot) and diastolic blood pressure (lower plot) ranges should be plotted on the same scale. Recruitment limits are indicated by the vertical lines. The y–axis represents the difference between the device and observer measurements. The 5 mmHg bands from +15 to –15 mmHg are indicated by the horizontal hatched lines. The 99 comparisons are presented in a difference-against-mean scatterplot. In this example, the systolic blood pressure plot depicts a poor device whereas the diastolic blood pressure plot depicts an accurate one.Problems Any problems encountered during the validation procedure, the date of their occurrence, the date of any repairs to the device and the effect of these on the validation procedure should be recorded. Operational report The following information should be provided with blood pressure measuring devices, and the final report should acknowledge that such information is available, and although this need not be presented in detail, any deficiencies should be listed in the report. Basic information The information provided in operational manuals is often deficient. Without appropriate specifications and operational instructions, it is difficult to obtain optimal performance. List of components All major components of the system should be listed. The dimensions of the bladders supplied and those of the range of bladders available should be indicated. Method(s) of blood pressure measurement The basic method of pressure detection (e.g. auscultatory or oscillometric) should be stated, and if more than one method is used, the indications for changing methods and the means of denoting this on the recording should be stated. With Korotkoff sound-detecting devices, whether phase IV or phase V is being used for the diastolic end-point must be disclosed. If data are derived from recorded measurements, such as mean pressure, the method of calculation must be stated. Factors affecting accuracy Many factors, such as arm movement, exercise, arm position, cuff or cloth friction may affect the accuracy of automated recordings. All such factors should be listed by the manufacturer. Operator training requirements Some automated systems require considerable expertise on the part of the operator if accurate measurements are to be obtained, whereas other systems require relatively little instruction. These requirements should be stated. Computer analysis Some automated systems are compatible with personal computer systems. The exact requirements for linking with computer systems and their approximate cost should be stated. If the automated system is dependent on its own computer for plotting and analysis, this should be made clear, and the cost of the computer facility, if it is an optional extra, should be stated. Clear instructions should be provided for setting recording conditions (e.g. frequency of recordings during defined periods and the on/off condition of the digital display); retrieving recordings and saving data to disk; retrieving data from disk; displaying numerical data and graphics; exporting data to statistical, graphic and spreadsheet software programs; and printing the results (partial or complete). If data cannot be exported, information on how they are stored should be available to facilitate the external analysis of several monitoring events. The manufacturer should list compatible computers (PC or other) and printers together with memory requirements, operating systems, compatible graphic adaptors and additional software or hardware requirements (including interfaces and cables if these are not supplied). Acknowledgements The report should state whether the equipment was purchased for the evaluation or donated or loaned by the manufacturer. The data analysis should ideally be carried out by the laboratory doing the evaluation. If it has been done by the manufacturers, this should be stated. Any consultancies or conflict of interest should be acknowledged by the investigator.