| JOURNAL HOME | CME HOME | THIS MONTH | PAST ISSUES | ETOC | COLLECTIONS |
| AUTHORS | REVIEWERS | EDITORIAL BOARD | FEEDBACK | RSS | HELP |
 
|
|
|
||
|
|
Anesth Analg 2009; 108:941-947 © 2009 International Anesthesia Research Society doi: 10.1213/ane.0b013e3181949ae6
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
For example, consider the event of tracheal extubation of a patient in an operating room (OR). Suppose that a DSS query is executed every 10 min and sends a message to an anesthesia technician after documentation of extubation that the OR needs to be turned over. If the 95th percentile of the time from extubation to generation of the DSS notification is 12 min, then there will be a delay of more than 12 min in 1 of every 20 cases. If such a latency of 12 min were managerially undesirable and could be reduced to 7 min by executing the DSS query every 5 min, then the query frequency should be increased.
Our experiences during the development and implementation of DSS indicate that OR managers become keenly aware of delays in the updating of screens at OR control desks. The expectations of OR managers and anesthesia providers using the DSS must be leveled to the expected performance of the system, otherwise trust may be diminished in the recommendations provided by such systems.7,8 Failure to consider latency may result in poor management and/or inadequate clinical control.9
In this study, we measured several sources of latency for two prototype events (Surgery Begin and Surgery End) that anesthesia providers typically document (Table 1). These two events are used extensively by the DSS currently running at Hospital A. We also determined the impact of latency on the timing of decisions made by the DSS (i.e., message generation). Our analysis included the impact on decisions of 1) tardy documentation, 2) delays due to the scheduled frequency of the DSS queries, and 3) documentation that anticipates the occurrence of future events (i.e., with an entered time later than the current time). We compared results with a different hospital (Hospital B) to evaluate whether latencies are similar. We used the findings from Hospital B to explore the impact of 4) absent documentation.
METHODS
De-identified data from the AIMS at Hospital A were analyzed after a determination by the IRB that "the study does not qualify as human subjects research." The IRB of Hospital B approved the retrospective review of documentation practices, including event latency.
The data from Hospital A comprised 48,010 cases from January 1, 2006 to July 15, 2007. At the start of the study interval, Hospital A’s AIMS had been fully implemented in the ORs for approximately 2
months. The 2733 cases (5.7%) for which providers documented more than one Surgery Begin or Surgery End event were excluded due to computational complexities in following the audit trail to calculate latency. For example, providers sometimes incorrectly documented the case-specific Surgery Begin event for both the supine and prone portions of an anterior posterior cervical fusion. Because the sample sizes were so large, eliminating these cases had no effect on the latency analysis.
The AIMS records the time that documentation was written to the database (Documentation Time) and the time the event occurred provided by the user (Event Time). The Documentation Time was obtained from the local workstation clock, synchronized to a network time server. The user’s event button defaults to the workstation system clock, but providers can modify this time before it is written to the event table in the AIMS database. If the time of an event is subsequently edited, the prior Documentation Time and Event Time recorded in the database Event table are copied to the Event audit table and replaced by the new values. These various times are shown graphically in Figures 1a and 1b. We determined for Hospital A that the 95th percentile time from when the provider presses a button to enter an event until that event was stored in the database used by the DSS was 57 s (median = 30 s).
|
the event of interest occurred, and at time 
the next DSS query executes. The provider has not yet documented the event in the anesthesia information management system (AIMS), so the DSS does not know that it has already taken place. At time 
the provider documents the, backdating it to the actual time 
the next scheduled query executes, at which time the DSS finally learns that the event has occurred. Event Latency is calculated as the interval from the time of the actual event 
, the next scheduled query executes, at which time the DSS learns that the event has occurred (although the time is wrong). After the initial documentation, at time 
, the provider corrects the event time We define a DSS query to be an interrogation of the AIMS database using structured query language that potentially results in a communication (e.g., a text message to a pager, a popup message on an AIMS workstation screen, or an update to a whiteboard). For the study, DSS queries were considered to execute every 1, 5, or 10 min, with the times synchronized to the hour (e.g., a query running every 5 min executes at 2:00 pm, 2:05 pm, 2:10 pm, and so forth). The DSS query time was therefore calculated as the next scheduled query after the first Documentation Time in that interval. Determination of the first Documentation Time required examination of both the database Event table and the Event audit table, in case the event had been edited (Fig. 1).
We defined event latency as the difference between 1) the time when a DSS query would have resulted in the first message being sent and 2) the final time when the event was recorded in the database as having occurred. The effect of q 1 min, q 5 min, and q 10 min scheduled DSS query frequencies on latency was calculated.
We considered two events (Surgery Begin and Surgery End) as prototypes for events used by DSS queries and measured their latencies. Latencies were adjusted downward if a smaller value would have been inferred from documentation of a subsequent event, as follows. The maximum value for the Surgery Begin latency was calculated using the Documentation Time of the Surgery End event if a user forgot to document Surgery Begin. The rationale for our procedure was that the DSS would infer that Surgery Begin had already occurred once Surgery End was noted. The maximum value for the Surgery End latency was calculated using the documentation time of the "Patient leaves the OR" event, for similar reasons. If the provider forgot to document the time when the patient left the OR, then the time that the case was closed in the OR was used instead because that time was applied automatically by the AIMS.
Users at Hospital A sometimes indicated a future time for events, either accidentally or deliberately, resulting in a negative value for the event latency if the DSS executed before the anticipated time. For example, the time that the patient left the OR might be entered a few minutes in the future, as records are usually closed and printed prior to transporting the patient from the OR table to the stretcher. We truncated the minimum value for event latency at –10 min, since a well designed DSS should discount events entered far into the future as being unreliable. In the Results, we show that our selection of this minimum value of –10 min had no effect on our results, and that this interval was a reasonable choice for the AIMS.
Although all percentiles reported in the paper for Hospital A were calculated to the nearest 1 s, they are reported to the nearest 1 min, for the following reason. We had expected that an examination of the seconds part of the Surgery Begin and Surgery End event times would reveal that these values were uniformly distributed between :00 and :59 s. Instead, 63.9% of the Surgery Begin and 69.0% of the Surgery End times occurred at even minute values (e.g., 9:01:00 or 10:17:00), not the 1.7% (1/60) rate expected for an event that occurred randomly during each 1 second interval of a given minute. The remainder of the seconds part of the Surgery Begin and Surgery End times were distributed uniformly between :01 and :59 s. This phenomenon was caused by short-cut buttons in the AIMS that allowed users to adjust times in 1 min increments (after first truncating the seconds to 0) in place of manually adjusting the minutes and seconds.
The data from Hospital B comprised 40,902 cases performed from October 1, 2006 through October 1, 2007. Only one Surgery Begin and Surgery End time was analyzed for each case. At the start of the study interval, the AIMS had been in use for 1 mo, but before installing this AIMS, clinicians at Hospital B had been using an AIMS from a different vendor for several years.
For every percentile of the latency reported in the paper, 95% Clopper–Pearson (conservative) confidence intervals10,11 were <2 min, and most were <1 min. Thus, the upper and lower bounds were not presented as separate columns in the tables. Descriptive data are presented as the mean ± sd.
RESULTS
Average Latencies Were Approximately Half of the Query Interval
For events occurring randomly within each query interval, the latency would be expected to equal half the query interval, provided times were entered using the current clock time and were not adjusted manually. The observed average latencies were indeed close to half the query interval.
The median Surgery Begin latencies for the 5 min and 10 min query intervals were approximately 1 min longer than half the query interval. The median Surgery End latency was 1 min shorter than half the 10 min query interval (Fig. 2, Table 2) The reason for the slight differences was not specification of the Surgery End time in the future (i.e., resulting in negative latency), because that only occurred in 0.04% of cases. Rather, the result appeared because the providers adjusted the Surgery Begin event time backwards when they documented the event to a greater degree than they did for the Surgery End event time. This increased the interval between the documented time of Surgery Begin and the first query interval after such documentation occurred. The effect was not discernible for either event with the 1 min query interval because the effect was obscured by the rounding issue described in the Methods and by the average 30 s delay from when a provider clicked an event button until the data appeared in the database.
|
|
Longest 5% of Latencies Exceeded the Query Interval
Since event times are random within each query interval, if clock times were applied as above, the 95th percentile of latency would be expected to be 95% of the query interval.
The 95th percentile for the latency of the Surgery End event was equal to the 10 min query interval (Table 2). In contrast, the 95th percentile for the 1 min query interval was 5 min and for the 5 min query interval was 7 min.
The 95th percentile for the latency of the Surgery Begin event was longer than the query interval for all 3 query intervals studied: 5 min for the 1 min queries, 7 min for the 5 min queries and 15 min for the 10 min queries.
The explanation for these findings is that the anesthesia providers often documented the Surgery Begin event several minutes after the event had occurred, as opposed to the Surgery End event, for which the documentation was more contemporaneous. The implication of this variant behavior is that the latency of documentation of an event to be used by a DSS query needs to be considered when designing an appropriate query interval. If an event is typically documented at a time distant from the actual occurrence, there is little to be gained by querying more frequently for its occurrence. However, for decisions in which documentation is essentially immediate, a more frequent query interval may be useful.
Providers Rarely Edited the Times of the Begin or End Surgery Events
The DSS presents a recommendation based on the initial event time recorded. Edits to the Surgery Begin and Surgery End event only occurred in 0.69% and 0.68% of cases, respectively. The corresponding 95% confidence intervals are 0.62% to 0.77% and 0.61% to 0.75%, respectively. Therefore, the original DSS recommendation would almost always have been the same as that inferred from examining the final event time (i.e., the one displayed on the printed anesthesia record). The studied percentiles of latency (Table 2) are unchanged, and our results are insensitive to how we measured latency.
Differences in the Median Latencies Between the Hospitals A and B Were Within a Few Minutes, But Differences for High Percentiles (
90%) Were Large
Data from Hospital B with a different AIMS were compared to Hospital A to explore the impact of documentation practices on latency. We had expected since Hospitals A and B are large academic institutions with scores of anesthesia providers, that documentation practices would be similar, resulting in similar percentiles of latencies.
The median latencies for the Surgery Begin and Surgery End events were longer by a few minutes at Hospital B than at Hospital A.
However, for high (90% and 95%) percentiles, the differences were large because of greater delayed (tardy) user entry at Hospital B versus A. For example, for the Surgery Begin and 10 min query intervals, the 95th percentile event latencies were 30 min at Hospital B, versus 15 min at Hospital A (Table 3).
|
Furthermore, we found unexpectedly that anesthesia providers at Hospital B more frequently omitted the documentation of the Surgery Begin and Surgery End events compared with Hospital A (11% vs 1%, and 28% vs 7%, respectively).* An AIMS DSS relying on Surgery End would know that surgery has ended, even if not documented, at the time that the patient leaves the OR.
We repeated the analysis incorporating the 28% (95% CI: 28%–29%) rate of absent user entry for Surgery End.
The under-estimates of the true 95th percentiles of the event latencies were 24 min for 5 min query intervals and 28 min for 10 min query intervals. The corresponding lower 95% confidence intervals for these percentiles were the same.
DISCUSSION
Our group continues to focus on development of DSSs to improve the performance of OR managers in moving cases, scheduling add-on cases, and assigning staff relief on the day of surgery.7 The methodology described applies not only to our specific interests, but also to any DSS in which queries are based on the documentation of events in a database. We studied latency in the entry of two representative events that are important to OR management DSSs: Surgery Begin and Surgery End. The methodology can be applied to the latency of any clinician-entered event in the AIMS database.
The issues of latency raised in this study also apply to situations in which data are only being displayed without generating automated recommendations to facilitate decision-making, such as for an OR "whiteboard."12 As another example, if automated pages are sent to anesthesiologists who are medically directing residents or certified registered nurse anesthetists when their next patient arrives in the holding area, delays by the ward clerk in documenting the patient’s actual arrival time needs to be considered. Measuring the latency of the "Patient Arrives in the Holding Area" event would provide a metric through which the clerk’s attentiveness to this task could be measured.
We found that a single query interval is not suitable, such that the appropriate choice will need to be made for each decision being supported by a DSS. Analysis of event latency will suggest a query interval which results in acceptable performance of the query (e.g., notification is sent within 10 min of the event 95% of the time). Arbitrarily using a brief query interval (e.g., 1 min) for all queries may affect AIMS performance if some involve complex, central processing unit-intensive tasks or require prolonged reads of data from the database, typical of a DSS.13 A discussion of the methodology to analyze the impact of implementing a DSS on AIMS performance is beyond the scope of this manuscript. However, queries that execute quickly (i.e., within seconds) generally will not impact AIMS performance, whereas those that take long periods of time (i.e., measured in minutes) should be optimized to increase their efficiency, if possible, or an impact analysis performed.
Each message sent to a provider represents an interruption. Sending too many messages too frequently or during periods of high workload may have unintended consequences on patient care.14–16 Depending on implementation policy, running queries at frequent intervals may result in more messages, and thus more interruptions.
In this study, we focused on latency based on the final recorded event time as an index of DSS performance and trust. We think our approach is valid for several reasons. First, providers rarely changed the time of the Surgery Begin or End event once entered in the database and, even when they did so, only made small adjustments. Second, the timeliness, as reflected by a small latency of the DSS recommendation, should be based on when the event actually occurred, not a prior value present in the Event audit table. We think it reasonable to assume, under most circumstances, when the time of an event is changed, that the times will be adjusted closer to when the event actually occurred. However, in real-time, the DSS recommendation would have used the initially recorded event time.
Other AIMS are likely to function somewhat differently than ours, and users of those systems may document either more or less contemporaneously than at the two hospitals studied. The difference in results between Hospitals A and B highlights this caveat. Readers should not extrapolate directly from our results to a proposed decision support process. Rather, we strongly recommend that DSS developers determine the latency for the events they plan to use, following the principles that we have described. In addition, they need to develop strategies to deal with the possibility that the events targeted for use by the DSS may be absent or entered multiple times.
Footnotes
*These events were more consistently documented in the OR information system by circulating nurses, but that database was not accessible in real time. 
To incorporate the missing values for the Surgery End event into the statistical analysis, we took advantage of two observations for Hospital B. There was a positive association between 1) difference of the time of patient exit and the listed time of Surgery End and 2) the OR time (Spearman r = 0.48, 95% CI: 0.48–0.49). In addition, there was a positive association between 1) missing value for Surgery End and 2) the OR time (Spearman r = 0.38, 95% CI: 0.37–0.39). The median difference of the time of patient exit and the listed time of Surgery End was 20 min. Therefore, for each record missing a value for Surgery End, we calculated the event latency based on Surgery End having taken place 20 min before. This approach deliberately under-estimated the actual 95th percentile of absolute latency for the DSS at Hospital B, both because of the positive associations and the inherent assumption that the patient exit time was always documented at the time of that event, and never later.
Accepted for publication September 15, 2008.
Franklin Dexter is editor of Economic, Education, and policy for the journal. This manuscript was handled by Steven L. Shafer, Editor-in-Chief, and Dr. Dexter was not involved in any editorial decisions related to this manuscript.
REFERENCES
- Kheterpal S, Gupta R, Blum JM, Tremper KK, O’Reilly M, Kazanjian PE. Electronic reminders improve procedure documentation compliance and professional fee reimbursement. Anesth Analg 2007;104:592–7
[Abstract/Free Full Text] - Spring SF, Sandberg WS, Anupama S, Walsh JL, Driscoll WD, Raines DE. Automated documentation error detection and notification improves anesthesia billing performance. Anesthesiology 2007;106:157–63[Web of Science][Medline]
- Sandberg WS, Sandberg EH, Seim AR, Anupama S, Ehrenfeld JM, Spring SF, Walsh L. Real-time checking of electronic anesthesia records for documentation errors and automatically text messaging clinicians improves quality of documentation. Anesth Analg 2008;106:192–201
[Abstract/Free Full Text] - Wax DB, Beilin Y, Levin M, Chadha N, Krol M, Reich DL. The effect of an interactive visual reminder in an anesthesia information management system on timeliness of prophylactic antibiotic administration. Anesth Analg 2007;104:1462–66
[Abstract/Free Full Text] - Reich DL, Kahn RA, Wax D, Palvia T, Galati M, Krol M. Development of a module for point-of-care charge capture and submission using an anesthesia information management system. Anesthesiology 2006;105:179–86[Web of Science][Medline]
- St. Jacques P, Sanders N, Patel N, Talbot TR, Deshpande JK, Higgins M. Improving timely surgical antibiotic prophylaxis redosing administration using computerized record prompts. Surg Infect 2005;6:215–21
- Dexter F, Willemsen-Dunlap A, Lee JD. Operating room managerial decision-making on the day of surgery with and without computer recommendations and status displays. Anesth Analg 2007;105:419–29
[Abstract/Free Full Text] - Lee JD, Moray N. Trust, self-confidence, and operators’ adaptation to automation. Int J Hum Comput Stud 1994;40:153–84[Web of Science]
- Dexter F, Macario A, Traub RD. Enterprise-wide patient scheduling information systems to coordinate surgical clinic and operating room scheduling can impair operating room efficiency. Anesth Analg 2000;91:617–26
[Free Full Text] - Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 1934;26:404–13
[Free Full Text] - Hahn GJ, Meeker WQ. Statistical intervals. A guide for practitioners. New York: Wiley, 1991:82–4, 100–5
- Xiao Y, Dexter F, Hu P, Dutton RP. Usage of distributed displays of operating room video when real-time occupancy status was available. Anesth Analg 2008;106:554–60
[Abstract/Free Full Text] - Null L, Lobur J. The essentials of computer organization and architecture. Boston: Jones & Bartlett, 2006:564–8
- Healey AN, Primus CP, Koutantji M. Quantifying distraction and interruption in urological surgery. Qual Saf Health Care 2007;16:135–9
[Abstract/Free Full Text] - Sevdalis N, Healey AN, Vincent CA. Distracting communications in the operating theatre. J Eval Clin Pract 2007;13:390–4[Web of Science][Medline]
- Healey AN, Sevdalis N, Vincent CA. Measuring intra-operative interference from distraction and interruption observed in the operating theatre. Ergonomics 2006;49:589–604[Medline]
This article has been cited by other articles:
|
|
 |
|
  F. Dexter, E. O. Bayman, and R. H. Epstein Statistical Modeling of Average and Variability of Time to Extubation for Meta-Analysis Comparing Desflurane to Sevoflurane Anesth. Analg., February 1, 2010; 110(2): 570 - 580. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. Dexter, R. H. Epstein, J. D. Lee, and J. Ledolter Automatic Updating of Times Remaining in Surgical Cases Using Bayesian Analysis of Historical Case Duration Data and "Instant Messaging" Updates from Anesthesia Providers Anesth. Analg., March 1, 2009; 108(3): 929 - 940. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
