http://orcid.org/0000-0003-3611-8740
Figures
Abstract
The organisational context of healthcare settings has an essential role in how research evidence is used in clinical practice. The Alberta Context Tool (ACT) measures 10 concepts of organisational context with higher scores indicating a more positive work environment and potentially better use of research evidence in patient care. We assessed the psychometric properties of the ACT in Emergency Departments (EDs). This validation study was conducted as part of a multi-centre trial of triage, treatment and transfer (T3 Trial) of patients with stroke admitted to EDs. Stratified sampling with proportional allocation was used to recruit ED nurses from 26 participating hospitals at baseline. Nurses completed a survey containing the ACT. Structural validity was investigated by exploratory and confirmatory factor analysis. Reliability was assessed using Cronbach’s alpha and intraclass correlation coefficients. Item-rest correlations and the average inter-item correlations were also assessed. 558 ED nurses completed the survey, comprised of 433 surveys without missing data. Our exploratory factor analysis produced a 14-factor structure, explaining 62% of variance of organisational context. For eight of ten concepts, item loadings matched the factor structure of the original ACT. Confirmatory factor analysis of the 10 ACT concepts showed moderate model fit (p = 0.001, root mean square error of approximation: 0.049, standardised root mean squared residual: 0.048). Cronbach’s alphas showed very good internal consistency for nine of ten ACT concepts (α>0.7; 0.45–0.90). Item-rest correlations indicated that most ACT items (50 of 56 items) within any concept related well to the total score of the concept. Average inter-item correlations indicated potential redundant items for three concepts (feedback processes, leadership, staffing) that were above the threshold of 0.5. While identifying a few shortcomings for some ACT concepts in an ED context, the majority of findings confirm reliability and validity of the original ACT in an Australian population of ED nurses.
Citation: Schadewaldt V, McElduff B, D’Este C, McInnes E, Dale S, Gunaratne A, et al. (2019) Validating the Alberta Context Tool in a multi-site Australian Emergency Department nurse population. PLoS ONE 14(4): e0215153. https://doi.org/10.1371/journal.pone.0215153
Editor: Liza Heslop, Victoria University, AUSTRALIA
Received: October 7, 2018; Accepted: March 27, 2019; Published: April 9, 2019
Copyright: © 2019 Schadewaldt et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The conditions of ethics approval do not allow the authors to distribute or make available the data from this study directly to other parties. These restrictions were imposed by the respective ethics committees because the data contain potentially identifying and sensitive participant information. Applications for data access can be made by contacting the Nursing Research Institute at t3.nri@acu.edu.au.
Funding: The T3 trial was funded by a National Health and Medical Research Council (NHMRC) Project Grant 1024812 (2012–2017). DAC was supported by a fellowship from the National Health and Medical Research Council (NHMRC; 1063761 co-funded by Heart Foundation). Non-material support provided by Trial sponsor, the Australian Catholic University, to house members of the trial team including authors SM, SD, EM, BM, AG and VS. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: I have read the journal's policy and the authors of this manuscript have the following competing interests: During the duration of the T3 trial and following receipt of funding, SM was appointed to the Research Committee of the National Health and Medical Research Council. Other authors declare that they have no competing interests. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Implementing research findings into clinical practice is crucial to provide high quality care. However, researchers and clinicians struggle to achieve and sustain the translational process from evidence into practice despite available implementation strategies [1–3]. Implementation research has identified context as influential in terms of health professionals’ research utilisation [4–7] and the implementation and dissemination process of research-based knowledge in a clinical setting [8]. ‘Context’ refers to a number of human factors such as leadership, staff interaction and staffing levels and also structural factors such as workplace culture, available resources and physical work space [9]. In this paper we refer to context as ‘organisational context’ of a health care setting comprising all of the above factors. Evidence is growing that supports that the use of research to inform practice is greatest in those settings with positive perceptions by staff of their workplace context [10–12]. Conducting an assessment of organisational context can identify those factors that may impact upon whether or not evidence-based resources such as clinical practice guidelines will be used by clinical staff [11]. Staff are then enabled to address the modifiable factors of organisational context, thus helping to provide the optimum environment for the introduction of evidence based practice [13].
The Alberta Context Tool (ACT) was developed to assess the organisational context of healthcare settings and research utilisation behaviour of health professionals [14]. It measures those aspects of the organisational context that have an impact on how well clinicians may use research in clinical practice [14]. The ACT has been broadly used in research projects to assess organisational context in acute and long-term care facilities as follows: 1) to identify and confirm associations between context and research use or adherence to best practice by clinical staff, which showed that a positive organisational context fosters research use [10–12, 15–18]; 2) to identify associations between context and patient outcomes, which showed improved health outcomes for nursing home residents in facilities with higher context scores [19]; 3) to identify associations between context and health care delivery, which showed staff in nursing homes with lower context scores reported more often being under pressure when delivering care [20]; and 4) to identify associations between context and staff characteristics, which showed that there was no relationship between organisational context and years of clinical practice [21, 22].
To our knowledge, there is no research that has reported the application of the ACT by health professionals working in the Emergency Department (ED). One study protocol has been published, stating the planned application of the ACT in paediatric EDs but results have not been published as of January 2018 [23]. The majority of studies have been undertaken in nursing homes and acute paediatric wards that differ from the high acuity work environment of an ED. ED staff are under constant pressure to make quick decisions, address a multitude of diseases and injuries whilst working with various disciplines [24, 25]. Limited research using the ACT has been undertaken in Australia, including its application in general medical wards [22] and stroke units [26, 27]. Therefore, we undertook a validation study of the ACT in an ED nurse population to determine the psychometric properties of the ACT within this setting and clinician group.
Methods
Design and setting
A cross-sectional survey was used for this validation study that was part of a larger implementation project, the T3 Trial, a cluster randomised controlled trial undertaken to evaluate the impact of a multidisciplinary intervention to improve treatment, triage and transfer of patients with stroke in the ED on 90-day patient outcomes [28]. The study was undertaken in 26 hospitals across three Australian states and one territory. The survey was undertaken prior to commencement of any pre-implementation data collection, as part of a process evaluation to help explain how the organisational context may have contributed to the subsequent success or failure of uptake of the T3 intervention.
Ethics approval
Approval from the Human Ethics Research Committees from the following sites have been obtained: Australian Catholic University (2012 16N), Sydney Local Health (Royal Prince Alfred Hospital Zone) (HREC/12/RPAH/32) (NSW Lead Ethics Committee); ACT Government Health Directorate Human Research Ethics Committee (HREC) (ETH.1.13.009); Uniting Care Health HREC (1231); The Alfred, Melbourne HREC (503/12); Northeast Health Wangaratta HREC; and South West Healthcare Warrnambool HREC (5/2012).
Recruitment, sample and data collection
Nurses on the ED staff roster at participating hospitals were eligible to participate, including night shift staff. Casual staff and agency staff were excluded. The nurse unit manager (NUM) of each ED was contacted initially to identify the number of nurses working in their ED, number of full-time and part-time nurses and number of enrolled and registered nurses. Based on the total number of nursing staff, EDs were classified as small (≤ 50 nurses) or large (> 50 nurses). At small EDs, 40% of nurses were sampled, resulting in a projected sample size of approximately 80 nurses from five small EDs. At larger EDs, 20% of the nurses were sampled, resulting in approximately 500 nurse participants with a total projected sample of approximately 580 nurses. We used stratified random sampling with proportional allocation to obtain a sample representative of the number of nurses qualified as enrolled nurses (EN) or registered nurses (RN), and working full-time or part-time within each ED.
Each NUM was provided with a set of surveys, a set of replacement surveys, an administration protocol, and a list which specified the total number of surveys required from the hospital and how many surveys were to be distributed to each type of nurse (EN/RN; full-time/part-time). The NUM or another delegated ED staff member was responsible for distribution of the paper-based self-administered survey. They self-selected eligible staff within each nurse category. The administration protocol stated that if a staff member declined to fill out a survey, the person distributing the surveys was to give the survey to a nurse with the same category (e.g. RN full-time replaced with RN full-time). Furthermore, if a survey was not returned within three weeks, a replacement survey was to be provided to another nurse within the relevant category.
An information sheet explained the purpose of the study and the survey. Surveys could be returned to the site contact in a sealed envelope or returned directly via reply-paid envelope to the T3 research office. The return of the survey implied consent to participate. The site contact gave verbal reminders to complete the survey at seven and fourteen days following the survey distribution.
Data collection instrument
The ACT nurse version [29] was used for this study following permission from the tool developers. The ACT was constructed originally with two foci in mind. First, in order to promote its use in clinical settings, the tool needed to be quick to complete [14]. Secondly, the tool had to assess modifiable elements of context in order to enhance the implementation and uptake of research evidence in practice [14]. The development of the ACT was based on the theoretical construct of context as outlined in the Promoting Action on Research Implementation in Health Services (PARIHS) framework [7] and other literature reporting contextual factors that play a role in research utilisation by health care providers. The PARHIS framework is a theoretical model to describe and assess how research evidence is implemented in practice and the PARHIS context domain comprises concepts of culture, leadership and evaluation [7]. These concepts were expanded by the tool developers to 10 concepts of organisational context: (1) leadership, (2) culture, (3) feedback processes (evaluation), (4) connections (social capital), (5) formal interactions, (6) informal interactions, (7) structural and electronic resources, (8) staffing, (9) time and (10) space, with these latter three sub-concepts of organisational slack [14, 30]. A number of items were developed to measure each concept. An exploratory factor analysis (EFA) during the tool development phase resulted in a 13-factor structure that explained 59.3% of variance of organisational context [14] with variances around 60% and above commonly accepted as satisfactory [31]. Eight of ten concepts are represented by a single factor, with the other two concepts comprising more than one factor (resources comprise three and informal interactions two factors). Following pilot-testing and refinement in samples of nurses and other health professionals, the final tool contained 56 items. Reliability testing resulted in Cronbach’s alpha at or above the conventional standard of 0.7 for 9 factors and showed acceptable values of > 0.54 and < 0.7 for 4 factors [14].
All ACT items are rated on five-point Likert scales. Response options for leadership, culture, feedback processes, connections, staffing and space items are strongly disagree, disagree, neither agree nor disagree, agree, strongly agree while items assessing formal and informal interactions, structural/electronic resources and time are classified as never, rarely, occasionally, frequently and almost always. Higher scores indicate a more positive perception of the work context. The ACT is not designed to produce a single overall score of context and the tool developers advise analysing each concept as an individual scale [32].
Our survey also obtained demographic information on age, gender, time since qualified, length of time working in ED, professional position, level of education, specialisation in emergency nursing or critical care nursing, employment status (full-time, part-time) and working rotating shifts.
Statistical analysis
The statistical analysis was carried out with STATA 14 [33] and R statistical software [34]. In accordance with scoring instructions, the ACT items from three concepts (formal interactions, informal interactions, structural/electronic resources) were recoded to 0 for never/rarely, to 0.5 for occasionally and to 1 for frequently/almost always and an overall sum for each concept was calculated [32]. The remaining concepts kept the item scoring of 1 to 5 and a mean score was calculated for each concept. For ACT concepts, items with more than 10% of observations with missing data, or with more than 90% of observations in a single category were reviewed and considered as potentially unnecessary or irrelevant items in this population [35]. Sociodemographic characteristics of the sample are reported as number and percentages for categorical variables and means and standard deviations for continuous variables. Cases with missing data were excluded from the analysis.
Structural validity
Since the ACT has not been used in a sample of ED nurses, we undertook an exploratory factor analysis to investigate the factor structure of the instrument without imposing any specific framework. Principal component analysis (PCA) was performed followed by varimax rotation. In order to follow the original ACT validation study, we set the eigenvalue cut-off score as ≥1.0 for factors to be retained in the model and factor loading coefficients of ≥ 0.35 for items to be retained in the factor [14].
A confirmatory factor analysis (CFA) was also performed to verify the ACT’s original structure in our ED nurse population. We modelled the data to the ACT factor structure within a structural equation modelling framework and used three measures to assess model fit: the root mean square error of approximation (RMSEA); the standardised root mean squared residual (SRMSR); and the comparative fit index (CFI). A RMSEA of < 0.07, an SRMSR < 0.8 and a CFI > 0.95 indicate good fit [36]. Additionally, we report the p-value of the model χ2 test, which tests for difference between the sample and fitted covariance matrices. This test is sensitive to sample sizes > 200 and thus likely to reject the null hypothesis, that the model fits exactly. Three of the ACT concepts (informal interactions, formal interactions and structural/electronic resources) were developed as a comprehensive list of potential interactions staff may have had, or available resources that they utilised. Rather than representing true scales with conceptually related items, the informal/formal interaction and resources concepts included lists of independent items assessing various ways of interaction between staff and use of different resources [15]. Concept scores for these three concepts were calculated as the sum or count of the number of items within each concept that respondents participated in or used. Since a factor analysis groups conceptually related items under a factor, it was unlikely that the factor analysis would show an acceptable model fit. ([15, 37]). In order to identify the impact of the ‘count-based’ or non-scaled concepts on the model fit, we tested three models with i) all 10 ACT concepts, ii) seven concepts based on true scales and iii) three concepts comprising the count-based concepts.
Internal reliability
Cronbach’s alpha was used to assess internal consistency of the factors from the 10 concepts identified in the original study. Reliability measures and their cut off scores for good scale quality are summarised in Table 1. We also assessed the change in Cronbach’s alpha associated with removal of each item from the concept to determine if the reliability of the scale would improve by deleting individual scale items. Values of > 0.70 have been cited as acceptable, whereas values of > 0.80 are preferable [38, 39].
Internal consistency was also assessed using item-rest correlations, i.e. correlations between one scale item and the remaining items of the concept. It is assumed that items contributing to the total of the scale correlate positively with the overall scale. In reference to the original ACT development, a correlation coefficient of > 0.3 is considered acceptable for items to remain in the scale [14].
In addition, inter-item correlations were assessed, and coefficients between 0.15 and 0.5 were considered as acceptable [40]. Correlations below 0.15 may indicate that items do not relate well to the concept they are measuring and coefficients above 0.5 may indicate that items are redundant because they are measuring the same aspect of the concept [40]. For specific constructs such as the 10 concepts of organisational context, coefficients of 0.4–0.5 are preferable [40].
Since the ACT is completed by individual health professionals but is commonly used as a measure of organisational context of an entire unit (ED) in the analysis, it is important to establish that individual-level data from health professionals reflect unit-level context. The intraclass correlation coefficient (ICC) is an estimate of agreement of individuals’ scoring about the mean score within a unit [41]. One-way random effects ANOVA, defined as ICC (1,1) by Shrout and Fleiss (1979) was used to determine if responses from individuals could be aggregated to unit-level, i.e. if high interrater agreement exists within the respective EDs [41]. ICC (1,1) values range from 0 (no agreement) to 1 (perfect agreement), however, in applied research an ICC (1,1) ≥ 0.10 is indicative of perceptual agreement among individuals within the same unit [15]
Results
Sample characteristics
Overall 558 completed surveys were returned. Five EDs provided more than their specified quota of surveys, 14 provided fewer and seven provided the correct quota. The total number of surveys distributed, the number of nurses approached to complete a survey and the number of nurses who declined to complete a survey are unknown, therefore we are unable to provide a response rate for the study. The demographic characteristics of the sample are shown in Table 2. The majority of respondents (n = 413, 74%) were female with almost half of the participants aged between 25 and 34 years (n = 253, 45%). The mean number of years nurses worked in the ED was 6.1, ranging from 0.1–38 years. A third of the nurses were educated at Bachelor’s degree level (n = 186, 33%) with 39% of the nurses (n = 217) holding a postgraduate certificate or diploma degree. More than half (n = 325, 58%) of the nurses worked in the role of a registered nurse with other roles listed in Table 2.
Between 10% and 12% of observations had missing data for demographic information and 55 (9.9%) nurses did not complete any information on the demographic part of the survey. For the ACT, there were 433 nurse surveys with no missing data. Missing data for individual ACT items was minimal, ranging from 4 (0.7%) to 20 (3.6%). Response patterns showed no single category with more than 90% of observations.
Structural validity—Factor analyses
Our exploratory factor analysis produced a 14-factor structure, explaining 62% of variance associated with organisational context (Table 3). S1 Appendix highlights those loadings (blue cells) that match the original item loadings. With the exception of formal interactions and informal interactions, the items of the remaining 8 ACT concepts loaded onto their corresponding factor (S1 Appendix).
Factor loadings.
For the first four factors all items loaded on their corresponding concepts, which are leadership, feedback processes, connections and culture (S1 Appendix). Factor 4 comprises all items of culture with one item (being a member of a supportive work group) cross-loading onto leadership. However, this item has a low factor loading of 0.36 for the leadership factor. The combined variance for the first four factors explaining organisational context is 26% (Table 3). All items of the three organisational slack concepts–time, staffing, space–loaded on their relevant factors. One item of the concept space (adequate space to provide patient care) cross-loaded onto staffing with a factor loading of 0.51.
The four items of formal interactions loaded onto three factors with one item (participation in team meetings) not loading onto any of the retained factors at a loading of ≥ 0.35. Informal interactions were represented in three factors, one reflecting interactions with other clinicians, one reflecting interactions with professionals involved in research and one comprising two items, one of which (occurrence of informal bedside teaching) cross-loaded onto structural resources. This two-item factor was not part of the original ACT structure. Structural and electronic resources were represented in three factors, reflecting three sub-types of resources: 1) formal resources–noticeboards, policies and clinical practice guidelines; 2) external and electronic resources–workshops, computerised decision support, email reminders and websites; and 3) traditional resources–library, textbook and journals.
Confirmatory factor analysis results.
Model fit statistics from the confirmatory factor analysis are presented for all three models in Table 4. The low p-values for all three models (p<0.001) indicate that the data do not fit the predefined models exactly. The RMSEA and SRMSR show acceptable fit for model one with all 10 ACT concepts and model two with the seven true scale concepts. Model three based on the three count-based concepts shows unacceptable fit with a RMSEA of 0.087 and a borderline acceptable fit with a SRMSR of 0.078. The CFI for all three models lies below the conservative cutoff of >0.95, however, model two is close to good model fit with a CFI of 0.924.
Reliability
Internal reliability.
Cronbach’s Alphas for 9 of 10 ACT concepts were above 0.7, showing very good internal consistency (Table 5). An exception was the concept titled formal interactions that had an alpha of 0.45, which indicates weak internal consistency.
Item-total statistics.
Cronbach’s Alpha did not increase if a particular item within the scale was deleted for 9 of 10 concepts, hence none of the items seemed redundant. The exception was for the organisational slack concept space. Cronbach’s alpha of 0.75 would increase to 0.87 if the item titled we have adequate space to provide patient care was deleted.
Item-rest correlations.
In accordance with the original ACT validation, we chose an item-rest correlation coefficient of > 0.3 as indication for a reliable measure. Item-rest correlations indicate that most items (50 of 56 items) within a concept related well to the total score of the concept. Seven of the 10 ACT concepts had item-rest correlation coefficients > 0.3 for all items with the majority of items having coefficients of > 0.4 (Table 5). The concepts named feedback, leadership and staffing indicated the highest item-concept correlations with maximum coefficients ranging from 0.59 (leadership: leader focuses on success) to 0.78 (staffing: both items).
Six of 56 items had item-rest correlations below the cut-off score of 0.3. Informal interactions (occurrence of hallway talk) and structural/electronic resources (library) had one item each with values of 0.23 and 0.27, respectively. The concept of formal interactions had all four items scoring below the cut-off score (0.20–0.29), indicating weak correlations between the concept formal interactions and its items.
Inter-item correlations.
The average inter-item correlation or item-to-item correlation ranged from 0.17 (formal interactions) - 0.78 (staffing) (see Table 5). Only three concepts (connections, time and space) lie within the preferred range of 0.4–0.5. One concept (culture) is just below the threshold of 0.4 whereas three concepts (leadership, feedback processes and staffing) lie above the cut-off coefficient of 0.5 for good scale measurement, indicating that items within these concepts measure the concept in very similar ways and could potentially be redundant. None of the concepts had average inter-item correlations under the cut-off of 0.15, indicating that all items related well to the concept.
Intraclass correlation coefficient.
The ICC (1,1) shows some degree of agreement among nurses for all domains with strong agreement (ICC (1,1) ≥ 0.10) for six out of ten domains (Table 5). Therefore, nurse responses within one ED were very similar and can be aggregated to unit-level responses to reflect ED organisational context.
Discussion
This validation study in a large sample of 558 ED nurses from 26 Australian hospitals confirms the structural validity and reliability of the ACT to assess organisational context within this setting. That means, the ACT can be applied in an ED setting to evaluate the structural and human factors that may impact on the use of research evidence by nurses. Reliability as well as validity measures met the standards for good scale quality for almost all items within the scale. We achieved a high number of returned surveys, further adding to the strength of these findings.
Compared to the paediatric nurse sample (n = 752) of the original study testing the ACT [14], our sample of ED nurses included more males (4% vs 14%), and nurses who had been working for fewer years on the trial unit (7.7 vs 6.1). Our sample also included more nurses with a Master’s degree (1.7% vs 9%). The original factor analysis was based on a sample of 704 paediatric nurses (after deletion of incomplete cases) from six hospitals [14] whereas our sample (with a complete dataset) consisted of 433 nurses from 26 EDs making it one of the largest hospital samples in which the ACT has been used and thus contributing to context assessment of emergency care settings. The results of our validation study corroborate the application of the ACT in a population of ED nurses, which to our knowledge has not been previously investigated.
Validity and reliability
Our exploratory factor analysis produced a 14-factor solution that accounted for a total of 62% of the variance associated with organisational context. For 8 of 10 concepts, item loadings matched the original factor structure. This finding mirrors well the scale structure of the ACT. The first report about the ACT described an exploratory factor analysis resulting in 14 factors based on 51 items and explaining 70% of variance of organisational context [30]. After refinement of the ACT, a later pilot study reported a 13-factor structure based on 56 items and accounting for 59% of variance [14].
In regards to internal reliability, alpha coefficients in our study were at or above the standard of 0.7 for 9 of 10 concepts. Similarly, previous validation studies found Cronbach’s alphas ≥ 0.7 for 7 of 8 concepts (organisational slack comprising time, staff and space) [30], for 9 of 13 concepts (based on the 13-factor structure) [14] and for 8 of 10 concepts in a sample of healthcare aides in nursing homes [15]. Two studies using translated versions of the ACT in non-Canadian populations presented Cronbach’s alphas of ≥ 0.7 for 10 of 13 concepts (based on the 13-factor structure) in a German population of health professionals in nursing homes [42] and for 5 of 8 concepts (organisational slack comprising time, staff and space) in a Swedish population of nursing home nurses [43]. In line with these studies, the ICCs for the ACT concepts showed high agreement among ED nurses within their respective ED, supporting the ACT’s purpose to assess the nurses’ specific organisational context. Most recently, Squires et al. re-analysed the ACT data from five different datasets, collected from their own research in nursing homes and paediatric hospitals but also from other researchers using the tool in acute adult hospitals and community care [37]. These studies further confirm internal reliability of the ACT.
Our findings align well with results from initial ACT validation studies in differing professional groups. In particular, for the concepts feedback processes, leadership, connections, staffing and time, all items load exclusively on one factor, confirming the item structure for these concepts. Apart from one item that is cross-loading from culture at a negligible low loading of 0.38, the six culture items also load onto one factor. The concepts culture, leadership and feedback processes have been key concepts in implementation research since they were used to define context in the PARIHS framework [7]. All of these concepts show a high degree of internal consistency with reliable alpha measures (apart from space) and item-rest correlations of > 0.3, meaning that all items relate well to the overall concept. However, limitations for internal reliability were seen for leadership, feedback processes and staffing with average inter-item correlations above 0.5 that indicate that items within these concepts measure the constructs in very similar ways and thus may capture a rather narrow focus of the construct. Potentially, in an ACT ED nurse version, some of these items could be removed, as they may not add much to the measurement of these concepts.
When considering the three sub-concepts time, space and staff combined as organisational slack, all items match the original item-structure of organisational slack with one item from space (adequate space to provide patient care) cross-loading onto staff. The cross-loading of the space item is not surprising as it fits the staff items on the factor that measures sufficient means (space and staff) to provide patient care. Interestingly, as shown by the reliability assessment, by deleting the adequate space to provide patient care item Cronbach’s alpha for space would increase. This item also has a low item-rest correlation of 0.38, indicating a weak correlation of this item to the overall scale. These findings are in accordance with other ACT validation studies that showed low alpha coefficients [15, 44] and low loadings [44] for the adequate space item. However, it appears appropriate to keep this item in the scale as it is conceptually essential to assess the organisational context regarding space. We speculate that adequate space in the ED may be difficult to capture with the ACT items because EDs have different spatial allocations for resuscitation, assessment or fast track patients and nurses move across various bays within the ED.
Three ACT concepts (resources, formal and informal interactions) were developed to rate the frequency of use of resources or frequency of interactions and their mean concept scores are derived from a sum or count-based method, which is different from those of the other concepts. Their scales represent lists with potentially independent items within one concept rather than representing true scales with conceptually related items [37]. Since a factor analysis examines relationships between items, it is reasonable to expect that items of these concepts do not load on one factor and that items have low item-rest correlations, indicating that items measuring this concept do not relate strongly with each other.
This was particularly evident for formal interactions in our study with three items loading onto three separate factors and one item (participation in team meetings) not loading onto any factor at a coefficient of > 0.35. The item attendance at continuing education outside the hospital loaded onto factor 5 that represented traditional resources such as library, textbook, journals. It can be argued that attending continuing education matches the concept of traditional resources. However, the item participation in family conferences does not seem to be consistent with the content of other items (interactions with research nurses, quality improvement representatives and someone who champions research) on the same factor, representing informal interactions of non-direct care providers (factor 8). Considering the short stay of patients in the ED, nurses in our sample may have found it difficult to relate to questions about patient rounds and family conferences as these may not be part of formal interactions in EDs. In fact, formal interactions such as planned meetings or family conferences may be very limited in EDs and an ACT scale for ED nurses may benefit from fewer items on this concept.
Cronbach’s alpha for the formal interaction concept is also low and all four items correlate below the quality standard of > 0.3 with the rest of the scale. Problematic reliability measures for the concept of formal interactions has been a consistent finding in previous validation studies of the ACT [14, 15, 37, 43]. The tool developers ascribe this to the fact that items for this concept were not developed as a true scale but rather represent a list of items assessing the frequency of staff participating in meetings. Consequently, these items may not necessarily function as one theoretical concept and load onto other factors with a better conceptual match. Items relating to formal interactions were therefore not removed from the tool although a factor analysis has shown that items are not related [15, 37].
The concept of informal interactions includes the frequency of interactions with other health professionals. Our factor analysis aligns with the original study for two sub-types of informal interactions, namely interactions with non-direct care providers (research nurses, quality improvement specialists and research champions) and direct care-providers (nurses, physicians, other healthcare providers and clinical specialists) [14]. Different to the original study [14], items in our factor analysis loaded onto a third factor and included interactions with unspecific personnel in the form of hallway talks or bedside teaching. Our analysis suggests a third sub-type of informal interactions of ED nurses, namely a theme related to where the interaction takes place but not with whom. The item occurrence of hallway talks had a low item-scale correlation (0.23), suggesting that it does not fit well within the overall scale of informal interactions. As mentioned above, this can be explained by the nature of this concept that includes items potentially not associated with each other to represent one common concept [37]. It could also highlight that ED nurses had difficulty rating the frequency of hallway talks because common ED layouts do not comprise hallways as they would be found on standard hospital wards or in nursing homes, where the ACT was originally tested.
Resources is another key concept for context already defined in the PARIHS framework and our factor analysis showed clearly defined sub-types of structural and electronic resources in line with the original validation of the ACT. Estabrooks et al. 2009 [14] determined three sub-types of structural resources: formal resources such as noticeboards, policies and clinical practice guidelines; traditional resources such as journals, textbook and library use; and electronic resources such as computerised decisions support systems, email reminders and websites. It is unclear from the Estabrooks publication onto which factor the item workshops/courses loaded, although this item loads onto the sub-type of electronic resources in our study. Items of formal resources, electronic resources and traditional resources loaded onto the same factors as the original study. Additional items from informal interactions (occurrence of informal bedside teaching) and formal interactions (attending continued education outside the hospital) load onto the traditional resources factor, which differs from the original study. However, these items can be understood as traditional resources of learning and thus fit the “theme” of the factor comprising traditional resources. Keeping in mind that the concept of structural and electronic resources has three sub-concepts, each including potentially independent items, their factor loadings suggest some relations between the items on the three sub-types, supporting the tool developer’s choice of categorising items into sub-types.
Some of the measurement problems highlighted with the exploratory factor analysis resonate in the results of the confirmatory factor analysis. The development of the three concepts informal interactions, formal interactions and structural/electronic resources as lists with independent items within each concept rather than scales with conceptually related items, is reflected in the model fit statistics. The purpose of a CFA is to assess a pre-defined categorisation of conceptually similar items and applying this to the three count-based concepts that comprised non-related items, we expected limited model fit ([15, 37]). This is consistent with the results of previous research in Canada [15, 37] and Germany [42] testing three separate models with i) all 10 concepts, ii) seven concepts including the true scale concepts and iii) three concepts including the count-based concepts. Similar to our CFA, models with the three count-based concepts performed consistently lower than the other 2 models [15, 37, 42]. Across all model-fit statistics the second model comprising true scale concepts performed slightly better than the model including all concepts. Accepting that this is due to the inclusion of the three count-based concepts, which the tool developers purposefully added to the ACT, our study supports the use of the ACT in an ED nurse population. However, the results also show that the ACT may benefit from amendments to individual items and concepts that better reflect the ED environment.
This study has some limitations. While the ACT is available for other health professions besides nurses, we collected perceptions on organisational context from ED nurses only. Since our trial focused on nurse-initiated care, we believe nurses were the most appropriate professional group to assess their work environment for this study. Therefore, validation of the ACT in an Australian ED nurse population refers to the ACT nurse survey only.
We acknowledge that the ACT is a self-report tool and thus responder bias may have been introduced. In a separate analysis (to be published elsewhere) we tested the association of responder and ED characteristics with the rating of context but could not identify significant influences on how individuals rated their work context.
While we used stratified random sampling, nurse unit managers self-selected eligible staff to complete the survey. For practical reasons it was not possible to apply a truly random selection strategy for this study.
The study comprised EDs of four Australian states and territories. In Australia, hospitals are managed at state-level so that EDs with various organisational systems were included in the study. This increases our confidence that findings are applicable to other EDs nationally and potentially internationally.
Conclusions
Findings of our validation study in Australian ED settings provide evidence for the feasibility, validity and reliability of the ACT. While some conceptual shortcomings for individual concepts were detected, these can generally be explained by the tool developers’ choice of concept and item construction. Other limitations may be related to the application of the ACT in an ED environment. The ACT was developed based on data from nurses in nursing homes and non-acute wards and some items to assess context do not relate well to the specific high-acuity work context of EDs. While our validity and reliability analysis predominantly showed good results for the ACT in an ED nurse population, an amended ACT version for the ED environment may yield even better results.
In lieu of another tool to specifically capture ED context and its potential impact on clinicians’ use of research evidence, the ACT can be used by clinicians and researchers who are involved in implementation projects. A strength of the ACT is that high concept scores are indicative of higher use of research in clinical practice. Individual concepts of the ACT can provide feedback about leadership qualities and availability of resources and may be an important tool for clinical staff to highlight and negotiate their conditions of work.
Supporting information
S1 Appendix. Factor analysis item loadings (rotation matrix).
https://doi.org/10.1371/journal.pone.0215153.s001
(DOC)
Acknowledgments
We thank the nurses and nurse unit managers who assisted with survey distribution and completion; and the research assistants’ Dr Oyebola Fasugba, Ms Rosemary Phillips and Ms Larissa Collins.
References
- 1. Middleton S, Levi C, Dale S, Cheung NW, McInnes E, Considine J, et al. The T3 Trial: Triage, Treatment and Transfer of patients with stroke in emergency departments. Int J Stroke. 2017;12(2S):15.
- 2. Salbach NM, Wood-Dauphinee S, Desrosiers J, Eng JJ, Graham ID, Jaglal SB, et al. Facilitated interprofessional implementation of a physical rehabilitation guideline for stroke in inpatient settings: process evaluation of a cluster randomized trial. Implementation Science: IS. 2017;12(1):100–. pmid:28764752.
- 3. Tricco AC, Ashoor HM, Cardoso R, MacDonald H, Cogo E, Kastner M, et al. Sustainability of knowledge translation interventions in healthcare decision-making: a scoping review. Implementation Science: IS. 2016;11:55–. pmid:27097827.
- 4. Cummings GG, Estabrooks CA, Midodzi WK, Wallin L, Hayduk L. Influence of Organizational Characteristics and Context on Research Utilization. Nursing Research Jul/Aug. 2007;56(4):S24–S39.
- 5. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629. pmid:15595944.
- 6. Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implementation Science. 2016;11(1):33. pmid:27013464
- 7. Rycroft-Malone J. The PARIHS framework—a framework for guiding the implementation of evidence-based practice. J Nurs Care Qual. 2004;19(4):297–304. pmid:15535533.
- 8. Estabrooks CA, Wallin L, Milner M. Measuring knowledge utilization in health care. International Journal of Policy Analysis & Evaluation. 2003;1:3–36.
- 9. Peters DH, Adam T, Alonge O, Agyepong IA, Tran N. Republished research: Implementation research: what it is and how to do it Implementation research is a growing but not well understood field of health research that can contribute to more effective public health and clinical policies and programmes. This article provides a broad definition of implementation research and outlines key principles for how to do it. Br J Sports Med. 2014;48(8):731–6. pmid:24659611
- 10. Cummings G, Hutchinson A, Scott S, Norton P, Estabrooks C. The relationship between characteristics of context and research utilization in a pediatric setting. BMC Health Serv Res. 2010;10(1):168. pmid:20565714
- 11. Estabrooks CA, Squires JE, Hayduk L, Morgan D, Cummings GG, Ginsburg L, et al. The Influence of Organizational Context on Best Practice Use by Care Aides in Residential Long-Term Care Settings. J Am Med Dir Assoc. 2015;16(6):537.e1–.e10. http://dx.doi.org/10.1016/j.jamda.2015.03.009.
- 12. Yamada J, Squires JE, Estabrooks CA, Victor C, Stevens B. The role of organizational context in moderating the effect of research use on pain outcomes in hospitalized children: a cross sectional study. BMC Health Serv Res. 2017;17(1):68. pmid:28114940
- 13. Estabrooks CA, Squires JE, Hayduk L, Morgan D, Cummings GG, Ginsburg L, et al. Does organizational context influence the use of best practices by healthcare aides in residential long term care? Implementation Science. 2015;10.
- 14. Estabrooks CA, Squires JE, Cummings GG, Birdsell JM, Norton PG. Development and assessment of the Alberta Context Tool. BMC Health Serv Res. 2009;9:234. pmid:20003531.
- 15. Estabrooks CA, Squires JE, Hayduk LA, Cummings GG, Norton PG. Advancing the argument for validity of the Alberta Context Tool with healthcare aides in residential long-term care. BMC Medical Research Methodology. 2011;11(107). pmid:21767378.
- 16. Förberg U, Wallin L, Johansson E, Ygge B-M, Backheden M, Ehrenberg A. Relationship Between Work Context and Adherence to a Clinical Practice Guideline for Peripheral Venous Catheters Among Registered Nurses in Pediatric Care. Worldviews Evid Based Nurs. 2014;11(4):227–39. pmid:24986790
- 17. Slaughter SE, Estabrooks CA. Optimizing the mobility of residents with dementia: a pilot study promoting healthcare aide uptake of a simple mobility innovation in diverse nursing home settings. BMC Geriatr. 2013;13(1):110. pmid:24138586
- 18. Squires J, Estabrooks C, Scott S, Cummings G, Hayduk L, Kang S, et al. The influence of organizational context on the use of research by nurses in Canadian pediatric hospitals. BMC Health Serv Res. 2013;13(1):351. pmid:24034149
- 19. Estabrooks CA, Hoben M, Poss JW, Chamberlain SA, Thompson GN, Silvius JL, et al. Dying in a Nursing Home: Treatable Symptom Burden and its Link to Modifiable Features of Work Context. J Am Med Dir Assoc. 2015;16(6):515–20. pmid:25805625
- 20. Knopp-Sihota JA, Niehaus L, Squires JE, Norton PG, Estabrooks CA. Factors associated with rushed and missed resident care in western Canadian nursing homes: a cross-sectional survey of health care aides. J Clin Nurs. 2015;24(19–20):2815–25. Epub 2015/07/17. pmid:26177787.
- 21. Manns P, Darrah J. Use of the Alberta Context Tool to understand characteristics of the workplace important to evidence based practice. Physiotherapy. 2015;101:e949.
- 22. Schultz T, Kitson A. Measuring the context of care in an Australian acute care hospital: a nurse survey. Implementation Science. 2010;5(1):60. pmid:20673373
- 23. Scott SD, Grimshaw J, Klassen TP, Nettel-Aguirre A, Johnson DW. Understanding implementation processes of clinical pathways and clinical practice guidelines in pediatric contexts: a study protocol. Implementation Science: IS. 2011;6:133–. pmid:22204440.
- 24. Flowerdew L, Brown R, Russ S, Vincent C, Woloshynowych M. Teams under pressure in the emergency department: an interview study. Emerg Med J. 2012;29(12):e2. pmid:22186010
- 25. Nielsen KJ, Pedersen AH, Rasmussen K, Pape L, Mikkelsen KL. Work-related stressors and occurrence of adverse events in an ED. Am J Emerg Med. 2013;31(3):504–8. pmid:23347716.
- 26.
Andrew NE, Middleton S, Grimley R, Anderson CS, Donnan GA, Lannin NA, et al., editors. Organisational culture, leadership and evaluation on delivery of stroke care. 5th Annual NHMRC Symposium on Research Translation; 2016; Melbourne, Australia: NHMRC.
- 27. Cadilhac DA, Andrew NE, Kilkenny MF, Hill K, Grabsch B, Natash AL, et al. Improving quality and outcomes of stroke care in hospitals: Protocol and statistical analysis plan for the Stroke123 implementation study. Int J Stroke. 2017;13(1):96–106. pmid:28914187
- 28. Middleton S, Levi C, Dale S, Cheung NW, McInnes E, Considine J, et al. Triage, treatment and transfer of patients with stroke in emergency department trial (the T3 Trial): a cluster randomised trial protocol. Implement Sci. 2016;11(1):139. pmid:27756434; PubMed Central PMCID: PMC5069775.
- 29.
Translating Research in Elder Care (TREC). Alberta Context Tool—Versions [cited 2018 31 January]. Available from: https://trecresearch.ca/alberta_context_tool.
- 30.
Estabrooks C, Squires JE, Adachi AM, Kong L, Norton PG. Utilization of Research in Acute Care Settings in Alberta (AKUTE) Technical Report (Report No. 08-01-TR). Edmonton: Faculty of Nursing, University of Alberta, 2008.
- 31.
Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate Data Analysis. 7 ed. Essex, England: Pearson; 2014.
- 32.
Squires JE, Hutchinson AM, Hayduk L, Cranley L, Cummings G, Norton PG, et al. Alberta Context Tool: User Manual: Knowledge Utilization Studies Program, Faculty of Nursing, University of Alberta, Edmonton, Canada; 2014.
- 33.
StataCorp. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP. 2015.
- 34.
R Core Team. R: A language and environment for statistical computing. 3.4.0 ed. Vienna, Austria: R Foundation for Statistical Computing; 2017.
- 35. Bennett DA. How can I deal with missing data in my study? Aust N Z J Public Health. 2001;25(5):464–9. Epub 2001/11/02. pmid:11688629.
- 36. Hooper D, Coughlan J, Mullen MR. Structural Equation Modelling: Guidelines for Determining Model Fit. The Electronic Journal of Business Research Methods. 2008;6(6):53–60.
- 37. Squires JE, Hayduk L, Hutchinson AM, Mallick R, Norton PG, Cummings GG, et al. Reliability and Validity of the Alberta Context Tool (ACT) with Professional Nurses: Findings from a Multi-Study Analysis. PLoS One. 2015;10(6):e0127405–e. pmid:26098857.
- 38. Bland JM, Altman DG. Statistics notes: Cronbach's alpha. BMJ. 1997;314(7080):572. pmid:9055718
- 39.
Nunnally JC, Bernstein IH. Psychometric Theory. 3 ed. New York: McGraw-Hill; 1994.
- 40. Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychol Assess. 1995;7(3):309–19. http://dx.doi.org/10.1037/1040-3590.7.3.309.
- 41. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–8. pmid:18839484.
- 42. Hoben M, Estabrooks CA, Squires JE, Behrens J. Factor Structure, Reliability and Measurement Invariance of the Alberta Context Tool and the Conceptual Research Utilization Scale, for German Residential Long Term Care. Front Psychol. 2016;7:1339. PMC5013130. pmid:27656156
- 43. Eldh AC, Ehrenberg A, Squires JE, Estabrooks CA, Wallin L. Translating and testing the Alberta context tool for use among nurses in Swedish elder care. BMC Health Serv Res. 2013;13(1):68. pmid:23421736
- 44. Mallidou AA, Cummings GG, Ginsburg LR, Chuang YT, Kang S, Norton PG, et al. Staff, space, and time as dimensions of organizational slack: a psychometric assessment. Health Care Manage Rev. 2011;36(3):252–64. Epub 2011/06/08. pmid:21646884.