Reliability and Validity of THINC-it in Evaluating Cognitive Function of Patients with Bipolar Depression

Introduction

Bipolar disorder (BD) is a chronic mental disorder characterized by manic, hypomanic and depressive episodes.¹ Over the past decade, it has been demonstrated that bipolar depression (BD-D) patients have significant cognitive deficits in several cognitive domains, including planning, operational memory, attention, problem-solving, inhibition control, and psychological flexibility, which result in poor social functioning.² Studies have shown that cognitive function (especially executive function and motor ability) is seriously impaired in patients with bipolar depression.³ Further, compared with major depressive depression (MDD), bipolar depression has a lower onset age, more mood episodes, more hospitalizations, and higher incidence of psychotic features. Numerous studies have shown that patients with bipolar depression at any age can have cognitive impairment, even in remission.^4–9

The cognitive impairment of bipolar depression patients is significantly greater than that of patients with MDD.¹⁰ The causes of cognitive impairment in BD-D patients are not clearly elucidated, Multiple factors related to cognitive impairment have included changes of hippocampal structure and function (neural circuit),¹¹ obesity,¹² insufficient or excessive sleep,^13,14 childhood adversity,¹⁵ medications,¹¹ immune dysfunction, and increased inflammatory cytokines.¹⁶ Patients with a higher level of education may have reduced learning and memory deficits,⁹ while a high level of cultural knowledge plays a protective role in the cognitive function of patients with bipolar depression.¹⁷

To address the cognitive outcomes of BD patients and reducing disease-related disability, effective assessment of cognitive impairment in BD patients is needed.⁸ Although HAMD and MADRS are used to assess depressive symptoms, both scales show minimal correlation with objective cognitive measurement,¹¹ and thus would rely on subjective clinical assessment and patient self-reports to evaluate cognitive function.⁹ Other scales used to evaluate cognitive function include: MCCB, which focuses on clinical tests of cognitive function in schizophrenic patients,^18,19 and MoCA, which is often used as a general cognitive screening and dementia assessment tool.²⁰ MMSE and MoCA are suitable for elderly patients but generally not for young patients.²¹ In 2010, the International Society for Bipolar Disorders (ISBD) suggested that five subtests of MCCB, the California Verbal Learning Test, Stroop Test, Trail Making Test–part B, and Wisconsin Card Sorting Test can be used to test cognitive function of patients with BD.²² The Brief Assessment of Cognition in Affective Disorders (BAC-A) can also be applied to assess the global function of patients with bipolar disorder.²³ However, these scales are paper-based tests, which are time-consuming and laborious. Due to the lack of suitable brief and quick measurement tools, measurements of cognitive function remain a challenge.²⁴

Currently, there is no “gold standard” tool to evaluate or measure the cognitive function for the patients with bipolar disorder.⁸ Therefore, a simple, easy to use and accurate cognitive impairment assessment which includes both subjective and objective evaluation, and good validity is needed.

The THINC-integrated tool (THINC-it) was released at the 2016 annual meeting of the European Society of Neuropsychopharmacology (ECNP) (website: https://thinc.progress.im/en). It evaluates the cognitive function of MDD and can be applied electronically on portable tablets. The test consists of Spotter, Symbol Check, Codebreaker, Trails, and PDQ-5-D. This set of assessment tools has been validated internationally and been proven to be an effective and sensitive tool for detecting adult MDD with cognitive dysfunction.²⁵ The time stability, reliability, and convergence effectiveness of THINC-it have also been shown in normal subjects.²⁶ In a study of 100 patients with generalized anxiety disorder (GAD) and 100 healthy controls, THINC-it effectively evaluated cognitive dysfunction of GAD patients where linear regression analysis showed that anxiety symptoms accounted for 70.4% of the variability of subjective cognitive disorder.²⁷ A recent study showed that the cognitive defects identified by THINC-it were correlated with overall and specific psychosocial defects, thus confirming the clinical value and utility of THINC-it as a cognitive screening tool for MDD patients.²⁸

The THINC-it is a self-administered tool which requires some instructions given to the subject who must be able to operate a computer independently. Objective and subjective methods are used to screen cognitive impairment of patients with depression,⁸ including cognitive tests of attention, working memory, processing speed, executive function, and cognitive function of capturing subjective perception.²⁹

THINC-it is recommended for early detection of cognitive impairment in mood disorders including BD-D.³⁰ However, there is currently no evaluation of THINC-it in bipolar depression. Therefore, we evaluated the reliability and validity of THINC-it to measure the neurocognitive impairment in a bipolar depression (BD-D) group compared with a healthy control (HC) group.

Materials and Methods

Subjects

All procedures in this study adhered to the Declaration of Helsinki (2008) Ethical Principles for Medical Research Involving Human Subjects and were approved by the ethics committee of Zhejiang University, and all study participants gave voluntary and written informed consent.

Patients with BD-D Group

Patients diagnosed with bipolar disorder, current episode depressive (according to DSM-5 criteria and confirmed on MINI questionnaire), ie, bipolar depression (BD-D), without any other physical and psychiatric comorbidities, were recruited from May 2018 to April 2019 in the department of psychiatry, the First Affiliated Hospital, Zhejiang University School of Medicine. The inclusion criteria were as follows: 1) Age range from 18–65 years, both male and female; 2) outpatients or newly admitted inpatients who had not received any new treatment; 3) The score of Hamilton Depression Scale (HAMD-17) was no less than 17; 4) Mood Disorder Questionnaire (MDQ) score was positive (more than 7 items); 5) IQ≥80 on the Simple Wechsler Adult Intelligence; and 6) Subjects who had signed informed consent and were able to understand the self-report questionnaire. The exclusion criteria were as follows: 1) Alcohol and/or substance use disorders, autism spectrum disorder, dementia or any other neurodegenerative disorders, learning disabilities, schizophrenia or other mental disorders, or other medical conditions that may affect cognitive function, eg, brain tumors, multiple sclerosis, Parkinson’s disease, etc.; 2) Patients on any drugs which may affect cognitive function, such as glucocorticoids, beta-blockers, opioid analgesics, central stimulants, etc.; 3) Patients who took benzodiazepines within 12 hours or alcohol within 8 hours of THINC-it administration; or 4) Patients who received electroconvulsive therapy (ECT) within the last 6 months.

Each participant completed the following: five tests of THINC-it, five criteria-related tests, the Wechsler Intelligence Test, HAMD-17, and the Sheehan Disability Scale.

Healthy Controls (HC) Group

Healthy controls were voluntarily recruited from Shangcheng District of Hangzhou City, Zhejiang Province, including junior high school and senior high school students, nurses, security guards, and social workers. They were aged 18–65, male or female. The MINI questionnaire confirmed that they did not meet any diagnostic criteria of previous psychiatric disorders. They also did not have any history of neurological diseases, physical diseases, alcohol dependence or smoking habits, or family history of psychiatric disorder among their first-degree relatives. In total, 80 healthy participants were recruited in the study. The exclusion criteria were the same as that of the BD-D group. Thirty healthy participants were randomly selected 1 week later to retest five items of THINC-it.

Cognitive function evaluation was assessed by two psychiatrists. Before the evaluation, the consistency of scale scores was tested and no significant difference between raters were found. As far as possible training related procedures were adopted to eliminate any errors caused by subjective factors.

Measurements

The general assessment tools include the iPad version of THINC-it tool. The THINC-it complete set of tests includes Spotter, Symbol Check, Codebreaker, Trails, and the Perceived Deficits Questionnaire for Depression-5-item (PDQ-5-D).

The corresponding five criterion tests are the Perceived Deficits Questionnaire for Depression (PDQ-D), Reaction Time Paradigm (RTI), Digit Symbol Substitution Test (DSST), Trail Making Test-Part-B (TMT-B), and One-Back Task (1-back) (see Table 1). PDQ-D, DSST, and TMT-B are Pen-and-Paper Tests, while RTI is an online computer test, which is a subtest of the Cambridge Neuropsychological Test Automated Battery (CANTAB), the reaction time is taken as the result of this test, 1-back is also a computerized test developed by E-prime programme. The Sheehan Disability Scale (SDS) was then assessed. The Chinese version of PDQ-D is effective in psychometric evaluation of subjective cognitive impairment in MDD patients with good internal consistency and reliability of retest.³¹ The Sheehan Disability Scale is a self-rated disability scale, which evaluates three areas of disability: work, social life, and family life, with sufficient reliability and validity.³²

Table 1 Description of Cognitive Tasks in THINC-it Tests and the Criterion Tests

Each participant took about 15 minutes to complete five tests. Considering that different test sequences may have a potential impact on the results of the THINC-it test and Pen-and-Paper test, all participants were tested in the following order. Firstly, the demographic registration was completed, then the HAMD-17 tests were assessed. The MDQ was used to determine whether the participants met the diagnosis of BD-D. Four of the Wechsler intelligence tests were carried out, including common sense questions, arithmetic, similarity tests, number sequencing tests, and the total score was converted to IQ values. Then the objective test items PDQ-D, DSST, TMT-B, 1-back, and RTI were completed. Five items of THINC-it were then tested. The THINC-it test was conducted in the order of PDQ-5-D, Spotter, Symbol Check, Codebreaker, and Trails. The BD-D group and the healthy control group completed the whole items in this order, and tried to achieve the same test link to reduce errors. Before each test, detailed instructions were given to the subjects. All five tests of THINC-it needed to be completed continuously in a single session. The order and content of the BD-D group and healthy control group were the same, so as to minimize errors.

Statistical Analysis

All data were analyzed by IBM SPSS Statistics 23.0 software package, including group data t-test for independent samples, chi-square test for classified data, reliability test, internal consistency test, concurrent validity test, and correlation analysis with clinical symptoms. The data that support the findings of this study are available on request from the corresponding author.

In the various tests of THINC-it, the score of Spotter and Trails were positively associated with the severity of the cognitive impairment. In terms of Symbol Check and Codebreaker, on the contrary, the original score is negatively associated with the severity of the cognitive function. In the original THINC-it design, the direction of each test result should be consistent. In statistical data, PDQ-5-D, Spotter and Trails are converted into standard Z score and multiplied by −1, so the results of five tests indicate that the higher the score, the better the cognitive function. The data of PDQ-D, RTI, and TMT-B were analyzed by the same methods.

Results

A total of 98 patients with BD-D and 80 healthy controls were recruited. In the BD-D group, 21 patients were under the age of 18, 13 patients did not complete the test, and one patient had bipolar disorder, current episode mixed, five patients had IQ<80. In the healthy control group, two participants were under 18 years old, one has a family history, and 16 participants did not complete the test. Finally, in total 58 patients with BD-D and 61 healthy controls were included in the analyses.

The BD-D and healthy control groups completed all cognitive function assessment in the first test, signed informed consent, including general demographic data, THINC-it complete set, five criterion tests. A week later, they were randomly retested, with 30 healthy controls included in the retest. In terms of demographic data, there were no statistically significant group differences in age, gender, years of education, and IQ (P>0.05). In the BD-D group the HAMD-17 score was 23.29±5.20 and Sheehan Disability Scale score was 14.50±6.75. The age of onset in BD-D group was 20.33±7.94, and the duration of the disease was 203.56±332.64 days. There was no statistical difference in IQ between two groups (see Table 2).

Table 2 Demographic and Clinic Characteristics of Subjects with BD-D Group and HC Group

The Z-score was adopted to calculate the mean difference (SE) between two groups with THINC-it test. There were significant differences in PDQ-5-D, Codebreaker, Spotter, and Trails Mean difference (SE). The results are presented in Table 3. There was no significant difference in Symbol Check (MD (SE)=−0.01 (0.18), P=0.938; 95% CI=−0.38 to 0.35). There were significant differences in Z-scores of THINC-it objective test (MD (SE)=0.44 (0.12), P=0.001; 95% CI=0.19 to 0.69). There was also a significant difference in the Z-score of the total test (MD (SE)=0.03 (0.10), P=0.748; 95% CI=−0.17 to −0.24); The results of THINC-it test are basically consistent with those results of five criterion tests (see Table 3).

Table 3 Mean Difference in Performance on Individual Criterion Tests and Composite Z-Scores Between Subjects with BD-D Group and HC Group

Reliability and Validity Test Results of THINC-it

Concurrent Validity

We used five criterion tests as the criterion, THINC-it five tests as the corresponding validity analysis, and applied the Pearson correlation analysis. In the BD-D group, PDQ-5-D and PDQ-D had the highest concurrent validity (r=0.812, P<0.001), the Codebreaker and DSST had high concurrent validity too (r=0.747, P<0.001). It is suggested that the parallel validity of PDQ-5-D and Codebreaker is similar to that of the criterion test tool. The concurrent validity of Symbol Check and 1-back (r=0.456, P=0.001) were poor. Only 26 patients in the BD-D group completed the RTI test with poor concurrent validity of Spotter and RTI (r=0.389, P=0.017), Trails and TMT-B (r=0.360, P=0.006) were lowest. Concurrent validity of objective tests (r=0.695, P<0.001); Concurrent validity of the overall test (r=0.662, P<0.001) (see Table 4).

Table 4 Psychometric Properties of THINC-it Among Subjects with BD-D Group

Internal Consistency

With regards to internal consistency of THINC-it, the Cronbach’s alpha of PDQ-5-D was 0.640. The internal consistency of THINC-it five total tests (PDQ-5-D, Spotter, Symbol Check, Codebreaker, Trails) was 0.412; the Cronbach coefficient of four objective tests (Spotter, Symbol Check, Codebreaker, Trails) was 0.641; in the criterion tests, the Cronbach’s alpha of PDQ-D was 0.939 because of many items. The internal consistency of PDQ-D was very high. The Cronbach’s coefficient of the total test (PDQ-D, DSST, TMT-B, 1-back) was 0.378, and the Cronbach’s coefficient of the criterion tests objective test (DSST, TMT-B, 1-back) was 0.521 (see Table 4). The results of internal consistency were poor.

Reliability Test of Retest

Thirty subjects in the healthy control group were randomly retested, and all subjects were retested with the THINC-it test 1 week after the first test. All retest subjects completed the task well, using intraclass correlation efficient (ICC) analysis. ICC was between 0.440 and 0.757, the Codebreaker (ICC=0.757, P<0.001) and Symbol Check (ICC=0.737, P<0.001) were acceptable repeatability. The PDQ-5-D (ICC=0.683, P<0.001), Spotter (ICC=0.643, P<0.001) and Trails (ICC=0.440, P=0.008) had poor repeatability, the ICC=0.793 (P<0.001) in THINC-it objective composite score, and ICC=0.781 in THINC-it total composite score (see Table 4).

The Correlation Between the Test Results with HAMD-17 and SDS

The correlation between the scores of the test and HAMD-17 was compared. The peak, skewness, and Shapiro–Wilk test of HAMD-17 did not follow normal distribution. As most of the items in THINC-it did not follow normal distribution, we used the Spearman coefficient to calculate the correlation. PDQ-5-D was positively correlated with HAMD-17 (P<0.01). There was no significant correlation between objective test of THINC-it and HAMD-17 score (P>0.05). In the correlation comparison with the total score of Sheehan Disability Scale (SDS), as the total score of SDS was not distributed normally through kurtosis, skewness calculation, and Shapiro–Wilk test, we used the Spearman coefficient analysis. PDQ-5-D was positively correlated with SDS (P<0.01). Similar to the previous HMAD-17 analysis results, there was no significant correlation in the objective tests of THINC-it and SDS score (P>0.05) (see Table 5).

Table 5 The Correlation Between the Results of Various Tests and the Total Score of Clinical Symptoms

Discussion

In this study, we validated the THINC-it tool to measure cognitive function in a bipolar depression group and compared with a healthy control group. Except for Symbol Check, there were significant differences in cognitive function between the two groups in Spotter, PDQ-5-D, Codebreaker, and Trails, which thus indicates that THINC-it can effectively evaluate the cognitive dysfunction of BD-D.

The study showed that the internal consistency reliability (Cronbach’s α) of the subjective test item PDQ-5-D of THINC-it was 0.640. The Cronbach coefficient of four objective tests was 0.641, and the total Cronbach coefficient of five objective tests was 0.412. Previous study of THINC-it had demonstrated the internal consistency reliability of PDQ-5-D (Cronbach’s α=0.769), four objective composite tests (Cronbach’s α=0.551), and five total composite tests (Cronbach’s α=0.370)²⁷ are basically consistent with our test results.

PDQ-5-D and PDQ-D, Codebreaker, and DSST had high concurrent validity, but the other three subtests of THINC-it (including Symbol Check and 1-back, Spotter and RTI, Trails and TMT-B) did not show high concurrent validity, and the concurrent validity results were basically the same as that of THINC-it’s previous internal tests.²⁷ In general, the concurrent validity was low, The reason may be that the criterion is similar to THINC-it, when the THINC-it test is completed, then the criterion test is done, equivalent to doing the same test twice, As the participants become more familiar with the test tools, the test scores are increased, and the results are different, thus affecting the concurrent validity value.

For the retest reliability test, we retested 30 subjects in the healthy control group randomly, Codebreaker (ICC=0.757) and Symbol Check (ICC=0.737) were acceptable repeatability. The value of PDQ-5-D (ICC=0.683) exceeded the minimum acceptable level of 0.65. Spotter (ICC=0.643) and Trails (ICC=0.440) had poor repeatability. In this test, ICC value is low, and the reason may be the same as repeated test, the second time more skilled than the first. In comparison with scores of clinical symptoms, subjective test PDQ-5-D was significantly correlated with HAMD-17 and SDS scores, which could well reflect the degree of depression of patients, The four objective tests of the BD-D group and HC group had no significant correlation with HAMD-17 and SDS scores (P>0.05). One study found that differences in workplace performance among MDD patients are largely determined by the level of cognitive impairment, not by the severity of depression.³³ Increasing evidence indicated that impaired cognitive function might be an independent symptom in mood disorder.^34–37

In one study, 262 unmedicated MDD patients (161 females and 101 males) were tested with 24-Hamilton Depression Rating Scale (HDRS) and Beck Depression Inventory. Cognitive performance was not significantly correlated with the standard depression severity scale commonly used in clinical trials.³⁸ Similarly, there was no correlation between the Chinese version of PDQ-D and DSST.³¹

In the comparison of mean difference between the bipolar depression group and the healthy control group, there were significant differences in PDQ-5-D, Spotter, Codebreaker, and Trails (P<0.05). However, in terms of Symbol Check, there was no significant difference between the depression group and the control group (P=0.938), and the test effect on cognitive function was slightly worse, with poor correlation with clinical symptoms. The test results in our study are basically the same as those of the THINC-it project team.²⁵ When the THINC-it psychological measurement was conducted, it was found that the convergence validity of Symbol Check task was low.²⁶

In our study, in order to reduce the test error, bipolar depression group, and healthy control group completed the testing in a strict order. However, variability in educational levels, depression severity, and cognitive function resulted in differences in the duration (1.5–4 hours), quality, and accuracy (about 50%) in completing the test. One study pointed out that the Symbol Check test was not a reliable alternative to the 1-back test.²⁶ In interpreting the results, we feel that it is not reasonable to conclude that the duration of test completion indicates the level of cognitive function as accuracy of response also needs to be taken into account. Further, the Symbol Check difficulty coefficient is large, which is more likely to result in the low score rather than the BD-D cognitive function decline.

Some subjects in the BD-D group of this study had high IQ values, which may offset the effects of cognitive impairment. In contrast, people with IQ higher than the average of the population by more than one standard deviation are less likely to be detected as “impaired”, even if their performance is significantly lower than their IQ prediction level.^5,39

In this study, five tests of THINC-it were programmed on an iPad, which was convenient and fast for self-administration. However, the performance of the tests is also dependent on adequate explanation and understanding of the instructions on how to complete the iPad version. It may take a long time for some participants to fully understand the rules and apply the skills to operate an electronic device. Hence, THINC-it tools which are programmed on tablets, or smartphones or personal computers may be a limitation, especially for elderly patients.⁴⁰

Limitations

First, despite the large sample, many patients were not completely familiar with the rules, and coupled with unstable operation of the software system, thus resulting in significantly lower numbers included in the final analyses. Second, some patients had previous hospital experience, and may have taken similar tests, which may have affected the test results. Third, some subjects in the healthy control group had exaggerated PDQ-5-D and PDQ-D scores, but the subjects had no significant depressive mood. Fourth, many subjects in the BD-D group were older, seldom used computers, and had difficulty in operating tablet computers, which could have affected the test results. Last, patients with bipolar depression were not completely medication free, which may have affected the test performance.

Conclusion

In the study, we validated that THINC-it has a high sensitivity in the evaluation of cognitive function in patients with BD-D, however, its reliability and concurrent validity are low, and it has no significant correlation with clinical symptoms. In the practice, it is suggested to apply THINC-it to evaluate the cognitive function of the patients with BD-D. In the future, we can amplify the sample size to validate the clinical application.