Variation in subsequent viral load testing and outcomes by visit type patterns in the first year of the COVID-19 pandemic at a large academic medical center in North Carolina

Introduction

People living with human immunodeficiency virus (HIV) faced high levels of HIV care interruptions during the coronavirus disease 2019 (COVID-19) pandemic (1,2). Telehealth was rapidly implemented in most HIV clinics across the United States (US) to facilitate continuity of care (3). HIV care continuity is crucial to attain antiretroviral treatment (ART) benefits, such as decreased HIV-related co-morbidities and improved life expectancy for people living with HIV (PWH), and reduced transmission risk for people at risk of acquiring HIV (4-6).

As HIV treatment relies heavily on regular laboratory testing to monitor HIV progression and immune status, viral load (VL) and cluster of differentiation 4 (CD4) count testing uptake and outcomes were frequently used to describe the quality of telehealth HIV care. However, study findings remain conflicting. While some telehealth studies found that neither VL outcomes nor CD4 counts varied by visit type or pandemic vs. pre-pandemic time periods (7-12), Galaviz et al. found increased telehealth visit volumes to co-occur with lower rates of undetectable VLs (13). Other studies noticed increased odds of viral non-suppression during shelter-in-place orders (14,15), and after the initial COVID-19 wave (16), as compared to the pre-pandemic epoch. Widely varying approaches, data sources, sample sizes, and foci on specific populations (e.g., veteran or pediatric/adolescent populations) limit the generalizability of study results.

This study is one of the first to evaluate variation in VL testing and outcomes following variation in HIV care visit types in the first year of the COVID-19 pandemic at a large academic medical center in North Carolina (NC) serving a diverse population of PWH. Using data on all HIV care visits of the entire patient population served at a large HIV care provider in NC, our findings provide additional understanding of conflicting literature by assessing variation in subsequent VL testing and outcomes related to visit type patterns during the first year of the COVID-19 pandemic.

Methods

Design, setting, and data

We used a retrospective observational study design to evaluate variation in VL testing and outcomes by visit type in the first year of the COVID-19 pandemic among PWH. Electronic health records (EHRs) data from Duke University’s Infectious Diseases (ID) Clinic in Durham, NC were used. The Duke ID clinic provides HIV care to a heterogeneous patient population of around 2,000 PWH. Study data comprise all visits of eligible patients, i.e., adult PWH cared for by the Duke ID clinic between March 16, 2020 to December 31, 2022. EHR data were extracted using Epic’s SlicerDicer tool, a reporting tool within Epic EHR management software (17) in June 2023. Visit-level data were aggregated across patients by HIV care pattern and VL history. This manuscript is part of a larger study on telehealth and in-person care use during the COVID-19 pandemic, a detailed description of the data used in this study has been reported previously (18).

Ethical statement

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Researchers from Duke University and University of South Carolina collaborated to conduct this study and sought ethical approval for their research and collaboration at both institutions. The study protocol was determined to be exempt from review by the Institutional Review Boards (IRBs) of Duke University Health System (Pro00112920) and the University of South Carolina (Health Sciences South Carolina IRB; Pro00128577). The IRBs waived the requirement for informed consent.

Variables of interest

Visit type patterns were described as whether PWH received (I) only in-person; (II) a combination of telehealth and in-person; or (III) only telehealth HIV care in the first year of the pandemic (i.e., between March 16, 2020 and March 15, 2021). March 16, 2020 was selected as beginning of the pandemic as that day marked a clear and sudden drop in in-person clinic visits at Duke’s ID clinic. NC’s stay at home order was enacted shortly thereafter on March 30, 2020. The key outcome variables of interest were VL testing in 2022 and viral suppression (VS) among people who had a test in 2022 as VL is the recommended clinical marker to monitor a patient’s response to ART (19) and can be used to assess longer-term outcomes associated with telehealth. VL testing was categorized into two groups: (I) not having any VL test recorded in 2022 or (II) having at least one VL test recorded in 2022. VS was categorized into two groups: (I) VS at all tests in 2022 [<200 copies per milliliter blood (c/mL)] or (II) at least one VL test was unsuppressed in 2022 (≥200 c/mL).

Analysis

Differences in VL testing and VS by visit type patterns were assessed using Pearson’s Chi-square (χ²) tests.

Two different approaches of tipping point (TP) analysis were performed to explore the sensitivity of our results. TP analysis is an approach to analyze missing data under the assumption that missing data are related to unobserved variables and to identify the ‘tipping point’ at which differences in VS by visit type patterns become statistically significant (20,21). TP scenarios were generated to describe missing VL outcomes among PWH that had no VL test in 2022 (18–30.9% across visit type patterns).

The first approach of TP analysis explored the increasing proportion of PWH with at least one unsuppressed VL test among those without a VL test (“non-testers”) keeping the treatment effect of visit types constant. The observed proportion of PWH with unsuppressed VL among those who had a VL test in 2022 (“tested”) was used as the starting point (multiplier =1). Following recent studies reporting that people with unsuppressed VL were more likely to delay or miss VL tests (22,23), we incrementally increased the proportion of unsuppressed VL in our missing data from non-testers. The multipliers ranged from 1 to 4 increasing by 0.1 increments. For example, the multiplier 2.3 describes a scenario where the proportion of PWH with an unsuppressed VL test among non-testers is 2.3 times the proportion of PWH with at least one unsuppressed VL test among tested PWH.

For the second approach of TP analysis, the treatment effect for mixed visit types and telehealth only compared to in-person HIV care was calculated using the proportion ratio of unsuppressed VL tests. The treatment effect was incrementally decreased and tested the assumption that telehealth use (only and in combination) is inferior to in-person care. Telehealth only and mixed visit types were compared to in-person care separately by calculating the relative difference in treatment effects, which describes the probability of in-person care being more effective than telehealth or mixed visit types. TP scenarios were generated by incrementally adding 1-percentage point (pp) to the proportion of unsuppressed VL tests among non-testers using mixed visit types or telehealth only. The pp increase ranged from 0–20. The proportion of unsuppressed VL tests for in-person care remained constant at the VS rate among tested PWH. χ² tests were performed for each scenario. P values below 0.05 were considered statistically significant. Analyses were performed in StataSE version 18 (StataCorp, College Station, TX, USA).

Results

Among the 1,835 PWH who received HIV care at the Duke ID clinic in the first year of the COVID-19 pandemic, most patients were 50 years and older (63%), non-Hispanic Black (58%), male (74%), and living in a county different from the ID clinic location (65%; Table 1). About half (53%) received exclusively in-person HIV care, 32% PWH received a combination of telehealth and in-person HIV care, and 15% PWH received exclusively telehealth HIV care. About 20% of PWH did not have any VL test recorded in 2022, 72% of PWH were virologically suppressed at all tests in 2022, and 8% had at least one unsuppressed VL test. Among those who had at least one VL test documented in 2022 (n=1,461), 90% were virologically suppressed at all tests in 2022, and 10% had at least one unsuppressed VL test.

VL testing in 2022 varied significantly by visit type patterns in the first year of the pandemic (P<0.001; Figure 1A). The proportion of people who had no VL test recorded was higher among people who used telehealth only as compared to people who used in-person care only or a combination of telehealth and in-person HIV care (31% vs. 19% and 18%, respectively). We found no significant variation in VS by visit type among people who had at least one VL test in 2022 (Figure 1B).

Figure 1 Systematic variation in viral load testing and virologic suppression in 2022 with patterns of HIV care visit types during the first year of the COVID-19 pandemic*. (A) Variation in viral load testing by visit type. (B) Variation in virologic suppression by visit type. ^a, P values are calculated by Pearson’s Chi-square test. *, March 16, 2020 through March 15, 2021. COVID-19, coronavirus disease 2019; HIV, human immunodeficiency virus.

Our first approach for the TP sensitivity analysis followed the assumption that VS is different among non-testers compared to tested PWH (Figure 2). We identified the following scenarios at which changes in VS by visit type patterns reached statistical significance: multiplying the proportion of PWH with at least one unsuppressed VL test among non-testers by a factor of 2.1 reached significance at α =0.05 with P=0.045. Multiplying the proportion of PWH with at least one unsuppressed VL test among non-testers by a factor of 2.8 reached statistical significance at α =0.01 with P=0.009.

Figure 2 TP sensitivity analysis #1: Pearson’s Chi-square test results and proportion of people with at least one unsuppressed VL test by TP multiplier and visit type. Tipping point scenarios incrementally increased the proportion of people with HIV with at least one unsuppressed VL test among those who had no VL test in 2022. Tipping point multipliers ranged from 1 to 4. Pearson’s Chi-square tests were used to assess the variation in virologic suppression by visit type patterns for each tipping point scenario. *, statistical significance at α =0.05; **, statistical significance at α =0.01. HIV, human immunodeficiency virus; TP, tipping point; PWH, people living with HIV; VL, viral load.

The treatment effect (proportion ratio of unsuppressed VL tests) among tested PWH for mixed visit types vs. in-person only care was 1.14 indicating that PWH using mixed visit types were 1.14 times as likely to have an unsuppressed VL test than in-person only users. The treatment effect for telehealth only vs. in-person only care was 1.37. For our second approach for the TP sensitivity analysis, we incrementally decreased the effect of telehealth only (Figure 3A) and mixed visit types (Figure 3B). We identified that telehealth only reached significance at α =0.05 with a 3-pp increase in the proportion of unsuppressed VL tests among non-testers with P=0.04 (treatment effect: 1.70). Telehealth only reached significance at α =0.01 with a 7-pp increase in the proportion of unsuppressed VL tests among non-testers with P=0.007 (treatment effect: 2.14). Telehealth only reached significance at α =0.001 with a 13-pp increase in the proportion of unsuppressed VL tests among non-testers with P<0.001 (treatment effect: 2.81).

Figure 3 TP sensitivity analysis #2: Pearson’s Chi-square test results and treatment effect of people with at least one unsuppressed VL test by TP percentage point increase and visit type. Tipping point scenarios incrementally increased the proportion of people with HIV with at least one unsuppressed VL test among telehealth and mixed visit type users who had no VL test in 2022. Percentage point increases ranged from 0 to 20. Pearson’s Chi-square tests were used to assess the variation in virologic suppression comparing telehealth only vs. in-person only and mixed visit types vs. in-person only for each tipping point scenario. (A) Telehealth only versus in-person only. (B) Mixed visit types vs. in-person only. *, statistical significance at α =0.05; **, indicate statistical significance at α =0.01; ***, indicate statistical significance at α =0.001. HIV, human immunodeficiency virus; PWH, people living with HIV; TP, tipping point; VL, viral load.

For mixed visit type use, TP analysis #2 showed that significance at α =0.05 was reached with an 11-pp increase in the proportion of unsuppressed VL tests among non-testers with P=0.043 (treatment effect: 2.36). Mixed visit types reached significance at α =0.01 with a 16-pp increase in the proportion of unsuppressed VL tests among non-testers with P=0.009 (treatment effect: 2.91).

Discussion

Telehealth HIV care was used by almost half (47%) of the PWH cared for by the Duke ID clinic between March 16, 2020 and March 15, 2021. We found that visit type patterns were significantly associated with VL testing uptake in 2022 but not VS among people with at least one VL test in 2022.

Among telehealth only users the proportion not having a VL test documented in 2022 was higher than among in-person care users and those using a combination of telehealth and in-person HIV care. While our data does not allow us to further examine the characteristics of telehealth only users, we hypothesize that the lack of VL testing may stem from two factors: the inability to draw blood during telehealth encounters and patients’ transferring care to another clinic. Firstly, the lack of ability to draw blood during a telehealth encounter is an often-cited barrier to or disadvantage of telehealth (13,24-28). As demonstrated in pilot studies (29,30), alternative laboratory monitoring strategies including home-based VL testing and monitoring may present a feasible option to guide the optimal integration of telehealth in HIV care, promote successful and equitable HIV care, and to overcome this disadvantage of telehealth HIV care. Secondly, our study population likely included people who moved and/or transferred care to another clinic during or after the pandemic. While we do not have access to data to observe care engagement and outcomes of PWH who left our HIV clinic, clinic transfers can be associated with undesirable HIV care outcomes (31). Future research is needed to understand whether people who do not have a VL test documented in Duke’s ID clinic EHR have discontinued their care or continued it elsewhere.

We found that 90% of PWH who had at least one VL test documented in 2022 at Duke’s ID Clinic were virologically suppressed at all tests. Including all PWH in our study population, VS was reached by 72% of people with HIV, which is 5 percentage-points above the rate of VS in NC (32). Finding that the proportions of PWH having at least one unsuppressed VL result in 2022 is not statistically significantly different across visit type patterns supports telehealth as a viable HIV care option that can lead to desirable HIV care outcomes as described by others (7-11,33,34). Nevertheless, reports of lower viral suppression rates among telehealth users in some studies and especially among people experiencing housing instability (13-16), merit serious consideration. Prudence requires continued monitoring of VL testing and sustained efforts to encourage antiretroviral medication adherence to maintain VS among telehealth users.

Borrowing the TP approach from clinical trial methodology, we were able to provide additional, important context information to evaluate telehealth in HIV care and its performance under “worse” circumstances as compared to the observed. Firstly, we were able to characterize those PWH who did not have a documented VL in 2022 and identify the ‘tipping point proportion’ of viral non-suppression that would result in a statistically significant difference of VL outcomes by visit type. While research shows that missed HIV care visits are associated with undesirable clinical outcomes (35), our findings identify a large TP at which the use of telehealth would be associated with undesirable outcomes as compared to in-person care supporting the robustness of our findings. The proportion of PWH with an unsuppressed VL among non-testers would have to be twice (or almost three times) as high as the proportion of PWH with an unsuppressed VL among testers to reach significance with a 95% (or 99%) confidence level. This large difference in the proportion of viral non-suppression necessary to result in worse outcomes associated with the use of telehealth may support the suitability of telehealth. Nevertheless, longitudinal studies following PWH in their HIV care treatment for a longer period are needed to assess VL outcomes over time among people who have missed HIV care and/or laboratory visits. Secondly, we were able to identify the change in treatment effectiveness of telehealth only or mixed visit types to reach statistically significant differences of VS by visit type. Our analyses showed that large TPs would be needed for significant results indicating that in-person care is more effective than telehealth or mixed visit types. This finding supports the robustness of our initial findings that there is no significant difference between visit types. For telehealth only, the relative difference in treatment effect would need to be almost twice (or more than three times) as large to reach significance with a 95% (or 99%) confidence level. For mixed visit types, the relative difference in treatment effect would need to be almost 10 times (or more than 13 times) larger to reach significance with a 95% (or 99%) confidence level. Identifying these large differences in treatment effect needed to reach statistical significance therefore supports the suitability of telehealth as an HIV care option with the potential to achieve desirable HIV care outcomes and ART benefits. However, these TP scenarios were artificially designed, and studies evaluating real world evidence are warranted to inform telehealth scale-up and promotion in HIV care.

Limitations

We note several limitations of this study. Firstly, we only included EHR data from one HIV care clinic in NC, which limits generalizability of our findings. Secondly, because this study used aggregated data multivariable analyses including other potential confounders of VL testing and outcomes could not be performed. Nevertheless, findings from this study warrant further individual-level and multi-variable analyses to immerse deeper into telehealth and HIV care disparities and explore causal relationships. Thirdly, in our study we use subsequent outcomes data measured in calendar year 2022 (≥1 year after visits occurred). While this may result in the inclusion of people moving or switching HIV care providers, it allows to focus on longer-term outcomes and prevents an overemphasis of short-term COVID-19 pandemic-related changes in VL testing and VS. Fourth, using Chi-squared tests, we tested for a difference of outcomes which does not allow any direct conclusions regarding non-inferiority of telehealth compared to telehealth. The small proportion of people using telehealth only care in our study may have prevented detecting significant differences. As mentioned above, a more detailed analysis of individual level data should investigate equivalence or non-inferiority of visit types.

Conclusions

We found a significant difference in VL testing but not in VL outcomes among telehealth users who had VL testing results at least one year later. By using TP analysis, we further identified that the proportion of PWH having unsuppressed VL tests would need to be more than twice as high among people who had no VL test as compared to people with VL tests to show statistically significantly different VS rates by visit type. This finding supports the robustness of our finding that VL outcomes do not vary by visit type. Taken together, these data provide some support for telehealth as a viable HIV care option to achieve desirable HIV care outcomes at the individual and population level, with the caveat that in this analysis VL testing uptake was lower among telehealth only users. As VL testing is crucial to monitor treatment success, strategies such as remote VL testing at home or through a local laboratory could help to ensure regular VL testing among PWH who use telehealth HIV care.

Acknowledgments

We greatly appreciate the regulatory support from Katherine Link at Duke University. This publication was made possible with regulatory help from the Duke University Center for AIDS Research (CFAR), an NIH funded program (5P30AI064518). We acknowledge that parts of the data presented in this manuscript were presented at the Conference on Retroviruses and Opportunistic Infections (CROI) 2024 conference (abstract #1075).

Footnote

Peer Review File: Available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-69/prf

Funding: Research reported in this publication was supported by the Agency for Healthcare Research and Quality (AHRQ) Grants for Health Services Research Dissertation Program (R36HS029267 to V.Y.) and the SPARC Graduate Research Program from the Office of the Vice President for Research at University of South Carolina (USC; to V.Y.), the National Institutes of Health (NIH) Ruth L. Kirschstein National Research Service Award (5T32AI007392 to V.Y.), and a NIH funded program (5P30AI064518). The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ, USC, Duke University, or NIH.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://mhealth.amegroups.com/article/view/10.21037/mhealth-24-69/coif). V.Y. received supports from the Agency for Healthcare Research and Quality (AHRQ) Grants for Health Services Research Dissertation Program (R36HS029267); the SPARC Graduate Research Program from the Office of the Vice President for Research at University of South Carolina (USC); and the NIH Ruth L. Kirschstein National Research Service Award (5T32AI007392) to present parts of the data presented in this manuscript at the CROI 2024 conference. This publication was made possible with regulatory help from the Duke University Center for AIDS Research (CFAR) and an NIH funded program (5P30AI064518). The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Researchers from Duke University and University of South Carolina collaborated to conduct this study and sought ethical approval for their research and collaboration at both institutions. The study protocol was determined to be exempt from review by the Institutional Review Boards (IRBs) of Duke University Health System (Pro00112920) and the University of South Carolina (Health Sciences South Carolina IRB; Pro00128577). The IRBs waived the requirement for informed consent.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Winwood JJ, Fitzgerald L, Gardiner B, et al. Exploring the Social Impacts of the COVID-19 Pandemic on People Living with HIV (PLHIV): A Scoping Review. AIDS Behav 2021;25:4125-40. [PubMed]
2. Harkness A, Weinstein ER, Atuluru P, et al. Latinx Sexual Minority Men's Access to HIV and Behavioral Health Services in South Florida During COVID-19: A Qualitative Study of Barriers, Facilitators, and Innovations. J Assoc Nurses AIDS Care 2022;33:9-21. [PubMed]
3. Dawson L, Kates J. Delivering HIV Care and Prevention in the COVID Era: A National Survey of Ryan White Providers. Kaiser Family Foundation; 2020. Available online: https://www.kff.org/report-section/delivering-hiv-care-prevention-in-the-covid-era-a-national-survey-of-ryan-white-providers-issue-brief/
4. Jewell BL, Smith JA, Hallett TB. Understanding the impact of interruptions to HIV services during the COVID-19 pandemic: A modelling study. EClinicalMedicine 2020;26:100483. [PubMed]
5. Maqutu D, Zewotir T. Optimal HAART adherence over time and time interval between successive visits: their association and determinants. AIDS Care 2011;23:1417-24. [PubMed]
6. Siddiqi AE, Hall HI, Hu X, et al. Population-Based Estimates of Life Expectancy After HIV Diagnosis: United States 2008-2011. J Acquir Immune Defic Syndr 2016;72:230-6. [PubMed]
7. Sorbera M, Fischetti B, Khaimova R, et al. Evaluation of virologic suppression rates during the COVID-19 pandemic with outpatient interdisciplinary HIV care. J Am Coll Clin Pharm 2021;4:964-8. [PubMed]
8. Koay WL, Prabhakar S, Neilan A, et al. Brief Report: Supporting Access to HIV Care for Children and Youth During the COVID-19 Pandemic With Telemedicine and Rideshare. JAIDS Journal of Acquired Immune Deficiency Syndromes 2021;88:384-8. [PubMed]
9. Salgado S, Felzien G, Brumbeloe J. Georgia Leverages Telehealth to Expand HIV Care Management in Underserved Areas. Am J Prev Med 2021;61:S55-9. [PubMed]
10. McGinnis KA, Skanderson M, Justice AC, et al. HIV care using differentiated service delivery during the COVID-19 pandemic: a nationwide cohort study in the US Department of Veterans Affairs. J Int AIDS Soc 2021;24:e25810. [Crossref] [PubMed]
11. Mohr KB, Lee-Rodriguez C, Samiezade-Yazd Z, et al. Impact of the Coronavirus Disease 2019 Pandemic on Antiretroviral Therapy Initiation and Care Delivery for People With Newly Diagnosed HIV in an Integrated Healthcare System. Open Forum Infect Dis 2022;9:ofac639. [Crossref] [PubMed]
12. Yelverton V, Ostermann J, Natafgi N, et al. Disparities in Telehealth Use in HIV Care During the COVID-19 Pandemic: Study Findings from South Carolina. Telemed J E Health 2024;30:1594-9. [Crossref] [PubMed]
13. Galaviz KI, Shah NS, Gutierrez M, et al. Patient Experiences with Telemedicine for HIV Care During the First COVID-19 Wave in Atlanta, Georgia. AIDS Res Hum Retroviruses 2022;38:415-20. [Crossref] [PubMed]
14. Spinelli MA, Hickey MD, Glidden DV, et al. Viral suppression rates in a safety-net HIV clinic in San Francisco destabilized during COVID-19. AIDS 2020;34:2328-31. [Crossref] [PubMed]
15. Spinelli MA, Le Tourneau N, Glidden DV, et al. Impact of Multicomponent Support Strategies on Human Immunodeficiency Virus Virologic Suppression Rates During Coronavirus Disease 2019: An Interrupted Time Series Analysis. Clin Infect Dis 2022;75:e947-54. [Crossref] [PubMed]
16. Norwood J, Kheshti A, Shepherd BE, et al. The Impact of COVID-19 on the HIV Care Continuum in a Large Urban Southern Clinic. AIDS Behav 2022;26:2825-9. [PubMed]
17. SlicerDicer. Epic electronic health records management software. Verona, WI, USA: Epic Systems Corporation; 2023.
18. Yelverton V, Ostermann J, Yarrington M, et al. Telehealth and in-person HIV care during the COVID-19 pandemic at a large academic medical center in North Carolina. PLoS One ; in press. [Crossref]
19. Guidelines for the Use of Antiretroviral Agents in Adults and Adolescents with HIV. Department of Health and Human Services; 2024. Available online: https://clinicalinfo.hiv.gov/en/guidelines/adult-and-adolescent-arv
20. Jin M, Liu R, Robieson W. Modified reference based imputation and tipping point analysis in the presence of missing data due to COVID-19. Contemp Clin Trials 2021;110:106575. [Crossref] [PubMed]
21. Yan X, Lee S, Li N. Missing data handling methods in medical device clinical trials. J Biopharm Stat 2009;19:1085-98. [Crossref] [PubMed]
22. Mbira TE, Kufa T, Sherman GG, et al. Compliance to Viral Load Monitoring Schedules Among Women Attending Prevention of Vertical HIV Transmission Services Before and During the COVID-19 Pandemic in Ehlanzeni District, Mpumalanga, South Africa. AIDS Behav 2024;28:868-85. [Crossref] [PubMed]
23. Fox MP, Brennan AT, Nattey C, et al. Delays in repeat HIV viral load testing for those with elevated viral loads: a national perspective from South Africa. J Int AIDS Soc 2020;23:e25542. [Crossref] [PubMed]
24. Ennis N, Armas L, Butame S, et al. Factors Impacting Video Telehealth Appointment Completion During COVID-19 Pandemic Among People Living with HIV in a Community-Based Health System. AIDS Behav 2022;26:407-14. [Crossref] [PubMed]
25. Yelverton V, Qiao S, Weissman S, et al. Telehealth for HIV Care Services in South Carolina: Utilization, Barriers, and Promotion Strategies During the COVID-19 Pandemic. AIDS Behav 2021;25:3909-21. [PubMed]
26. Eaton EF, Burgan K, McCollum G, et al. Expanding access to substance use services and mental health care for people with HIV in Alabama, a technology readiness assessment using a mixed methods approach. BMC Health Serv Res 2022;22:919. [PubMed]
27. Hickey MD, Sergi F, Zhang K, et al. Pragmatic randomized trial of a pre-visit intervention to improve the quality of telemedicine visits for vulnerable patients living with HIV. J Telemed Telecare 2023;29:187-95. [PubMed]
28. Voisin DR, Edwards T, Takahashi LM, et al. COVID-19, Retention in HIV Care, and Access to Ancillary Services for Young Black Men Living with HIV in Chicago. AIDS Behav 2023;27:535-44. [PubMed]
29. Diaz JE, Ghanooni D, Atkins L, et al. Challenges and Opportunities with at-Home Blood Collection for HIV-1 Viral Load Monitoring among Sexual Minoritized Men who use Stimulants. AIDS Behav 2024;28:3809-18. [PubMed]
30. Teran RA, Sobieszczyk ME, Chiasson MA, et al. Longitudinal Viral Load Monitoring Using Home-Collected Dried Blood Spot Specimens of MSM Living with HIV: Results from a Feasibility Pilot Study. AIDS Behav 2021;25:661-6. [PubMed]
31. Krentz HB, Worthington H, Gill MJ. Adverse health effects for individuals who move between HIV care centers. J Acquir Immune Defic Syndr 2011;57:51-4. [PubMed]
32. North Carolina Department of Health and Human Services. HIV Care Outcomes in North Carolina; 2023. Available online: https://epi.dph.ncdhhs.gov/cd/stds/figures/2023_Care_Outcomes-HIV.pdf?ver=1.2
33. Smith E, Badowski ME. Telemedicine for HIV Care: Current Status and Future Prospects. HIV AIDS (Auckl) 2021;13:651-6. [PubMed]
34. Fadul N, Regan N, Kaddoura L, et al. A Midwestern Academic HIV Clinic Operation during the COVID-19 Pandemic: Implementation Strategy and Preliminary Outcomes. J Int Assoc Provid AIDS Care 2021;20:23259582211041423. [PubMed]
35. Walburn A, Swindells S, Fisher C, et al. Missed visits and decline in CD4 cell count among HIV-infected patients: a mixed method study. Int J Infect Dis 2012;16:e779-85. [PubMed]