iRefer: A Novel Application for Making Referrals at a Student-Run Free Clinic

,


Introduction
Referring patients who need specialized care to the appropriate provider (e.g.medical specialist, housing shelter) is a central role of a student-run free clinic (SRFC). 1,2Of the three SRFCs for which referral data could be found through published annual reports or personal communication, an average of 34.9% of patient encounters resulted in a referral. 3,4Common services for which patients are referred include dental care, physical therapy, and laboratory work. 5Considering the widespread use of referrals at student-run clinics, improving the process through which patients are connected with specialty care providers is likely to improve access, optimize the completion of specialty services, and ultimately improve care.
Mobile Clinic Project (MCP) is a free clinic founded in 2001 at the University of California, Los Angeles (UCLA) that provides medical services for the homeless and underserved each Wednesday evening at a street corner in West Hollywood, California and two Saturdays a month at a community center in Santa Monica, California.The patient population is primarily adults and includes patients with and without insurance.MCP is entirely student-run, with volunteers drawn from medical, undergraduate, public health, and dental schools.Primary care medical visits, chronic disease management, wound and foot care, social work services, immunizations, and medications are provided by the clinic at no cost to patients.The organization depends heavily on grant funding, donations, and student and faculty volunteers.
Referring free clinic patients to specialty caredefined here as care, whether medical, social, community-based, or legal, that is beyond what can be provided by a free clinic-presents unique challenges.For one, clinics or shelters offering free services may provide assistance contingent on the patient's insurance status or possession of a photo identification.Furthermore, information about the specialty care providers can frequently change.An annual update of MCP's referral database found that nearly 20% of 141 specialty providers had either changed their services offered (12), modified eligibility criteria (10), or closed down (6) over the past year.Careful management of referral site information is necessary to avoid referring a patient for a service they are ineligible to receive.
Student-run free clinics have adopted a number of approaches to facilitate referrals.The Mount Sinai medical school-run clinic trains a referralsspecific committee dedicated to building relationships with off-site care providers and selecting the appropriate referral for each patient. 4The University of Central Florida clinics invite outside providers to in-house specialty care nights where patients can see multiple specialists in one session. 6The University of Kansas free clinic has a Director of Physician Relations who calls local specialty physicians to arrange appointments. 7The methods implemented by these clinics allow them to further patients' care even when needs exceed the capacity of the clinic.
Maintaining accurate, current, and easily accessible information about potential referral sites is essential for clinics to connect patients with care in a manner that is streamlined and effective.During most of its history, MCP trained a ten-student "referrals committee" in charge of managing these tasks.Referral site information was collected in one-page paper documents describing each site.By 2014 the referrals committee was managing a list of 140+ referral sites.Searching through these documents soon became a logistical bottleneck.Simply training the committee on referral site details required 20+ hours per member, leaving less time for soliciting patient feedback about referrals or building relationships with the specialty care providers themselves.
As a result, we created a web-based software app to easily store, search, and update information about referral sites.A literature search did not identify any existing software tools designed for this purpose at free clinics.In this paper, we describe the process of building the app, our pilot testing of the system, and how it has helped our clinics.

Methods
Two medical student authors designed a password-protected, web-based app called iRefer to aid in the matching of patients to the most appropriate specialty care provider.We used a Microsoft Access database to store provider information and ASP.NET, JQuery as the web-based scripting language.Printable, patient-friendly Microsoft Word documents describing each provider are stored on a Microsoft Windows web server file system.Analytics software Google Analytics is used to track the number of times documents are printed.The password authentication uses the UCLA username and password that is used throughout the University.

Pilot Testing the iRefer App
Pilot testing of iRefer was conducted prior to its release at MCP.We compiled written adaptations of past clinical cases and asked participants to decide the appropriate referral with or without access to the app.
Case scenarios were adapted from six real patient encounters over a four-week period at MCP (Appendix 1).Scenarios were written before the app was created to reduce the potential for bias.Student executives of the referral committee reviewed the cases and selected the three most appropriate referral sites for each patient.
Two groups of participants were recruited: 1) members of the referral committee (excluding those who reviewed the case scenarios prior to the pilot) and 2) MCP volunteers who were not members of the referral committee.While referral committee members had extensive experience with the clinic's referral protocols, non-committee members did not have prior training in referral site details.Participants were then provided with three scenarios to complete using the app and the other three to complete without the app.For each participant, it was randomly assigned which scenarios would be completed under which testing environment.A participant's referral selection was considered "correct" if it was one of the three preselected choices of the executive reviewers.A time limit of 7 minutes was enforced, which is similar to the amount of time between referral consultations at our free clinics.Statistical analysis for statistical significance of data was conducted using Fisher's exact test.

Evaluating the Impact of iRefer
Three months after iRefer was released for use at our free clinics, a written survey to evaluate the app's usability and impact was administered to all staff members who used iRefer at least one time.The four statements were: 1. iRefer is easy to use 2. iRefer helps me make more appropriate referrals 3. iRefer helps me make referrals more quickly 4. iRefer improves the quality of care ultimately provided to patients at MCP Participants were asked to provide responses on a 5-point Likert scale where 1 means "strongly disagree" and 5 means "strongly agree."

Building the App
The design of iRefer was conceived by two medical student authors and implemented by a software engineer employed by UCLA's technology services department.Its key feature allows volunteers to enter information about a patient's needs and returns recommendations for the most appropriate sites to refer the patient.
A Microsoft Access database includes an entry for each of the 141 specialty providers to which MCP has referred patients in the past three years.Information about referral sites was obtained over the phone or through site visits.Each site is labeled with a single "service category" that indicates the main service it offers and multiple "additional services" that indicate any further types of care it provides.All sites are also assigned a number of stars from 0 to 5 that reflects volunteers' subjective rating of how the provider performs relative to other members of its service category.Updates to the database, including adding and deleting entries, can be made at any time.
When the app is opened, users are prompted to enter the patient's age, gender, insurance status, and whether the patient possesses a form of identification (Figure 1).
A second page prompts for the category of service the patient requires (Figure 2).
Only one category can be selected; if multiple services are needed, the most important one is selected.A third page displays all of the additional services offered by providers in the selected category (Figure 3).
For example, if "Medical" is the chosen category, the third page will display amenities that medical providers offer (e.g.immunizations, health insurance enrollment), multiple of which can be selected.
The fourth page displays results (Figure 4).A referral site is included in the display if it is in the "service category," offers at least one of the selected additional services (or none if no additional service was selected), and accepts patients with the selected age, gender, and insurance/identification status.Results can be sorted by distance from the clinic, number of additional services matched, or number of stars.
Clicking a provider from the display page opens a one-page document that includes basic information about the referral site including address, phone number, services offered, eligibility requirements, and directions.A "print" button allows for wireless printing of the document to be given to the patient.Analytics software from Google Analytics records the number of times each document is printed.
Year-to-year maintenance of the application is overseen by a designated "iRefer Team" consisting of 1-3 students.This team is responsible for reviewing patient and user feedback, determining which suggestions are mostly likely to improve the system, and executing those changes to the database.Every three months, the iRefer Team reviews all of the data in the database to ensure accuracy.Prior to graduating, each iRefer Team member trains a successor to facilitate smooth transition.

Pilot test results
Fifteen MCP volunteers who were either referrals committee members (n=8) or non-members (n=7) participated in the pilot.When using iRefer, committee members made a correct referral 91.7% of the time while non-committee members were correct 81.0% of the time (p=0.04).Without iRefer, committee members made a correct referral 83.3% of the time while non-committee members were correct only 38.1% of the time (p<0.01).The distribution of correct answers for each scenario is illustrated in Figure 5.  Results from Use at Clinic iRefer has been in use at MCP for 6 months (November 5, 2015 to May 5, 2016).During this period, the clinic was open 25 Wednesdays and 25 Saturdays, and had 602 patient encounters.Over this period, 256 encounters (42.5%) resulted in a referral to a specialty provider.A total of 360 referrals were made since many patients received referrals for multiple services at a single encounter.Table 1 shows the most common reasons for referral.

Discussion
Student-run free clinics serve as an important primary point of care for the uninsured or homeless population in a community.Like other primary care providers, SRFCs rely on specialty care provider referrals to further the care of patients whose needs exceed the capacity of the clinic. 8Managing these referrals is a resource-intensive task, often requiring a dedicated committee and hours of personnel training.Improving the referrals process can allow an SRFC to not only save time, but also connect patients with a specialty care provider better suited to address their needs.
The iRefer app was created as an alternative to existing processes of managing referral site information that often require time-intensive memorization or manual search of paper documents.The system was designed such that non-technical users could operate and update the program with minimal upkeep by a computer expert.Ease of updating, in particular, was a priority since information about off-site providers changes frequently.The app also tracks how many times the clinic makes referrals to each provider, which has allowed us to focus on building relationships with those providers to whom we refer the most patients.An additional benefit of iRefer is that it lets us capture valuable patient feedback about their referral experience.In the past, we have asked patients to complete surveys about their visit to a specialty provider; that data was shared with volunteers anecdotally but never officially incorpo-

Referrals
No App iRefer rated into the way the clinic made referral decisions.Now, by setting the number of stars for a specialty care provider on iRefer, volunteers can translate patient feedback into a rating that can influence future referrals.In addition, wireless printing of patient-friendly referral sheets directly from the iRefer interface allows the clinic to save paper and avoid carrying heavy boxes of documents to each session.Quality control of the database over time is performed by a designated iRefer Team, which processes feedback and periodically ensures the accuracy of data.This team is essential to iRefer's success, as quality control, central leadership, and smooth hand-offs of information between transitioning staff are necessary for the long-term survival of a knowledge management system. 9he pilot test was designed to evaluate 1) how access to iRefer affects volunteers' referral decisions and 2) if separate groups of volunteers (referrals committee members vs. non-committee members) perform differently.The key difference between the two groups of participants is their level of training; referrals committee members spend 20+ hours per year studying information about specialists while non-committee members spend <1 hour per year.There was a significant difference (p<0.01) in performance between committee members and non-members when not using the app (83.3% vs. 38.1%),indicating the benefit of the existing referrals-specific training program.When given access to the app, committee members also saw significant benefit (91.7% with app vs. 83.3%without app, p=0.04) but to a lesser degree; the smaller magnitude of benefit in this group is likely because their previous memorization of the referrals information left little room for improvement.Non-committee members, on the other hand, improved when using iRefer (81.0% with app vs. 38.1% without app) to a level on par with the performance of committee members (p<0.01).Access to iRefer essentially eliminated the advantage conferred by committee members' prior training.Thus, the key benefit of iRefer is not necessarily an improvement in referral decisions compared to current methods but a dramatic reduction in the amount of training time needed to achieve those results.Based on these findings, the referrals committee has already begun to shift their focus away from memorizing site data and instead onto keeping information updated and investigating new sites.
Volunteers using iRefer at MCP sessions reported the app was easy to use and improved the speed and accuracy of their referral decisions, indicating that there were no unforeseen barriers to implementing iRefer in a clinical setting as compared to a simulated, pilot-testing environment.
iRefer and the evaluation of iRefer have a number of limitations.The pilot survey evaluating the usability of iRefer is limited by a very small sample size.Furthermore, evaluating the appropriateness of a referral, as we did in our pilot test, is an inherently subjective endeavor.Although efforts were taken to minimize subjectivity (e.g. three different answers were counted as "correct"), we acknowledge that referral sites marked as "incorrect" may have provided equally excellent care to the patient described in the scenario.One limitation of the app itself is that it does not provide a way to store patient-specific data due to legal complications we encountered when attempting to include this feature.As a result, a separate log must be kept of which patients are referred to which sites.In addition, although iRefer could potentially be used by anyone in Los Angeles to locate specialty care, network security requirements at UCLA mean the app is currently only available to those with a UCLA username/password.We hope in the future to host iRefer in such a way that makes it accessible to anyone in the community.

Conclusion
The iRefer app was viewed by nearly all users as an effective tool to improve the quality and efficiency of referrals at MCP.The app allows volunteers to electronically search information about referral sites and provides recommendations about the most appropriate sites based on patient information.Patient feedback about their visits to off-site providers can be incorporated into iRefer so that well-reviewed sites will be more highly recommended in future patient encounters.The system stores information about referrals made using iRefer in an analytics report that can be used by clinic administrators to modify referral practices.Overall, the iRefer app streamlined the referrals process at MCP and such a system should be considered at other free clinics.

Appendix 1. Case Scenarios Used in the Evaluation of iRefer
Patient 1 is a 25-year old male veteran who recently injured his arm while working a temporary construction job in Westwood.He initially tried to ignore the pain but it has gotten worse and he is worried that it is broken.You and the attending physician try to assess his arm but the patient pulls it back in pain.You're not sure if it's broken and feel that an X-ray will have to be performed to determine if the patient needs more serious care.The patient agrees that an X-ray would be helpful, although he is worried that because he does not have insurance that he will be ineligible for assistance.He has a valid California ID.He mentions that he's usually around Beverly Hills or Westwood and would like to receive care nearby.Because he is a veteran, he would also prefer to be referred to a clinic with experience caring for this population, if possible.Where will you refer Patient 1? Patient 2 is a 27-year old single mother of 5-year old twin boys, who recently became homeless.The family has been living out of the patient's car for the past week while she searches for permanent supportive housing relief.She's wondering if you can help.After asking more questions, you find out that she has a valid ID and is enrolled in Medical.If possible, she would prefer a shelter located Downtown since she knows a few people Downtown who "might be able to help me get a job".Because she has two young kids, she is hoping to find a place that is family-friendly.Where will you refer Patient 2? Patient 3 is a 30-year old mother of two who is coming to your clinic after having just left an abusive relationship with her boyfriend two weeks ago.She has heard that the community provides resources for victims of domestic violence but does not know where to start to seek support.While she does not feel that she is in immediate danger, she is worried about the legal status of her children since her boyfriend has mentioned repeatedly that he will attempt to gain visitation rights.She is not sure how the law works regarding child visitation and would like to consult with a legal expert.She is currently living in Culver City and would like to receive service as close as possible.She describes her current housing situation as "temporary and probably not good for the long haul" and would be interested in a domestic violence support resource that also offered shelter as well if such a center exists.She has ID and insurance.Where will you refer Patient 3? Patient 4 is a 65-year old man who arrives at clinic seeming to be in an altered state of mind.While he does not seem to be aggressive or dangerous, he is relatively incoherent.He mentions that he took a dose of a recreational drug (he doesn't mention which drug) earlier that day and is worried about the effects it is having on his body, since he has never felt this intensity of reaction before.He would like to come down off the drug and wants to know if there is anywhere he can go to help with the detoxification process.He lives in Downtown and would like to seek care in that area.Where will you refer Patient 4? Patient 5 is a 45-year old man who has been homeless for the past two months.He is HIV+ but mentions that he has received medical attention for this before and his symptoms are well-controlled.He would like to find shelter for the night.He tells you that he has stayed in several different shelters before and had some good experiences and some bad.His best experiences have been at those shelters that were geared specifically toward patients w HIV/AIDS, as the staff and clients at those shelters have been supportive and provided him with important resources and information to deal w his disease.He would like to find such a shelter if the volunteer knows of one.He would also like to stay near Downtown but is often to go much further if it means a safe bed for the night.Where will you refer Patient 5? Patient 6 is a 60-year old female who injured her right leg three weeks ago after a fall.She visited the emergency room immediately after the fall and was told that nothing was broken and that she should try to rest it.Because she is homeless and often walks many miles a day, resting her leg has been a challenge.The pain is getting worse and, after consulting with the attending physician at your student-run homeless clinic, you decide your patient should receive physical therapy.The patient agrees that this would be helpful.She mentions that she has no form of identification and also no insurance, although she would be very interested in enrolling in MediCal if there is such a clinic that might offer this service.She would prefer to receive care somewhere west of Westwood but is willing to travel further.Where will you refer Patient 6?

Table 1 .
Breakdown of Referrals