Special Report: Understanding the Architecture of AI

To see how AI might fit into psychiatric practice, it is important to first understand the entire ecosystem—comprising data, technologies, and organizational decisions—that underpins AI system development.

Data: Foundation models are trained on large datasets (some estimated at trillions of tokens), which are often sourced from the internet and may be inherently biased. This bias is a result of the wide variety of content on the internet, created by and reflecting people with varying perspectives from diverse historical and cultural contexts.

For example, online discussions around mental health conditions may be associated with stigma and carry negative perceptions. General-purpose foundation models available today are not trained on clinical data. It is important to question how this data is collected, as these choices directly impact the model’s performance.

Compute: Training foundation models from scratch requires massive computing resources (some estimated at tens or hundreds of millions of dollars), far beyond what is typically available in clinical practices or even at many large health care institutions. This raises important questions about who has access to such resources and the power to develop and steer these models.

Model development: With necessary data and computing resources, foundation models are then developed and trained. This process often includes fine-tuning steps that incorporate additional data to refine the model.

For example, a commonly used strategy during the refinement or alignment process is called reinforcement learning from human feedback (RLHF) strategy. RLHF relies on a large number of potentially nonrepresentative sets of people who judge model outputs against abstract human values to reward the model toward generating a helpful, correct, nontoxic, safe, or otherwise “better” response. However, this process can introduce biases or unintended behaviors, such as excessive agreeableness or sycophantic tendencies. Moreover, this process assumes that gold standards can be agreed upon and established through human judges.

Model evaluation: After development, foundation models undergo rigorous evaluation, typically involving benchmark tests across various tasks, such as general language understanding and comprehension, translation, knowledge question-answering, math competitions, legal or medical question-answering, or code generation. Model evaluation can also include what has been called AI red-teaming—“a structured testing effort to find flaws and vulnerabilities in an AI system,” as the White House described it in an executive order on safely developing AI. Like RLHF, AI red-teaming involves a large number of people. Red-teaming activities may involve putting on adversarial personas to generate harmful content or pretending to interact with the system as someone with a mental health condition.

Achieving 100% safety may be difficult throughout development, refinement, and evaluation. In addition, it is important to consider who makes the decisions about data, resources, and processes, and whether they have the expertise to ensure that the models are suitable for mental health applications.

Model hosting: Foundation models are often made accessible through application programming interfaces (APIs) or cloud services hosted behind paywalls because there are computational costs associated with running the models and serving their predictions (meaning responding to the models’ predictive output). They often come with built-in policies and guardrails, which may reflect values of the hosting organization but could also be legally mandated through regulatory frameworks. The quality of both models and their hosting services must be assessed, as policies governing these models can change rapidly, obscured behind the facade of the APIs.

Application: At the application layer, generative AI models can be integrated into general-purpose applications like ChatGPT and Copilot as well as into specialized applications. Specialized applications in mental health care could potentially include virtual triage assistants, psychotherapy chatbots, clinical decision support, patient education and communication, provider training, and many others, with active and responsible participation from a range of institutions and startups. These applications are likely to be built on top of foundation models with additional functionalities, such as remembering prior conversations, connecting to databases, or extending their capabilities through plugins to enhance the experience.

The field is rapidly evolving, with new techniques and advancements continuously emerging to improve model capabilities and user interactions. These applications require not only technological development but also careful refinement and evaluation to ensure alignment with clinical goals and safety standards, considering the generative and indeterministic nature of the underlying models and decisions made along the way about the data, values, policies, availability, and application context.

Organizations, clinics, users: Most mental health professionals will interact with generative AI as users or decision makers rather than developers. They must be vigilant in understanding AI’s role in mental health care. Therefore, it is important to consider whether the decisions made in the development and deployment of these models align with the needs and expectations of mental health professionals and patients. ■