Background information of the Local AI Medical Assistant

The Local AI Medical Assistant is basically a frontend to Google Health’s MedGemma model. This post aims to provide some background information on this healthcare model.  In the interests of transparency we include references to the model’s limitations.

The information contained in this post has been taken  from MedGemma/README.md at main · Google-Health/MedGemma · GitHub.  While the model is licensed under the Health AI Developer Foundations License, everything in MedGemma repository is licensed under the Apache 2.0 license. In the Local AI Medical Assistant the model runs within the Ollama framework.

MedGemma

MedGemma is a collection of Gemma 3 variants that are trained for performance on medical text and image comprehension. Developers can use MedGemma to accelerate building healthcare-based AI applications. MedGemma comes in two variants: a 4B multimodal version and a 27B text-only version. The Local AI Medical Assistant utilizes the 4B multimodal version which allows  both text and image input.

MedGemma 4B utilizes a SigLIP image encoder that has been specifically pre-trained on a variety of de-identified medical data, including chest X-rays, dermatology images, ophthalmology images, and histopathology slides. Its LLM component is trained on a diverse set of medical data, including radiology images, histopathology patches, ophthalmology images, dermatology images, and medical text.

MedGemma variants have been evaluated on a range of clinically relevant benchmarks to illustrate their baseline performance. These include both open benchmark datasets and curated datasets, with a focus on expert human evaluations for tasks.

The following sections present some common use cases for the model. You’re free to pursue any use case, as long as it adheres to the Health AI Developer Foundations terms of use.

Medical image interpretation

MedGemma’s pre-trained multimodal variants are well-suited for tasks like generating medical image reports or answering natural language questions about medical images. While its baseline performance is strong compared to similar models, MedGemma isn’t yet clinical-grade and will likely require further fine-tuning.

Medical text comprehension and clinical reasoning

MedGemma can be adapted for use cases that require medical knowledge. Such use cases may include patient interviewing, triaging, clinical decision support, and summarization MedGemma: 4b  has a strong baseline performance compared to similar models of their size, but developers should validate their adapted model’s performance and make necessary improvements before deploying in a production environment.

Adapting MedGemma

MedGemma is a developer model that requires validation on the developer’s intended use case. Based on those validation results, the user will likely need to further adapt the model to improve performance. Below are some types of adaptation developers can use to improve MedGemma’s performance for their use cases.

Prompt engineering/in-context learning

For certain use cases, MedGemma’s baseline performance may be sufficient after careful prompting, potentially including few-shot examples of desirable example responses within the prompt, in other words in-context learning. Prompt engineering may also use MedGemma to break the task into subtasks that can be performed separately. Adaptations using prompt engineering require the same level of validation as any other type of adaptation.

Fine-tuning

MedGemma can be fine-tuned for improved performance on the existing tasks it’s been trained on, or to add additional tasks to its repertoire. For an example of how to fine-tune MedGemma using LoRA (a parameter-efficient fine-tuning technique).

Full details about MedGemma 4b and MedGemma 27B can be found in the MedGemma Technical Report (https://arxiv.org/html/2507.05201v2) .  The abstract for this paper is reproduced below.

Abstract Artificial intelligence (AI) has significant potential in healthcare applications, but its training and deployment are challenging due to healthcare’s diverse data, complex spectrum of possible tasks, and the need to preserve privacy. Foundation models that perform well on various medical tasks and require less task-specific tuning data are critical to accelerating the development of AI for healthcare applications. In this technical report, we introduce MedGemma, a new collection of medical vision–language foundation models based on Gemma 3 4B and 27B. MedGemma demonstrates advanced medical understanding and reasoning on images and text, significantly exceeding the performance of similar-sized generative models and approaching the performance of task-specific models, while maintaining the general capabilities of the Gemma 3 base models. For out-of-distribution tasks, MedGemma achieves 2.6-10% improvements on medical multimodal question answering, 15.5-18.1% improvements on chest X-ray finding classification, and 10.8% improvement on agentic evaluations compared to the base models. Fine-tuning MedGemma further improves performance in subdomains, reducing errors in electronic health record information retrieval by 50% and reaching comparable performance to existing specialized state-of-the-art methods for pneumothorax classification and histopathology patch type classification. We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP. MedSigLIP powers the visual understanding capabilities of MedGemma and, as an encoder, it achieves performance comparable to or better than specialized medical image encoders. Taken together, the MedGemma collection provides a strong foundation of medical image and text capabilities, with potential to significantly accelerate medical research and development of downstream applications. More details about the MedGemma collection, including tutorials and instructions for downloading the model weights, can be found at https://goo.gle/MedGemma.