Google’s AI system AMIE showcases real-time video consultation abilities

When visiting a doctor, the consultation involves more than just verbal communication. Physicians pay attention to a patient’s cough, observe their walking pattern, or notice any obvious signs of discomfort. Now, Google Research and Google DeepMind are pushing the boundaries with AMIE, their research medical AI system, by showcasing its capability for real-time clinical video consultations. This demonstration is the first of its kind, exhibiting expert-level AI performance in this realm. AMIE is developed using Gemini and Project Astra and operates on a multi-agent architecture, enabling it to understand visual and auditory cues, guide virtual physical exams, and perform real-time diagnostic reasoning.

In a randomized study featuring simulated consultations with patient actors and a group of primary care physicians, AMIE was evaluated positively by clinical evaluators. It was assessed on several key clinical skills, such as the thoroughness of history-taking, accuracy in diagnosis, appropriateness of management, and communication quality. Additionally, patient actors showed a preference for the video-based experience over text chat.

Although AMIE is still a research system and further studies are necessary before it can be used responsibly in real-world clinical settings, it provides an intriguing glimpse into the future potential of AI in healthcare. More detailed information can be found on the Google Research blog.