The purpose of this study was to assess Ada, a smartphone app for diagnosing medical conditions, for its diagnostic accuracy, and its effect on emergency room (ER) patient outcomes. Through the use of a medical knowledge base, a machine learning system, and patient data, AI-based diagnostic tools can help improve specific aspects of healthcare delivery. Patients experiencing stomach pain were asked to self-evaluate their symptoms using the Ada-App before being seen by the emergency room physician. Comparisons between the App-diagnoses and the actual discharge diagnoses were used to determine diagnostic accuracy. Complications, overall survival, and length of hospital stay were all linked to delays in diagnosis and treatment. About 450 participants were enrolled and observed for 90 days in this prospective, double-blind trial. Compared to the traditional doctor-patient contact, in which the final discharge diagnosis is proposed in 80.9% of cases (95% CI [0.77, 0.84], P<0.001]), Ada was only able to advise it in 52.0% of cases (95% CI [0.47, 0.57]). When their diagnostic accuracy was evaluated jointly, however, Ada greatly improved the ER physician’s accuracy rate (87.3%, P<0.001) compared to when each was evaluated separately. Patients who were diagnosed and started on therapy right away had fewer problems and shorter hospital stays (P<0.001). At this time, the traditional patient-doctor relationship is more effective than a self-applied diagnostic tool based on artificial intelligence. However, AI techniques may also enhance physicians’ ability to make accurate diagnoses and boost care quality.