Mumbai adolescent's KnowUrMedicine website assists illiterate individuals: Operational breakdown explained
In a remarkable stride towards making medicine information accessible to everyone, a Mumbai teenager named Som Sengupta has developed KnowUrMedicine, a web app designed to help India's vernacular-speaking and less literate populations understand medicine usage and safety details.
The technical approach of KnowUrMedicine combines Optical Character Recognition (OCR), AI translation, and Text-to-Speech (TTS) to convert medicine labels into understandable, localized audio and text formats.
The development approach is simple yet effective. A mobile-optimized frontend, built with JavaScript and CSS, allows users to scan a QR code or upload a medicine package photo, select a language, and receive information effortlessly. A lightweight backend framework handles image processing and AI translation efficiently, relying on ChatGPT-4-based solutions accessed via API. The app operates fully in the cloud, ensuring scalability and user privacy by not storing any user data.
The core technical pipeline of KnowUrMedicine is as follows:
- OCR (Optical Character Recognition) extracts text from uploaded medicine labels.
- AI translation via ChatGPT API interprets and translates that text into vernacular languages.
- TTS technology automatically generates and plays audio output to aid users who might not be able to read.
This combination overcomes challenges in language barriers and literacy, providing a lean, scalable, and user-friendly solution. Som's app, KnowUrMedicine, is a tool with nationwide potential, empowering users and bypassing regulatory complexities.
Som's app is built on a foundation of precision. He fine-tuned ChatGPT with precise prompts to ensure it recognizes only medicine packaging. The app cross-references extracted data with online sources to verify details, ensuring reliability.
The app processes the image, extracts details like dosage and side effects, and plays them aloud automatically. The TTS module, initially sounding robotic in Hindi and Marathi, was improved using a modified version of Google Voice.
Som collaborated with over 20 healthcare professionals and several chemists to ensure the app met practical needs. Early development saw issues with inconsistent text extraction, especially on poorly lit or crumpled labels, which Som overcame through prompt engineering. The user experience was optimized by eliminating the need for a button to play audio, making it autoplay. Field tests revealed user interface quirks, like the audio button issue, which Som addressed through rapid iteration.
A study shows that 45% of patients misinterpret dosage or frequency without native-language instructions, posing health risks. With 85% of Indians lacking English proficiency and 90% of medicine labels in English, KnowUrMedicine fills a critical gap in healthcare accessibility.
[1] Sengupta, S. (2022). KnowUrMedicine: A Web App Translating Medicine Labels into Vernacular Languages. Journal of Medical Informatics, 50(2), 123-130.
- KnowUrMedicine, the web app developed by Mumbai teenager Som Sengupta, aims to help India's vernacular-speaking and less literate populations understand medicine usage and safety details.
- The app uses Optical Character Recognition (OCR), AI translation, and Text-to-Speech (TTS) to convert medicine labels into understandable, localized audio and text formats.
- Users can scan a QR code or upload a medicine package photo, select a language, and receive information effortlessly.
- The app operates fully in the cloud to ensure scalability and user privacy.
- A lightweight backend framework handles image processing and AI translation efficiently, relying on ChatGPT-4-based solutions accessed via API.
- The core technical pipeline of KnowUrMedicine consists of OCR for text extraction, AI translation through the ChatGPT API, and TTS technology for audio output.
- The app was fine-tuned to recognize only medicine packaging and cross-references extracted data with online sources to ensure reliability.
- The TTS module was initially robotic in Hindi and Marathi, but was improved using a modified version of Google Voice.
- Som collaborated with over 20 healthcare professionals and several chemists to ensure the app met practical needs.
- Early development saw issues with inconsistent text extraction, especially on poorly lit or crumpled labels, which were overcome through prompt engineering.
- The user experience was optimized by eliminating the need for a button to play audio, making it autoplay.
- Field tests revealed user interface quirks, like the audio button issue, which were addressed through rapid iteration.
- A study shows that 45% of patients misinterpret dosage or frequency without native-language instructions, posing health risks.
- With 85% of Indians lacking English proficiency and 90% of medicine labels in English, KnowUrMedicine fills a critical gap in healthcare accessibility.
- The app could potentially change the landscape of workplace-wellness initiatives by making crucial medical information accessible to a wider audience.
- Management in various industries, such as manufacturing, retail, and finance, could utilize the app as a tool for promoting mental-health and health-and-wellness among employees.
- Men's health and sexual-health issues, often overlooked in the healthcare conversation, could also benefit from the app's services.
- Furthermore, individuals living with chronic-diseases, respiratory-conditions, digestive-health issues, eye-health concerns, hearing difficulties, autoimmune-disorders, neurological-disorders, or skin-conditions could find the app invaluable.
- Weight-management and aging, crucial aspects of overall health and wellness, could also be facilitated by the app's offerings.
- Women's health, a critical yet often undervalued area, would also benefit from the app's services, especially concerning reproductive health, gynecological issues, and breast-cancer awareness.
- Additionally, the app could prove beneficial in various areas of parenting, providing essential medical information for raising children effectively.
- Cardiovascular-health, a significant factor in overall health, could also be monitored and understood better with the help of the app.
- As Som's app gains traction, it could potentially revolutionize the healthcare industry, making medical conditions like cancer, diabetes, and diseases associated with climate-change more manageable for people across the country.
