IIIT-H researchers turn AI lens on Indian thali, biryani to decode food culture
HYDERABAD: Researchers at International Institute of Information Technology Hyderabad are using artificial intelligence to understand Indian food in all its complexity from identifying mixed dishes on a thali to distinguishing varieties of biryani with the aim of preserving food culture and improving nutrition tracking.
The work, led by the Center for Visual Information Technology at IIIT-H, focuses on applying computer vision to Indian meals, which are far harder to analyse than standardised Western foods such as burgers or sandwiches. A typical Indian thali may contain rice, dal, roti, chutney and curd, often layered or mixed, making automated analysis difficult.
“If you are given a full plate of typical Indian food that not only has multiple dishes, but mixed ones like rice topped with dal, or a roti hidden under a papad, how do you understand what is there on a plate and eventually its nutritional value?” asked Prof. C V Jawahar, who is leading the project.
Why Indian food challenges AI
Most existing food tracking and nutrition apps rely on fixed menus and stable recipes, assumptions that rarely hold true for Indian cooking. Visual similarity across dishes and daily variations further complicate the task.
“In our everyday meals, sambar and dal can look alike one day because of similar ingredients like turmeric,” said Yash Arora, one of the researchers. “On another day, dal can turn green with the addition of palak. In cafeterias, menus can change overnight.”
According to the team, repeatedly retraining supervised models to handle such variations is impractical at scale.
A zero-shot approach
To address this, the researchers developed a zero shot system that can recognise new food items without retraining. Instead of classifying dishes from a fixed list, the system first identifies food regions on a plate and then matches them using retrieval based prototype comparisons.
“This makes the system scalable and flexible for real-world settings such as cafeterias and hospital messes,” Yash said.
The research, titled “What is there in an Indian thali”, was authored by Yash Arora and Aditya Arun under Prof. Jawahar’s guidance. It was presented at the 16th Indian Conference on Computer Vision, Graphics and Image Processing.
From hospitals to everyday use
The project began with a healthcare requirement, particularly the need to monitor nutrition intake for pregnant women. This shaped both the technical approach and the deployment model.
The team has built a working prototype that analyses images of Indian meals captured through an overhead camera at a kiosk. It can estimate the contents of a plate and assess nutritional components.
Looking ahead, the researchers plan to extend the system to a mobile app. To enable this, they have collected food images from multiple angles, allowing analysis through phone cameras rather than fixed installations. The team, however, acknowledges that several practical challenges remain unresolved.
