How our team helped speed up standard annotations for Computer Vision through human-in-the-loop AI models by developing new algorithms

The research group of the University of Toronto is working on tools to speed up standard annotations for computer vision. We’re a proud member of the research group, working closely with the leading expert in computer vision - prof. Sanja Fidler.

The Challenge

A modern approach to machine learning still heavily depends on annotated data, primarily connected through manual labor. It’s a time-consuming process prone to human error. We wanted to improve the process through human-in-the-loop AI models and new algorithms.

The Solution: Algorithms that assist manual annotators

Together with our partners, we have successfully developed algorithms that speed up the annotation process. Using recurrent and convolutional neural networks, the algorithms assist human annotators through both annotation creation and editing, while simultaneously learning to continuously improve from human assistance.

These state-of-the-art algorithms are fully integrated into the Toronto Annotation Platform, a web-based application that allows for larger, more efficient, and more accurate data annotation projects.

The result

Modern methods of machine learning depend heavily on extensive collections of annotated data, which to this day are still prepared with completely "manual" human work.

This solution outperforms the standard, manual annotation in both automatic (10% absolute and 16% relative improvement in mean Intersection over Union) and interactive modes (requiring 50% fewer clicks by annotators).

Project partners

• University of Toronto, (coordinator: Sanja Fidler, PhD, associate professor at the University of Toronto, director of AI at NVIDIA),
• Donna Shukaris, (VPRI, innovations & entrepreneurship manager),
• Abelium d.o.o., (head of development, Marko Boben, PhD)

Media posts and useful links

• Toronto Annotation Suite (TORAS) is available to the public!
• TORAS - presentation
• TORAS - how to use it

Similar projects

• Flare network - We are a core part of the Flare team, providing ongoing development of smart contracts for Flare Network for several decentralised on-chain mechanisms. We also provide the system's integrity and utility, leverage the distributed nature of the network, and enable its participants to act honestly while providing value and trust to the blockchain.
• INATrace in Rwanda - We developed a blockchain-based transparency and traceability technology, that makes coffee traceable from its production to the final package. The system confirms the coffee’s origin and quality, together with the fairness of its value chain, by tracking it all the way from the farmer to the end consumer.
• e-Su Register - We upgraded the e-Su Register information system by developing new functionalities, implementing EU standards and enabling parallelization to optimize the digital processes it supports
• One Stop Shop - SPOT - An all-in-one digital system that provides a supportive environment for entrepreneurs and easier access to all relevant information and funding. The platform was created and rolled out in cooperation with a public agency.
• GoOpti - A digital transformation of a transportation business into a cloud-based transportation marketplace, based on advanced algorithms. It’s a sustainable solution loved by end customers, as shown by the 93% satisfaction rate.
• 8ECM - We developed the congress’ website, digitalized processes, added analytics, and implemented its payment system.
• Multimedia portal SIO 2020 - We created a sustainable solution with three user-friendly, secure, and widely available platforms for video communication and management.
• ACE.Trade - We developed an entry-level app for crypto trading. It simplifies the first steps in the crypto world for initiatives and first time traders, while staying compliant with regulations.
• TAS - We actively participated in developing a Toronto Annotation Platform - a web-based application that allows for larger, more efficient, and more accurate data annotation projects.
• eSpis - Development and implementation of a digital system for reviewing court case files with support for search and filter functions to accelerate the process
• EO x Grid - Enhancing grid stability and renewable energy integration through accurate space-based PV forecasting. Funded under the European Space Agency's (ESA) ARTES 4.0 Downstream Applications program.
• EDISON WINCI - A project for developing the infrastructure, competencies, and elements of business models for logistical systems of contactless-charging electric vehicles.
• DIGITRUST - A digital transformation of a business with innovative digital solutions and a decentralized system
• Digital Solutions for Sustainable Cattle Farming - Developing advanced digital solutions that will help address the challenges of modern cattle farming. The main goal is to manage herds efficiently, improve animal welfare, and increase farm profitability by introducing advanced digital solutions.
• Di-Gozd - Development of an online and mobile application for a digital forest inventory
• ARRS J1-2481 - Mathematical and computational methods for polyhedral self-assembly.
• Phot2Bin - Detection of multiple stellar systems from single-epoch photometry.
• SOPOS - System for limiting the transmission of infections in buildings
• Photo2Chem - Stellar chemistry from photometric narrow-band imaging
• Infrastructure program I0-0047 - Upgrading services, providing support and maintaining scientific infrastructure
• Program group P1-0294 - Computationally intensive methods in theoretical computer science, discrete mathematics, combinatorial optimization, and numerical analysis and algebra with applications in natural and social sciences.
• KumuluzAPI - Establishing an API ecosystem of smart cities and communities
• PLASMA - A project for developing a banking and trading platform, upgraded with an innovative connection between the financial, fintech, and crypto worlds.
• SCI Journals - Managing, editing and hosting scientific and professional publications
• Kumuluz Crowdsensing - Researching new technologies for automatic data collection and the measurement of crowd data
• AI-FINREG - Regulation of modern FinTech services and protection of personal data
• Z7-7614 - Large networks in business systems analytics
• Z7-8271 - Improving the defibering process of mineral wool
• KKIPP - 4PR - A tool for creating intelligent digital technology platforms
• Da.Re. - The use of data science to restructure education