ITU/ WHO AI4H Assessment Platform
The ITU/WHO AI4H Assessment Platform is a software platform that provides a standardized end-to-end assessment framework for the evaluation of AI-based health applications.
Scope
The ITU/ WHO AI4H Assessment Platform is a software platform that supports the end-to-end process for assessing health AI algorithms at a global level for the benefit of health AI companies and regulators, who need to proof that a health AI product is fit for purpose in compliance with the medical, technological, and regulatory requirements.
The ITU/ WHO AI4H Assesment Platform is implemented as a open code software project and aims to produce the digital building blocks that comprises of the following 6 software packages - Data Acquisition Package (DAP), Data Storage Package (DP), an Annotation Package (AP), a Prediction Package (PP), an Evaluation Package (EP) and a Reporting Package (RP). The AI4H Assessment Platform is released under a modified BSD license and follows a standard development process.
Aiaudit.org is currently coordinating the implementation of two software packages of the Assesment Platform namely 1) Evaluation Package and 2) Reporting Package.
Evaluation Package
The Evaluation Package(EP) provides meaningful metrics that are agreed to be state-of-the-art and have the strongest expressibility to compare the performance of different AI models.
Aims
-
To develop testing measures and metrics for different quality dimensions, including interpretation, bias, uncertainty, and robustness.
-
To develop survey questionnaires to elicit information about the data preperation and ML model development processes with an aim for qualitative assessment of the applicability and safety of its solution outcomes
Reporting Package
The Reporting Package (RP) deals with the preparation and presentation of ML model evaluation results generated by the Evaluation Package (EP)
Aims
To provide a customizable reporting interface for supporting ease of comparison, classification and reproducibility of different AI4H model evaluation results
Outputs
Collaboration resources
You are welcome to inquire about the work stream and opporunities for collaboration directly with the work stream team.
- General contact Elora Schöerverth, elora@aiaudit.org, Alixandro Werneck, alixandro@aiaudit.org,Pradeep Balachandran, pradeep@aiaudit.org
Meetings
Regular meetings for this work stream take place at the below coordinates.
- Meetings Weekly, Tuesday, 4:00 PM, CET
- Zoom room Click here to join meeting
Communication
You can subsbscribe to the work stream mailing list to receive updates and join the asynchronous group chat.
- Group chat https://discord.gg/3RZEQfM6
- Mailing list
Tools
We use different tools in our remote work. They include shared documents, github projects for code as well as task tracking and a collaborative whiteboard for ideation. You can request access via the below links.
- Shared drive
- Github project
- Collaborative whiteboard https://miro.com/app/live-embed/o9J_lSKa6_w=/?moveToViewport=-1880,-657,3491,1785
You can find more information about the way we usually carry out our work remotely in teams here.
Milestones
Important reference material
This is a list of related work and resources relevant for this work stream. It comprises resources the work stream contributors consider good practice.