Measures & Metrics

Collection of SOTA ML quality assurance methods like bias, generalizations, robustness, uncertainty, and so forth about use cases and regulatory topics.


Scope

Since 2020, the Measures & Metrics workstream has researched regulatory aspects inside of Machine Learning for Health. We study a collection of SOTA ML quality assurance methods like bias, generalizations, robustness, uncertainty, and so forth about use cases and regulatory topics. Our expectation is finish the project until the end of 2022.

Today, our multidisciplinary team has students and professionals from different companies, laboratories, universities worldwide focusing on creating regulatory documents and providing better solutions to society.

We already developed a paper about Machine Learning for Health Audit in this workstream (see in publications). Our objective is to move to easy to maintain and use web resources.

Aims

Develop better regulations by publications, international standards about measures, and metrics inside Machine Learning for Health.

Outputs

  1. ITU/WHO
    Data and artificial intelligence assessment methods (DAISAM) reference
    Oala, Luis, Balachandran, Pradeep, Cabitza, Federico, Calderon Ramirez, Saul, Chiavegatto Filho, Alexandre, Eitel, Fabian, Extermann, Jérôme, Fehr, Jana, Ghozzi, Stephane, Gilli, Luca, Jaramillo-Gutierrez, Giovanna, Kester, Quist-Aphetsi, Kurapati, Shalini, Konigorski, Stefan, Krois, Joachim, Lippert, Christoph, Martin, Jörg, Merola, Alberto, Murchison, Andrew, Niehaus, Sebastian, Ritter, Kerstin, Samek, Wojciech, Sanguinetti, Bruno, Schwerk, Anne, and Srinivasan, Vignesh
    In Proceedings of the ITU/WHO Focus Group on Artificial Intelligence for Health (FG-AI4H) - Meeting I 2020
    Abs HTML PDF

Collaboration resources

You are welcome to inquire about the work stream and opporunities for collaboration directly with the work stream team.

  • General contact Luis Oala, luis@aiaudit.org

Meetings

Regular meetings for this work stream take place at the below coordinates.

Communication

You can subsbscribe to the work stream mailing list to receive updates and join the asynchronous group chat.

  • Group chat
  • Mailing list

Tools

We use different tools in our remote work. They include shared documents, github projects for code as well as task tracking and a collaborative whiteboard for ideation. You can request access via the below links.

  • Shared drive
  • Github project
  • Collaborative whiteboard

You can find more information about the way we usually carry out our work remotely in teams here.

Milestones


Important reference material

This is a list of related work and resources relevant for this work stream. It comprises resources the work stream contributors consider good practice.

  1. ITU/WHO
    Good practices for health applications of machine learning: Considerations for manufacturers and regulators
    Johner, Christian, Balachandran, Pradeep, Oala, Luis, Lee, Aaron .Y., Leite, Alixandro Werneck, Murchison, Andrew, Lin, Anle, Molnar, Christoph, Rumball-Smith, Juliet, Baird, Pat, Goldschmidt, Peter. G., Quartarolo, Pierre, Xu, Shan, Piechottka, Sven, and Hornberger, Zack
    In Proceedings of the ITU/WHO Focus Group on Artificial Intelligence for Health (FG-AI4H) - Meeting K 2021
    Abs HTML PDF
  2. ITU/WHO
    FG-AI4H Open Code Initiative - Evaluation and Reporting Package
    Schörverth, Elora, Vogler, Steffen, Balachandran, Pradeep, Leite, Alixandro Werneck, Li, Danny Xie, Ali, Kamran, Garcia, , Schneider, Dominik, Krois, Joachim, Lecoultre, Marc, Iyer, Shobha, Choudhary, Shruti, and Oala, Luis
    In Proceedings of the ITU/WHO Focus Group on Artificial Intelligence for Health (FG-AI4H) - Meeting K 2021
    Abs HTML PDF Code
  3. ITU/WHO
    Data and artificial intelligence assessment methods (DAISAM) reference
    Oala, Luis, Balachandran, Pradeep, Cabitza, Federico, Calderon Ramirez, Saul, Chiavegatto Filho, Alexandre, Eitel, Fabian, Extermann, Jérôme, Fehr, Jana, Ghozzi, Stephane, Gilli, Luca, Jaramillo-Gutierrez, Giovanna, Kester, Quist-Aphetsi, Kurapati, Shalini, Konigorski, Stefan, Krois, Joachim, Lippert, Christoph, Martin, Jörg, Merola, Alberto, Murchison, Andrew, Niehaus, Sebastian, Ritter, Kerstin, Samek, Wojciech, Sanguinetti, Bruno, Schwerk, Anne, and Srinivasan, Vignesh
    In Proceedings of the ITU/WHO Focus Group on Artificial Intelligence for Health (FG-AI4H) - Meeting I 2020
    Abs HTML PDF
  4. ITU/WHO
    Data and artificial intelligence assessment methods (DAISAM) Audit Reporting Template
    Verks, Boris, and Oala, Luis
    In Proceedings of the ITU/WHO Focus Group on Artificial Intelligence for Health (FG-AI4H) - Meeting J 2020
    Abs HTML PDF
  5. TODO
    TODO
    Last, First, and Last, First ...
    In 1900
    Abs HTML PDF Code
  6. TODO
    TODO
    Last, First, and Last, First ...
    In 1900
    Abs HTML PDF Code