InterVLS: Interactive Model Understanding and Improvement with Vision-Language Surrogates

Author:

Jinbin Huang, Wenbin He, Liang Gou, Liu Ren, Chris Bryan

Keyword:

Computer Science, Artificial Intelligence, Artificial Intelligence (cs.AI), Computer Vision and Pattern Recognition (cs.CV), Human-Computer Interaction (cs.HC), Machine Learning (cs.LG)

journal:

date:

2023-11-05 16:00:00

Abstract

Deep learning models are widely used in critical applications, highlighting the need for pre-deployment model understanding and improvement. Visual concept-based methods, while increasingly used for this purpose, face challenges: (1) most concepts lack interpretability, (2) existing methods require model knowledge, often unavailable at run time. Additionally, (3) there lacks a no-code method for post-understanding model improvement. Addressing these, we present InterVLS. The system facilitates model understanding by discovering text-aligned concepts, measuring their influence with model-agnostic linear surrogates. Employing visual analytics, InterVLS offers concept-based explanations and performance insights. It enables users to adjust concept influences to update a model, facilitating no-code model improvement. We evaluate InterVLS in a user study, illustrating its functionality with two scenarios. Results indicates that InterVLS is effective to help users identify influential concepts to a model, gain insights and adjust concept influence to improve the model. We conclude with a discussion based on our study results.

PDF: InterVLS: Interactive Model Understanding and Improvement with Vision-Language Surrogates.pdf