Let’s explore a specific example to clarify this idea. Below, you can see an explanation taken from the SHAP library, one of the most popular explanation methods. The model tries to predict median house value for California districts based on a number of features describing the district, such as median income, location (longitude and latitude), average number of rooms, etc.

For an individual district like the one depicted above, the explanation returns a ranked list of features with attached values that “explain” the effect they have on the output. A positive effect means the feature “pushes” the output towards higher values. Conversely, a negative effect pushes the output towards a lower value.

Explanations come in many shapes and forms and are constantly being developed. This area of research has been so active in recent years that it’s hard to organize the existing techniques into a coherent framework. Since research is so active, it’s entirely possible that new approaches will be developed sometime soon.

In this introductory post, I focus on providing a big picture of some of the most common techniques without delving too deep into the data visualization problems. In future posts, I will explore the visualization part more fully.

When we talk about explanations, we need to consider two separate aspects: the explanation type and the scope. Type pertains to the information extracted from the model and its form. Scope pertains to whether the explanations refer to individual decisions, subgroups, or the whole model.

Explanation Types

I like to group explanations into three broad types: feature weights, counterfactuals, and rules and trees. This categorization does not cover all possible methods, but it covers the most common ones.

Feature Weights

Features weights are numeric scores associated with each feature the model uses from the input to make a decision. The weights can be positive or negative, and their absolute value represents how much influence a feature has on the output. The image above, which I’ve described before, is an example of this type of explanation.

One way to think about weights is that they push the output in a given direction (positive or negative) with respect to the average output of the model and with a strength proportional to their amount. So, when we see a big positive weight and a high output value, we can attribute that value to that weight. This is a sort of simplistic mental model, but it’s good enough as a first approximation. As I mentioned in previous posts, features can be quantities or categories in a table but also individual pixels in an image or words in a text, and for these situations, the mental model breaks down a little. Another way to think about feature weights is as a kind of “emphasis.” If you want to understand the output, look at those features.

A special case of feature weights is a “saliency map.” Saliency maps are used with images to highlight regions of an image that are discriminative for the model to make decisions. Technically, they are often just the same feature weight mechanisms used for other data types (e.g., you can use general-purpose methods like SHAP or LIME to create saliency maps), but there are also some specialized ones designed specifically for image data.

The image below shows an example, once again, based on the same SHAP method I mentioned above.

Here, the features are individual pixels, and the red areas make the probability of the image being the animal on the left more likely. Similarly, the blue areas make that output less likely.

Counterfactuals

Another explanation method commonly used is counterfactual explanations. A counterfactual explanation of a single instance is the minimal set of changes one has to apply in order to change the output of the model. For example, imagine a bank's model to decide whether a customer should be given a loan. If the model predicts that the customer should not be given a loan, a counterfactual explanation provides information about what kind of changes would change the prediction from denied to accepted (there are a lot of ethical issues with this kind of decision that I am not going to discuss here). The image below shows an example (taken from a paper we published a while back).

The green arrows represent the changes necessary to flip the outcome from the current prediction to a different one.

While feature weights give information about which features drive the decisions, counterfactual explanations give information about what (minimal set of) changes produce a different outcome.

Rules and Trees

Rules and trees provide explanations in the form of logical structures such as “IF X is true and Y is true, THEN the predicted outcome is A.” The biggest advantage of this form of explanation is its highly interpretable format. Their meaning is self-explanatory: when the logical predicates of the antecedent part of the rule (X and Y) are true, then the consequent part is true (with a given probability and amount of confidence - two quantities that are often associated with rules). Trees are very similar except that they organize the predicates into branches of a tree. The figure below makes the concept easier to grasp.

Each node is a logical predicate, and each path from the root to a leaf node represents a rule. A rule is made of the predicates crossed in the path as antecedents and the outcomes of the leaves as consequents.

These logic structures work much better to describe the behavior of the entire model or parts of its input space rather than individual instances, although a rule can be used to explain a single instance by picking the rule (or multiple rules) that cover that instance.

Explanation Scope (Global, Local, or Subgroups)

One relevant aspect of explanations is at what level of granularity they operate. Some types of explanations are designed to provide information about individual instances, whereas others are designed to provide information about the whole model. For example, SHAP produces information at the level of individual instances, whereas surrogate rules and trees normally describe subsets of the input space or even the whole model.

This is particularly relevant for our purposes because different visualization approaches are necessary for entities representing individual instances versus whole models. As a general rule, any method focusing on individual instances will need a way to navigate the space of instances to decide which one to inspect. Common interaction and visualization strategies here include (visual) query languages (e.g., dynamic filters), visualization techniques to explore large sets of instances, and ways to aggregate the data to generate higher-level patterns. This is all a bit vague at this stage, but I will clarify it in future posts. For now, take this as a preview of what is possible to do in this space.

For methods that capture information about subsets or even the whole model, the data visualization challenge is typically more about visualizing the structures generated by the methods (e.g., how to visualize rules and decision trees) and navigating these data structures when they become large and complex. This is also a preview of something I will describe in more detail in a future post. For now, it suffices to know what kind of visualization challenges exist in this space.

Analytical Questions

When considering these classes of methods, it’s important to keep in mind that they tend to help answer different types of questions and are often complementary. I insist on knowing what type of questions one can answer because this aspect is often overlooked, and I think it’s crucial when deciding what method to use. The methods I outlined above answer these main questions.

• Feature weights: “What features drive this specific decision? Does this feature make this outcome more or less likely?”

• Counterfactual: “How can I change the values of this instance to obtain a different outcome?”

• Rules and tree: “What is the overall logic of the model? What features and values lead to specific types of outcomes?”

In reality, these methods can answer more questions. For example, when one aggregates feature weights from individual instances, it is possible to infer something about the whole model or parts of it. With rules and trees, it is also possible to focus on error analysis and look for specific subsets where the error rate is higher than expected.

Limitations and Misinterpretations

I want to conclude this post with a warning. Model explanations are models of models, and as such, they are not perfect! There are many ways in which model explanations can lead people to make incorrect inferences; therefore, it is very important to 1) understand what one can and cannot infer from the output of these methods and 2) remain vigilant and always verify the correctness of the information extracted.

A group of Microsoft Research researchers ran a very relevant study in 2020 to study how data scientists use some of these methods, and their findings are quite alarming. 

Kaur, Harmanpreet, et al. "Interpreting interpretability: understanding data scientists' use of interpretability tools for machine learning." Proceedings of the 2020 CHI conference on human factors in computing systems. 2020.

Despite the fact that they analyzed results from people who are experienced with machine learning, they found that these methods are often misused. This means that when using these methods, it is important to be extra careful in deriving correct interpretations of their results, especially when visualization is involved.

In my previous post of the series, I provided a broad-brush description of what kind of data is available in ML and how visualization can help extract useful information from it. In this post, I will focus on the first class of data sources I mentioned in the post, which is input-output data. In exploring visualization solutions in this area, I will mostly first focus on tabular data, that is, classic data sets with instances in rows and attributes in columns. Narrowing down the focus to this type of data will make it easier to describe the methods and the variations around the methods. At the end of this post, I will discuss ways in which the techniques I describe may extend or be adapted to other data types and highlight possible challenges we will need to overcome in the future to achieve that goal. The post is organized as follows: First, we cover what type of data we have with ML models. Second, we cover data visualization techniques for different data configurations. Third, we clarify what type of questions one may or may not be able to answer with these methods. Fourth, in the final part, we cover open issues and challenges. Enjoy the reading!

Model Data

When we observe models from the perspective of input and output we can identify the following primary sources of information:

• Data features and values. This is just the data used to train and test the model and the data that the model will eventually receive when used to make the actual prediction in production. If we focus exclusively on tabular data, the features are just nominal, ordinal, or quantitative variables with their associated value domains. For example, a customer data set may have demographic and behavioral data like age, region of residence, years the person has been a customer, visit frequency, etc.

• Model output (classes, scores, quantities). This is the output generated by the model when receiving a specific input (often called a “prediction” even if it does not have to be about predicting a future event). There are many possible types of outputs; here we will first focus on two basic types, classes and quantities. Classes (nominal or ordinal values) are the output of classifiers, and quantities are the output of regressors. One additional element to consider is the model scores many classifiers use to produce a specific class as an output. Many classifiers do not directly provide a specific predicted class as an output but a set of scores for each class that determine the predicted likelihood of a class. This is important because many data visualizations use these scores directly rather than the predicted class.

• Ground truth (and errors). Most machine learning models are trained using data that record past outputs so that the model can learn to mimic predictions in the future. For this reason, in ML, we often have data sets with “ground truth,” that is, the prediction the model should be able to mimic if it has learned to make predictions correctly. As you can imagine this piece of information is essential because it’s the one that allows us to discriminate between correct and incorrect predictions. In practice, this means that for each element in the data table (when we use training or test data), we have two fundamental values, the model prediction and the correct actual value. The difference between these two is essential to understanding where and when the model makes mistakes.

In addition to these primary sources, we have to also consider what kind of data we can derive from these sources. In other words, how can we transform these primary sources to generate useful information that derives from them? We will consider two main types of transformations: transformations of data features and values and transformations of model output. As we will see later on, transformations of the data input are essential to focus on specific aspects of the data and the model. In particular, the data input can be processed to derive the following structures:

• Data cubes: Groups of data formed by segmenting the data according to the values of one or more attributes.

• Data clusters: Groups of data formed by using a clustering algorithm that groups data items according to a similarity function.

• Data embeddings: Projection of the data into a low-dimensional space characterized by (typically) 2 or 3 axes that capture the main structure and variation of the data.

One final characterization we need is to distinguish between training, test, and production data and also between (output) data coming from different models. These distinctions are important because visualization tools can support the comparison of these different data sources.

Visualization Techniques

When we look at these characteristics we can start reasoning about what kind of visualizations one could build and for what purpose. We have two main classes of visualizations.

Visualizations that focus on model output (error analysis)

These visualizations focus exclusively on model output and the errors the model generates. Before focusing on visual representations, it’s important to characterize even further what kind of output and errors are possible because these affect what kind of visualization techniques are available for error analysis.

One way to look at this is to identify what kind of information we can gather from the output generated by individual instances. We have three possible cases according to what is predicted and what is the ground truth:

• Predicted class + ground truth class

• Predicted score(s) + ground truth class

• Predicted quantity + ground truth quantity

Data and error distributions

The simplest analysis is to look at how output and errors are distributed. How many data points are there in each class/value? How many errors are there in each one?

When the output is a single class or value, bar charts and histograms (or density plots) are the best representations for these simple cases. A bar chart can show the number of data points in each class and the number of errors (a stacked bar chart can do that). A histogram or density plot can do the same when the output of a model is a quantity. If one wants to look at specific instances rather than aggregate values, the strategy used in Model Tracker is a possible solution. The data items are arranged along the horizontal axis according to their model score. If the model output is discrete, a dot plot or any of the many variants of strip plots can be used for the same purpose.

One trickier case is when we want to visualize the quantitative scores generated by a classifier to decide which class is the most probable. In this case, the output is not just one single value but a multidimensional value. For this situation, multidimensional visualization techniques such as parallel coordinates, scatter plot matrices, and multidimensional glyphs can be used to visualize model output. The image below shows the Squares tool, which employs a bespoke version of parallel coordinates that enables the analysis of model output for this specific case.

Actual vs. predicted comparison

Another type of analysis one can do is to look at how the actual and predicted outcomes compare. When the model is a classifier, the output is a matrix of values with actual values in rows and predicted values in columns (or vice-versa). This kind of matrix is commonly called a “confusion matrix” and can be easily visualized as a matrix visualization. When the model is a regressor, one can do the same type of analysis by either binning the values and using the confusion matrix strategy or simply using a scatter plot, with one axis for the actual and one for the predicted values. A useful variant for this last case is a residual plot where instead of visualizing the predicted values, one visualizes the distance between the predicted and actual values.

One complication for confusion matrices is the very high number of classes. In that case, the matrix may not scale very well, and additional strategies are needed to make it more effective. A good example is Blocks, a visualization system developed to address this specific case. In Blocks, the hierarchical nature of the classes is used to group rows and columns together. The tool also employs matrix sorting strategies to make it easier to detect trends in how the model confuses certain groups of classes with others.

Visualizations that relate input and output (inferred logic and subset analysis)

The analysis of model data can also focus on the relationship that exists between input and output data. Relating input to output permits us to learn more about how the model behaves in specific conditions other than what type of errors it makes. The existing methods can be grouped according to three types of data organizations.

Feature-outcome plots

The most basic relationship we can inspect is the one between feature values and model outcomes. Depending on what types of features and outcomes one inspects, we can have different types of data arrangements and associated visualizations. In general, we have all the possible combinations between nominal, ordinal, and quantitative features and outcomes, therefore a total of nine types of relationships. But if we collapse nominal and ordinal together (by just remembering to keep the order of categories intact) we have only four possible combinations by matching quantitative/categorical input and quantitative/categorical output. To visualize these data it is sufficient to use a set of standard plots (my favorites!):

• Quantitative input + quantitative output: Scatter plots

• Categorical input + quantitative output: Dot plots or bar charts (not scaling well)

• Quantitative input + categorical output: Stacked area chart (or?)

• Categorical input + categorical output: Stacked bar charts or heatmaps/matrices

Visualizing these data, however, is more complex than it seems because there are several possible complications. The first one is that we often want to distinguish between correct and incorrect predictions, therefore all the plots above need to accommodate a comparison between these two sets. The second complication is that categorical inputs or outputs can have a high number of categories. When this happens some of the visualization techniques outlined above do not scale well. More specifically, bar charts, stacked area charts, stacked bar charts, and (to a lesser extent) heatmaps/matrices do not scale well when they have to accommodate a high number of categories. I will devote a separate post to this problem, but for now, it is important to be aware of the fact that this problem exists and is very concrete because many real-world data sets can have features and outcomes with a high number of categories.

A special case of input-output plots is PDP plots. These plots are built through a two-step procedure. In the first step, each data point is modified by changing the value of the feature being plotted (in a range between the domain’s minimum and maximum value) and the output of the model with the modified data point is recorded. In the second step, all the values are aggregated to generate a mean model response for a given value of the input feature. The result is one plot for each feature that depicts the relationship between input and output with a line chart. The image below shows an example with a tool called PDPilot which we developed in our lab (this is also Daniel’s work).

Each plot in this grid represents one of the features in the data set, and the black lines show the relationship between input and output calculated with the partial dependence plot procedure (the green lines represent the individual data points with the extrapolation across the feature value range described above).

Data cubes and clusters

When you combine multiple features at once, you obtain what is called a “data cube” in databases. The idea is that the data space can be partitioned into a data cube if we split the space according to the values of three data attributes. Imagine a data set with demographic data. The data space can be partitioned into small cells if we use the combinations of values of age, gender, and state. The same idea can be extended to combinations of any number of data features. This is relevant in our context because analyzing the behavior of a model in data cubes can be insightful. Imagine, for instance, if our goal is to find data subsets where model performance degrades considerably. Imagine a situation with a customer data set where the model's overall accuracy is high but much lower with a specific subpopulation of interest. That’s certainly something that can be a reason for concern. This is exactly the problem addressed in a 2019 paper titled “Slice finder: Automated data slicing for model validation,1” which introduces an algorithm to look for such subpopulations where performance degrades. A more visualization-oriented approach can address the same problem by creating visualizations that partition the data space according to selected data features. I can count at least three papers where this approach is used (SliceLens was developed in our lab by my student Daniel Kerrigan):

• Visual exploration of machine learning results using data cube analysis2.

• The what-if tool: Interactive probing of machine learning models3.

• SliceLens: Guided Exploration of Machine Learning Datasets4.

Here is an image from the What-If tool developed by Google, which includes visualization to carry out this type of analysis.

The data cubes approach is one of many ways to build subgroups from data. Another approach is to use clustering algorithms that group data according to a similarity function. These algorithms analyze the data and produce groups in which the data points tend to be similar according to the definition of similarity used in the similarity function. This is sometimes useful when we look for ways to group the data and want to use something other than a specific set of features to create data cubes. Once the clustering method returns the groups, one can perform the same type of analysis used for data cubes, which is mostly about inspecting and comparing model performance within and between groups.

Embeddings

One final option is to visualize data through “embeddings.” An embedding is obtained by transforming the input space into a new space that “describes” the data with a reduced number of synthetic features. Dimensionality reduction methods permit the creation of such embeddings. They they the original data set as an input and produce a user-defined set of new axes (features) one can use to project the data in 2D or 3D space. An example here will be more evocative than many words. In the image below you can see an example of embedding with the classic MNIST data set.

The method receives the pixel values of each image as an input and produces two or three axes to project the data in a point cloud visualization. As you can see the method preserves most of the structure. Interestingly, the visualization also helps detect potentially critical cases. Do you see the red squares in a cloud of green or cyan squares? This is where embeddings can be useful. They can help detect edge cases and areas of the input space where a model might have difficulties in making predictions. These methods can be used with the input data exclusively, even before the data are used to train a model, but they can also be used to see how the model makes decisions in this space and how errors are distributed. For example, a region where errors concentrate could be a hint that the model needs adjustments specific to the set of data points that lie in that region.

What can we learn from these visualizations?

Now that you have a full picture of what information and corresponding visualizations are available it is important to take a step back and reflect on what can and cannot be learned from analyzing models with these techniques.

Visualizations that focus exclusively on model output can answer questions regarding how (training, test, and production) data and errors are distributed across the output space. For example, these methods can help identify predictions that are particularly problematic, either because items of one class are often confused with others or because certain quantities are harder to estimate. In turn, this information can lead to different actions according to what the source of the problem is. In some cases, problems can stem from inaccurate labeling of the data or extreme outliers. In other cases, one may realize that the model does not have enough training data to learn to make a specific kind of prediction.

Visualizations that focus on input and output carry additional information that relates the input and the output space. For example, it is possible to learn that errors concentrate in a specific subpopulation of interest or that model output tends to have a specific relationship with model output. This is where ML visualizations can be helpful in testing the mental model of the user and identifying model behaviors that are counterintuitive or do not make sense. This is also where users can start drawing hypotheses about what cases are problematic for the model and produce additional testing (maybe with synthetic data) to see if the model makes systematic mistakes.

This second class of visualizations is characterized by the fact they help users draw inferences about model behavior and logic. What one needs to keep in mind, however, is that inferences made from data could always be inaccurate or even wrong, therefore further testing of the hypotheses and mental models generated from using these data visualizations is necessary.

Open issues and challenges

The techniques I described do not cover all possible situations, and they can easily break down when applied to more complex situations. One problem I already mentioned is scalability. When the number of classes or data items to show is too high, visualizations can easily reach a visual scalability limit.

Another big issue is how to analyze model data with other data types. In the beginning, I mentioned that all these methods apply to cases where the data handled by the model is tabular data, do these techniques work with other data types? It depends. The techniques that focus on model output can be applied to any other case where the output is a class or a quantity. However, some models have different or more complex types of outputs. For example, time series forecasting has a whole time series as an output. Machine learning methods that produce ranked lists have a whole ranking as an output. Some models also produce a probability and an associated level of uncertainty. All in all, we can’t assume that models produce only these simple outputs we covered and specific adaptations are needed to cover other cases.

The problem becomes even more complicated when we consider methods that associate the input and the output space. This is where the nature of the data can make a big difference because we can no longer treat data as a collection of features and associated values. Images, videos, text, time series, etc., all have completely different structures. Embedding can be used seamlessly if one has a way to calculate a distance function between the data objects but the analysis based on features does not apply easily. Similarly, the data cubes analysis may or may not apply depending on the nature of the data (image metadata can be used for data cubes for example).

Finally, I want to mention that more and more problems in ML are configured as an unbounded input and output space, and it is not at all evident how model data visualizations could be applied to these cases. Models that handle unbounded input and/or output spaces require a complete rethinking of the problem.

Thanks for reading until the end! Please like the post and leave a comment. It’s very useful for me to learn about my readers’ thoughts about the articles I post here.

Leave a comment

If you are not a subscriber, sign up to receive updates when I post new articles here. If you like what I am writing, please help me spread the word by letting your friends and colleagues know about this series and the newsletter.

Share

This is Post #3 of my series on Visualization for Machine Learning. If you want to know more about the series, start from the introductory post, which also includes links to the posts published in the series so far (this one included). If you are new to this newsletter and want to receive future updates, subscribe now to receive the updates directly in your inbox.

Subscribe now

In my previous posts, I mentioned the idea of organizing the series around what data is available to visualize coming from ML models. In this post, I will provide a first overview of this organizing principle. My intent is to give you a better sense of what I have in mind without providing too many details yet. This little framework will help us place specific visualization ideas in a broader context. The framework includes three main sources of information derived from machine learning models.

1. Model input-output.

2. Model explanations.

3. Model components.

Let’s take a look at each one …

Model Input-Output

Since models generate some output for any given input, visualization can be used to depict data generated from the model and any derivation of these data. Since input and output are logically connected, visualizations can focus on these data to describe model behavior and make inferences about its logic. One can, for example, visualize how different characteristics of the data correlate to specific outputs and the number of errors the model produces. A common example is multidimensional projections, which form a sort of “map” or “landscape” of the data space and can help generate glimpses of how the model behaves in different “regions” of the data space (we will explore these and other techniques in way more detail in a future post).

The image above shows an example from the “embedding projector” tool developed by Google and available within Tensorflow.

Model explanations

Model explanations are ML methods that help people gain a better understanding of the model’s logic. I will explain these methods and their various intents and applications in more detail in a future post when we will cover data visualizations based on explanations. For now, one way to think about them is that they are a sort of “models of models” and that they help with extracting information from models that is hard to draw exclusively from looking at the input-output behavior. Very often these methods try to estimate the importance of different data attributes or features, but they are not limited to that goal. Visualization can be used to explain specific data instances or entire groups of data. A good example we will examine in more depth in future posts is the technique called SHAP and its associated visualizations, which provide information about the weight a model gives to the features it uses.

This is an example coming from the SHAP library showing the importance of each feature for each individual data point in the data set.

Model components

Finally, one also wants to look inside models to understand how they work and what role individual components play. For traditional models with a somewhat intelligible structure, visualizing model structure and components comes naturally and exposes some of the internal logic directly. For more complex models, like those based on deep learning, visualizing the behavior of individual components or parts of the architecture can help understand their role in producing complex tasks. In language models, for example, it is common to visualize the “attention” mechanism, which represents which of the preceding words the model uses the most to predict the next token.

The image above, coming from the Bertviz tool, provides a common example of visualizations of the “attention mechanism,” which permits us to understand which, among the preceding words, the models use the most to make the next word prediction.

So, summarizing, once a model has been trained we can identify the following elements. Data comes into a model as an input and produces some output. The data produced from the model, as well as the model itself, can be used to generate model explanations. If you look inside the model, you can find various components, a bit like the engine of a car or a plane. For this reason, we also consider model components and the information they receive or produce. That’s the material we work with when we visualize machine learning1!

In practice, this subdivision may not be as neat as what I presented here. For example, one could have methods that integrate or relate model explanations with model structure or model data with model explanations. But as a first approximation, these three categories help us organize the existing work in a useful logical structure.

One additional observation is that these three types of data correspond to the three types of focus I outlined in my previous post, which are understanding model behavior, logic, and mechanics.

Data types

In the description above, I did not consider what type of data a model handles, which typically corresponds to what type of data has been used to train it. These are some of the most common data types:

• Tabular: Think spreadsheet. A data set is made of rows and columns, where the rows represent some objects of interest and the columns have some attributes that describe properties of these objects.

• Temporal: This is often a tabular data set with temporal information added. That is, it is a data set where the values are anchored to and depend on specific times.

• Spatial: Data with a spatial nature. Very often spatial data sets are geographical data sets, but they can also describe different types of space. Similarly to temporal data sets, the information here is anchored to locations or regions. (Spatio-temporal is also an interesting case: data sets that describe spatial phenomena that change over time (e.g., weather, urban mobility, animal migrations).

• Multimedia: This is any data set based on images, audio, video, or a combination of them. Multimedia data sets are very different from the data types mentioned above because they do not have an intelligible tabular structure.

• Language: This is data made of text and whole sentences. This is also a type of data that is not amenable to tabular structures, is characterized by linguistic components, and often requires special transformations before being handled by a model.

• Networks: This is data characterized by relational structure, which can often be described as a series of nodes and connections between the nodes. Social networks are an example of this type of data; people are connected by friendship.

Recognizing data types is important because most visualization techniques can handle only specific data types. For example, maps work for geographical data, statistical plots work for tabular data, and link-node diagrams work for network data. By knowing what type of data a model handles we can narrow down the set of visualizations that will work for that type of data.

In terms of output, most traditional models produce two types of outcomes: labels or quantities. Classifiers produce categorical labels, like when classifying transactions as frauds or what type of object is in an image. Regressors produce quantities, like when predicting the time it will take for a rideshare car to arrive.

Models can produce a wide variety of other outputs, including rankings and groupings, as well as complex objects like full images, text, etc. Importantly, these outputs are most often associated with performance metrics that derive from ground truth information from training and test data. For this reason, visualizations also handle performance data derived from these metrics, often in terms of error quantification.

As I explained earlier, the data part of data visualization is not limited to a model's input and output but also includes ML explanations and data coming from internal components. Characterizing this type of data is crucial, and more work is needed to better understand this aspect of data visualization for ML. Most of the design work visualization designers need to do in this space is to understand what type of information can be extracted from models and how such information helps answer important questions we have about machine learning models.

Data phase

Machine learning is not only about model development but also about assessing and monitoring what a model does after deployment. While the type of information extracted from models may look the same between these two phases, models in production can produce additional information that models in development do not have. For example, models can experience data-shifting problems as well as edge cases that are simply not visible when a model is being developed. For this reason, visualization can also be used as a monitoring and assessment tool to understand how a model behaves once it has been deployed.

Another important situation to consider is when a model is used by a person to support a human-driven task. For example, in health care and banking, it is increasingly common to find ML models that support decision-making. For these cases, visualization can also play a major role by making the output of the model and its uncertainty more intelligible and helping the decision-makers integrate the output of the models in their decision-making process (e.g., to understand when it makes sense to follow a model’s recommendations and when not).

Data granularity

Another distinction I’d like to make regarding “what to visualize” is about the granularity of information. In explainable AI, researchers often make a distinction between local and global explanations, that is, methods that provide information about single instances (e.g., how did my model make this single specific prediction?) versus methods that aim at giving an understanding of the entire model (e.g., what are the most important features in this model?). In visualization, we can do the same, except that we can add an intermediary step focusing on subgroups, groups of instances grouped according to some criteria like sharing a specific attribute (e.g., all instances that are false positives, false negatives, true positives, and true negatives). In summary, visualization can handle data at three different levels of granularity: individual instances, subgroups, and entire data sets. How to create subgroups is part of the data design process I already mentioned above. Different ways to aggregate data into subgroups can lead to different visualizations and can enable people to answer different types of questions.

Data from multiple models

One last element I’d like to cover is comparison. I have already mentioned how relevant comparison is for model understanding. This means that when we build visualization tools we need to consider if we want to support model comparison as a task. Many techniques developed for model visualization work only for individual models and it’s not immediately evident whether and how they can scale to comparison of two or even multiple models. In fact, when I look into the literature of data visualization for ML I can’t find too many solutions that work for model comparison. This is surprising given the relevance of this task and my hope is that we will see new methods developed in this space soon.

Conclusion

This post gave you an overview of what is possible to visualize in machine learning, or in other words, where the data comes from. As we focus more on the data side of data visualization, it is important to emphasize two aspects that are too often overlooked. First, different data sets allow us to ask different types of questions and ultimately produce different types of knowledge. Making this association between data, questions, and knowledge is essential to avoid visualizing data without a particular aim. Second, data extraction and transformation are part of the visualization process, so deciding what data to extract from the model and how to transform it before it is visualized is at least as essential (if not more essential) than deciding what visual presentation to use. This post forms the basis for many of the upcoming posts, which will go much deeper into the data sources I have outlined above.

Previous posts of the series

If you missed the previous posts of the series, you should definitely take a look!

Reader’s feedback

This is a long-term project. These posts are initial drafts for something more structured I’d like to build once the series is done. It may be a book, a course, or something else. If you like what I am writing please let me know how this is useful to you and how I could be of more help also! If there is anything specific you’d like me to cover, I’d be happy to hear it. Just leave a comment below.

Leave a comment

Also, if you know someone who might be interested in this series, can you please send them a link to the newsletter? The more people we have here, the richer the conversation is going to become for everybody. Thanks!

Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs. Sungsoo Ray Hong, Jessica Hullman, Enrico Bertini. Proc. of the ACM Conference on Computer Supported Cooperative Work (CSCW), 2020.

The paper is an interview study on ML Interpretability aimed at understanding current practices and needs in the industry. In that study, we categorized users into three main classes: model makers, model breakers, and model users. These roles are a good starting point for our analysis, even though, as you will see in a moment, I will need to add more to create a complete picture of who may benefit from visualizing machine learning.

Roles

These are the relevant roles identified in the paper, plus some important ones to recognize and consider.

Model makers (developers)

The people who build ML models and solutions (you can also call them developers, data scientists, or the fancier MLOps). Within this class, I used to group ML researchers, but I now think researchers need a separate class. Researchers tend to focus more on developing new methods. Developers tend to focus more on developing solutions with existing methods and verifying that the solution works for their target application. Of course, this distinction is, in practice, not as neat, but to understand how visualization supports ML development, this distinction is essential.

Model breakers

Those who verify that the model works as intended. This category includes a wide variety of people, including some model makers. However, it typically includes product managers, risk officers (especially in highly regulated markets), and model adopters who must verify that things work as expected before an AI solution is integrated into an organization. The role of risk officer is discussed less in the literature but is extremely important and in strong need of support. Companies in regulated markets like banking have specialized personnel to ensure that models behave as expected and comply with regulations. The fact that developers and evaluators are not on the same team is crucial to ensure undesired biases and influences. The Model Risk Managers’ International Association (MRMIA) has an interesting technical report describing the role of ML risk officers.

Model consumers (users)

These are domain experts or laypeople who use the developed ML models to carry out tasks in their environment to achieve their goals. Here, we have healthcare professionals, business experts, HR teams, etc., who use ML/AI systems to make decisions in their environment. This category also includes the important subcategory of scientists who use ML to explore scientific phenomena through various types of ML modeling. Model consumers are way less defined in the current literature; this is where I see an important gap. In any case, my intuition is that visualizations here could help with two high-level goals: onboarding and interpretation. In onboarding, model users explore the model decision space to understand how it behaves and generate a mental model. In interpretation, visualization can play a major role in putting model output in context so that users can integrate knowledge from the model more meaningfully.

Model explainers

This class did not exist in our original paper, but I am now convinced it is an important group of people to consider. Explainers are educators, researchers, and enthusiasts interested in using visualization to describe how ML works to other people. This is where people develop “explanatory visualization,” which walks the readers through several steps to help them develop an intuition and understanding of how some ML technique works. The Financial Times, for example, published this fantastic explainer on Large Language Models not too long ago.

And if you want to blow your mind you should definitely take a look at the incredible, but now discontinued, distill.pub, an online magazine/journal that focused on ML visualizations and started when nobody was talking about AI and ML yet, except the experts at work (their first explainer was published in 2016 🤯). If you have never heard or it, do yourself a favor and check the amazing articles they published over the years. Take a look at the “Activation Atlas” if you want to start somewhere.

Finally, if you want a taste of the efforts of academic visualization community in this space, take a look at the Workshop on Visualization for AI Eplainability, which is at its 7th edition already, and you’ll find even more amazing examples.

Model researchers

As I mentioned above I think ML researchers deserve a separate category. Researchers are ML experts who develop new ML methods and techniques—the cutting-edge. While developers are more into using existing methods to solve specific applied problems, researchers are more into developing new capabilities and methods. When I mention researchers, one should not equate this to academic researchers because a lot of innovation in ML today is happening in companies. However, what ML researchers have in common is that their goal is to develop new capabilities for ML and AI, and as such, they have a stronger focus on understanding how models do what they do rather than what they do. For this reason, this is the main class of users who have some interest in looking inside models to understand what specific components or parts of the architecture do. While the academic community has developed many of such visualizations, I have a hunch that looking inside models is something that interests almost exclusively model researchers (and this is, of course, not to say that because of that, they are not important).

I want to conclude this part by clarifying that an individual can cover many of these roles at once and that this is a very coarse categorization. However, despite its limitations, this categorization is helpful to reason about what goals and tasks exist in ML visualization and how we could provide support to these roles.

Goals

In the following, I cover a few common goals that can be supported by data visualization. This list, like the roles outlined above, is by no means exhaustive, but I think it can help us reason about what visualization can do in this space.

Improve model performance

Model makers are, above all, interested in building models that make as few mistakes as possible and have high accuracy (“What is the model performance?”). But this is not the only criterion possible. Model makers also want to see what kind of errors the model makes because some errors are more problematic or riskier than others, a goal they partially share with model breakers. So even if two models have similar performance, one may still prefer a model that makes certain kinds of errors or errors that are more manageable than others (“Where and when does the model make mistakes? Where do the mistakes come from? How can we build a better model?”) Relatedly, model makers often want to check if the model “generalizes” well, that is, whether it’s going to behave as expected with data the model will receive in the future, once it’s in production. While some statistical methods exist to deal with generalizability, manual inspection as a sanity check is also important (“How does the model behave with instances I know might be problematic? In what areas of the input space is the model more uncertain?”) There is no substitute for looking at specific instances where the model makes mistakes. It is not rare for model makers and breakers to have a library of problematic or hard cases they keep testing when assessing a new model. In future posts, we will cover visualization methods that guide users in detecting and interpreting problematic instances, for example, by arranging instances according to their model score to detect areas where the model may be more uncertain or according to subsets analysts want to inspect.

Compare models

Working with people from industry, I have learned that comparing models is a much more frequent and relevant task than one may initially think. This recent paper, “Operationalizing machine learning: An interview study” explains how pervasive and relevant model comparison is. Model makers and breakers continuously maintain multiple models they compare for various purposes: to compare different modeling strategies, to compare new models with legacy models or models they have in production, and to perform A/B testing with models they deployed. Risk officers often develop “challenger models” (Deloitte has a nice report on them if you want to learn more about the topic) to compare with models they receive from the development team. So, as you can see, model comparison is everywhere.

This is interesting because comparison is a key research area in visualization. Visualization is extremely powerful in helping people compare objects or entities. Still, the design space of comparisons is often large, and one needs to understand how visual comparison works to make it effective. In this space, my friend Prof. Michael Gleicher is a real expert. If you are unfamiliar with his work, I strongly recommend this classic paper, “Visual Comparison for Information Visualization.”

Unfortunately, and surprisingly, model comparison is not well supported by existing visual analytics tools and it has not been identified as a fundamental task (“How do the models compare? How do they differ in terms of decisions and errors they make? Why do they differ?”) The image below shows an example of model comparison technique from Model Tracker, a tool developed by Microsoft Research and presented in this paper in 2015 (!), “ModelTracker: Redesigning Performance Analysis Tools for Machine Learning.” The little squares are individual instances, and the horizontal axis represents the model score. The lines and arrows you see represent changes between two different versions of the model.

Assess risk

Once models are developed, it is essential to have people check what risks they carry. In many settings, this role can be covered by a project manager responsible for the product or, in regulated markets, by model risk officers. In many situations, model evaluators must be different from model builders because they do not have any specific attachment or stake in the model's performance. They succeed when they find problems that builders cannot detect or anticipate. Model breakers typically ask questions about potential points of failure. For example, they may ask, “Can we trust this model? Does its logic make sense? Where could the model fail? What are specific cases where the model should never fail? Where could the model fail unexpectedly? Does the model have relevant biases? Does it comply with regulations?” Although the techniques model breakers and model builders use may end up being similar, their focus is different. Model builders test models to generate insights about how to improve them, while model breakers test models to assess where, when, and how they may fail.

Internalize model logic and calibrate trust

While many models are used to automate tasks, there are many situations where models help people carry out some tasks. For example, models can recommend specific courses of action (e.g., buy or wait?), flag entities/objects to review (e.g., equipment to maintain because it may break soon or anomalies to review in a security setting), rank items (e.g., which business franchises one needs to review to avoid future problems). Sometimes, some of these tasks influence very sensitive decisions, so the stakes are high. Common ones known in the literature are recidivism predictions, ranking people for job hiring, ranking people to decide who to call to prevent hospital readmission, predictions of adverse health outcomes, and many more. The stakes are high! Because of that, the visual interfaces we build to support these tasks downstream (and maybe even upstream) model output are very important.

This is where research is not very well developed yet, and we need to accumulate more knowledge. In any case, the type of questions model users pose are quite different from the ones model makers and breakers have. Users want to understand how much they can trust the output and what they can do with it. In turn, this also means that visual interfaces in this space need people to help with how much uncertainty there is in a recommendation and what the logic that drives a given output is. Common questions users may have include: “Can I trust this recommendation? Does the recommendation make sense? Why does the model provide this recommendation? How uncertain is the model on this prediction?”

Model user tasks may exist globally or locally. At a local level, users are interested in assessing individual cases/decisions they are confronted with. At a global level, they need to understand how the model works in general (i.e., its logic), whether they can trust it, and what specific predictions may be more problematic. Local-level tasks pertain more to the actions and decisions users have to make. Global-level tasks may be part of an initial onboarding and setup phase to help users understand how a model works and internalize its logic.

Understand how models do what they do

Machine learning models are so complex that even their developers are not sure how they do what they do. This is especially true for generative AI, where researchers try to understand how certain surprising capabilities emerge from their structures. In this space, models are studied almost like the way physical phenomena are studied in natural sciences: look at how they behave and what roles individual components seem to have and build explanations and theories around them. Visualization could play a major role here. In the same way, microscopes are used to understand microorganisms and telescopes to understand the cosmos, and visualization, in a way, could be used to understand machine learning models. The articles I mentioned from distill.pub are a lot like that. The authors look at visualizations to develop intuitions about how models do what they do and I hope there will be more works like these ones.

Of course, there is way more than what I captured in the paragraphs above, but my hope is that these can be a source of inspiration to look more closely at how visualization could help in this space.

Why use visualization?

After reading about roles and goals, a legitimate question is: Why use visualization? When and why does visualization play a role in achieving these goals? It’s hard to give an exhaustive answer. In fact, part of the research we will need to do is to have more and more specific answers to this question. But let me try to sketch something here. The way I like to think about this problem is that data visualization can help us develop an understanding (or at least intuitions) about three main aspects of ML models:

• Behavior: How does the model behave?

• Logic: What logic governs this model?

• Mechanics: How does the model do what it does?

Visualization of data coming (or derived) from models and their components can help us draw inferences about these three elements and this is where its real value lies. In future posts I will come back to this idea. In particular, I will focus on data visualizations that focus on specific types of data one can derive from models and how this information helps us draw specific kinds of inferences and intuitions.

Conclusion

That’s all I have to say for now regarding roles and goals. There are aspects of this problem that I am still struggling with, but I am confident the picture will become clearer as this series develops. If you have any ideas or comments to share please leave a comment below. This is very much a work in progress, and I need your help to think these things through! Do you find this helpful? Does it help organize more clearly ideas about how visualization plays a role in this space? Please let me know!

Leave a comment

If you are even vaguely familiar with technology, you must have heard machine learning (ML) is all the rage right now (note that people often talk about ML and AI interchangeably even if, in principle, ML is a subset of methods for AI). While ML has existed for several decades, recent breakthroughs have made it extremely popular.

In this new series of posts, we will explore connections between data visualization and machine learning. More precisely, I will focus on how data visualization can help practitioners and end-users better understand machine learning data, behavior, structure, and decision mechanisms.

You may ask, “Why use data visualization for machine learning? What is special about Data Visualization that ML can benefit from?” Also, if you are familiar with Data Visualization, you may ask, “What is special about ML that requires specific visualization solutions?” Let me answer the two questions separately.

Why use Visualization in ML?

Data visualization is needed because ML models are complex objects increasingly used for applications that impact many people’s lives, directly or indirectly. ML is everywhere. It’s in your phone, in the stores you use to buy stuff, in your bank, in your watch, in the computers and devices your doctor uses to take care of you, in the phone you use to take pictures of your family, etc. This is an interesting combination because ML is pervasive, takes place in sensitive settings, and is very complex. Increasingly complex.

Let me clarify what I mean by complex. The most fundamental feature of machine learning is that 1) the machine learns through a series of examples (often a massive amount of them in modern applications) and 2) the program that is learned is (most often) not intelligible; that is; humans can’t directly observe and comprehend what the model has learned and how it has learned it. In most cases, there is no explicit representation a human can review to understand and predict how a model will behave with new unseen data. It’s quite amazing if you think about it. We can teach machines to do something without giving them explicit rules. This is a remarkable feature, but it comes with a huge cost: we either blindly trust what these machines do or find ways to verify what they have learned and how they behave. Since we do not provide explicit rules to the model and the logic is not explicitly encoded, we have to find ways to observe the model to help us draw inferences about its behavior and logic.

In a way, the fundamental problem is one of abstraction. We need to find abstraction layers that translate the language of models into languages humans understand. This is effectively not new. Humans often build very complex things that perform complex tasks, and we must find ways to make people interact with such complexity through well-crafted abstractions.

In his legendary “The Design of Everyday Things,” Don Norman explains the problem very well. Even something as simple as kitchen appliances has complex internal logic and mechanisms that users do not need to know to operate them. However, when designers build interfaces for these appliances, they can make choices that considerably impact their usability and understanding.

What is new, however, is the sophistication of the type of decisions and outputs ML systems generate and their potential impact on society. An unusable toaster is not that big of a deal compared to an AI system providing recommendations to medical doctors or hiring managers.

Caruana et al. provide a remarkable example of what can go wrong with ML if a human does not verify what the model has learned. In a landmark paper on intelligible ML models published in 2015 (before the advent of super complex deep learning models!), the authors recount the story of a series of models trained to predict the probability of death of patients affected by pneumonia on a clinical trial. When the researchers trained an intelligible model, they discovered that the model had learned “that patients with pneumonia who have a history of asthma have lower risk of dying from pneumonia than the general population.” This is clearly absurd because asthma is a major risk factor. The researchers later found that patients with asthma were sent directly to the ICU, which in turn made patients with asthma have a higher probability of survival. This is a good example of what can happen with ML models: they can pick patterns from the data that do not reflect reality accurately and can lead to dangerous decision logic. For this reason, models often need human supervision, and human supervision needs the careful design of visual representations that make the logic and behavior of models understandable.

Why do we need ML-centric Visualization?

The second question we must address is why we need to study data visualization specifically for machine learning. Can’t we just use what we know about data visualization and apply it to ML? In principle, yes, but ML poses some unique challenges. ML models are complex objects, and it’s not obvious what aspects of the model development and use should be visualized. For example, one can visualize the data used to train a model, the model's structural components, or the model's behavior when it’s used in production. In addition, ML models are dynamic objects that respond to inputs and generate outputs (e.g., using the example above, they can take data about a patient and return the probability of that person dying); they are not just static data. In a way, they are more similar to simulation models, which produce outcomes on demand according to the different information and parameters one feeds them with.

In turn, this means that visualizing ML models often requires the non-trivial ideation of model probing and querying mechanisms that guide the user towards behaviors of interest. In a way, the data visualization problem we need to solve with ML visualization is not mainly about which visual representation to use, even though this is very important, but more about what information to extract in the first place and how to interact with the model so that we can understand how it works and behaves. Models can generate as much data as you want; you “only” have to feed them with some information, and they’ll respond with something. Understanding what aspects of a model one needs to investigate is one of the main design decisions one has to make.

For this reason, the main principle I’ll use to organize existing ML visualization techniques in the series is what information these techniques visualize rather than how they visualize it. Accordingly, I plan to organize the content around three main classes of visualizations:

1. Visualizing ML Data: Every model receives some data as an input and produces some data as an output. What can we learn by visualizing these data?

2. Visualizing ML Explanations: In ML, there are techniques to create “explanations” of model decisions. How can visualization help us understand and explore these explanations?

3. Visualizing ML Internals: ML models have an internal structure (and architecture) made of model components. What can we learn by visualizing the behavior of these components?

Of course, this is not carved in stone. I might make changes or add new categories as I develop the series.

Overview

As of now, these are the posts I plan to write and post for the series:

1. Who Needs Visualization for ML?

2. What Is There To Visualize?

3. Visualizing ML Output

4. Visualizing Model Explanations

5. Visualizing Model Internals

This is very tentative. There is a high chance that I will need to break these topics down into smaller parts and add or remove pieces as I develop the individual posts.

I am writing this with a bit of trepidation. Starting a new series is a big task and this seems bigger than the previous one. Wish me luck! I hope you’ll enjoy what I have to offer in this series. In the meantime, if you have any questions, suggestions, or requests, add a comment below.

Leave a comment