Where Does Data Visualization Fit In an AI-Driven World?

Reporting on a panel we organized at Northeastern University on the intersection of AI and Data Visualization

It’s surprising to me how little I have written so far on the intersection of AI and Visualization in this newsletter. Not only because it’s a timely topic, but also because I have been active in this area with my research for quite some time. A large chunk of the research I have done in the last 5-6 years has been devoted to using visualization as an instrument to analyze, explore, and understand machine learning models and their behavior.

Anyway ... the focus of this post is not my research but a panel I organized with my colleague Paolo Ciuccarelli at Northeastern University’s Center for Design on the intersection of AI and Visualization. The panel stems from a larger internal initiative, where a group of faculty meets regularly to discuss ideas at the intersection of AI and Visualization. For this event, we invited a group of people whose work is already at this intersection. In this post, I’ll summarize the main ideas presented by the panelists. If you want to watch the whole thing you can find it in the video down below.

Menna El-Assady (harmful rationalizations, visual/verbal, strapping UI interfaces on top of models)

Menna is an Assistant Professor at ETH Zurich where she does research at the intersection of visualization and artificial intelligence. In her talk, she presented some of the work she has done in the past and some current work. One thing that stood out for me is this idea of “harmful rationalizations,” which I think means that when we use interfaces to explain and interpret the behavior of AI systems we may end up with ways to explain and describe their behavior that is not accurate and does not generalize to future use. This reminds me of Don Norman’s concept of designing interfaces that produce accurate mental models of how an artifact works. If we do not have good mental models we may not be able to use these systems effectively. An interesting problem here is that AI systems often behave stochastically, which means building mental models on previous interactions can lead to failures. I think one important point in Menna’s talk was that designing effective interfaces and visualizations to combat harmful rationalizations requires innovative ways to interact with models and in some sense solutions that generate more rather than less uncertainty. Another element that caught my attention was the idea that solutions in this space require a mix of visual and verbal interfaces. I am intrigued by the verbal aspect of visualization in general, and more specifically in this space where creating a “dialogue” between the human and the machine seems particularly relevant. Menna concluded with an exhortation to avoid just strapping a visualization on top of an existing AI system and call it a contribution. I believe that what she meant by that is that a deeper level of interaction is needed and that a more important role for visualization in this space is to help people form an accurate understanding of how a model works.

Hendrik Strobelt (vis community vs. AI community, do we matter?)

Hendrik Strobelt is a senior researcher at IBM Research and he has been working at the intersection of visualization and machine learning for a long time. His work is notable for the many visualization tools he built to support the model exploration of researchers and practitioners. He is also one of the few people I know who sits across the ML community and the Vis community. Not surprisingly, given his background, his talk focused on the role visualization researchers and practitioners play in the ML community. I think his take was that there is a deep disconnect between the two communities and also that what we do in visualization does not seem to influence or matter enough in the ML community. Similarly to what Menna mentioned above, it seems like most of the visualization work happening in ML and AI is quite cosmetic and does not take into account the decades of research that exist in visualization. Hendrik, asked, “Do we own the business?” while mentioning a company (apparently worth $17 million) that developed a product based on visualizing data through a projection/embedding, meaning, I guess, something that is trivial for anyone working in visualization. Another important point was about the development of technologies that allow researchers to understand the role of individual elements of a model. This is an area where we have seen quite a lot of activity from visualization researchers, but this is also a very small niche. In a way, if I interpret his worry correctly, we have a miscalibration problem where many visualization researchers focus on problems that are too narrow compared to the many other problems that exist. I found this quote from him quite poignant, “I highly doubt that there will be a job that's called model corrector.” As someone who has done work in this space, I sympathize with his assessment of the situation.

Melanie Tory (what can AI do for visualization?)

Melanie is the director of Human-Data Interaction research at the Roux Institute (an institute of Northeastern University - she is a close colleague of mine!). Melanie did work across Visualization and AI while at Tableau Research where she worked on natural language interfaces for data visualization. Melanie started with a funny experiment where she asked ChatGPT the main question of the panel “Where Does Data Visualization Fit In an AI-Driven World?” and obtained some reasonable answers on the use of visualization for transparency and explainability. But the rest of her talk focused on the complementary question of what can AI do for visualization (and data science more in general). This is very interesting because when we think about visualization and AI we often look at how visualization can help communicate and understand AI, but what AI can do for visualization is equally interesting and valuable. Here Melanie mentioned the idea of AI guidance in choosing and building the “right” visualizations for a given problem and the idea of recommending specific insights in a given data set, which has been a line of research in visualization and databases for many years. Data preparation and cleaning are other areas where AI can help a lot, especially considering how much time people normally spend doing that. Finally, she mentioned the more interesting and innovative idea of using AI to interact with visualization in novel ways. Language models already try to create visualizations as a result of data questions asked using natural language and this is where probably we will see more and more advancements. But interaction can also be driven by gestures and other modalities that can benefit enormously from an AI that learns to absorb intentions from humans in natural ways. Melanie did not mention this example directly but I think this is another great example of what she was describing. By the way … Did you see this Tableau demo showing a solution to interact with visualizations through speech and gestures?

Michael Correll (understanding the unique value of visualization)

Michael is a research professor at Northeastern University and, like Melanie, a member of the Roux Institute (yes, he is also a colleague of mine!). Michael does a lot of interesting work on visual communication of statistical data, among many other things. The main idea behind his talk was that visualization is a human product that existed for thousands of years and that it will keep existing as the primary vehicle for generating and communicating knowledge for many years to come, whereas AI is contingent on this specific period. Michael argued that there is a tendency to find an AI solution to every problem and that it’s not evident that AI is the right solution for many of the problems we have with data. I especially enjoyed his mention of John Turkey and the demo of Prim9, the research prototype he developed in the 70s that allows people to observe data with many dimensions. Michael argued that this is a fundamentally different approach to data problems than throwing more AI and automation into a problem. I guess this is the age-old dichotomy between artificial intelligence and intelligence amplification. Many of us still seem to believe augmenting humans is better than substituting them. Another really good point of Michael’s statement was about the fact that AI without tons of highly curated data does not seem to work well, but many data problems have to do with small data sets with low quality and this is where visualization tends to shine. The statement ended with an exhortation to better understand our value and to focus on it.

Conclusions

There is much more that we discussed in the panel and I encourage you to watch the recording if you are interested in the topic. As you can easily grasp even from a summary of the main statements there is a lot to discuss and explore in this space! As someone who has done research in this space actively for quite some time, I must say I am a bit disoriented. I do not know what to expect. What I know is that visualization is important and that we need to keep developing ways for humans to communicate information to other humans, whether an AI is supporting us with this task or not. Similarly, I think we are going toward a world of machines with increasing complexity so developing tools to understand and assess them is going to be crucial. So, visualization will keep playing a role there.

—

What is your opinion? What is the role of visualization in an AI-driven world? Do you see any specific challenges or opportunities? Write a comment below to let me know what you think!