great post and I share a lot of your thoughts on this. I think regarding the current deficiencies of LLMs, I am reminded a phrase from Sam Altman at the World Economic Forum earlier this year that haunts me still "these are the dumbest the models will ever be".
"What would the effect of providing people with such capabilities be? I don’t know, but it’s certainly an interesting direction to explore!"
One reason I am not a fan of frontier LLM models is that don't really support human-AI collaboration but promote straight displacement of work. In this case, I think student will use LLMs to just do all the work for them, similar to how high school kids used GPT3 when it was first released. Hopefully I am wrong and this technology forces us to try to evolve our critical thinking skills to be more innovative than what LLMs can produce.
In terms of the data viz process I think GPT is around the 50% percentile mark in terms of performance compared to the average knowledge worker. I am expecting GPT5 to be in the 80-90th percentile based on the scaling laws of AI.
Based on your experience with GPT with respect to data viz, where which level do you think the latest GPT model falls under? https://arxiv.org/pdf/2311.02462
Level 1: Emerging, equal or somewhat better than an unskilled human
Level 2: Component, at least 50th percentile of skilled adults , ex. Watson, LLMs, writing/simple coding
Level 3: Expert, at least 90th percentile of skilled adults, ex. grammarly, Dall-e
Thanks for the comment! I don't think these examples demonstrate that LLMs do not support human-AI collaboration. There is nothing wrong with having an AI suggest ideas to you. What is wrong is to take the output and use it as it is without any critical thinking. Humans must still acquire all the skills and can't defer important decisions to machines. If anything, at the very least, because someone needs to be held accountable for these decisions.
agree with that assessment for the current frontier LLMs and the importance of us still developing critical thinking.
I am less certain that human in the loop AI systems with humans validation will last much longer given how quickly these models are advancing. Maybe my critical thinking skills are below average but for problems that don't require niche domain specific context, I think Claude 3 Opus is a already better critical thinker from a multi-disciplinary, system 2 thinking framework.
I agree that things can and should be automated, if possible. But sooner or later, a machine needs to do what it is told by someone and explain what it has done. I can't see how this part will be removed. There are many tasks where the goal is for the human to learn something. In those cases, the machine will need to provide explanation and education, at minimum. In any case, I am skeptical that we can create systems that we can trust completely, so verification will always be needed. But maybe I am completely off here!
great post and I share a lot of your thoughts on this. I think regarding the current deficiencies of LLMs, I am reminded a phrase from Sam Altman at the World Economic Forum earlier this year that haunts me still "these are the dumbest the models will ever be".
"What would the effect of providing people with such capabilities be? I don’t know, but it’s certainly an interesting direction to explore!"
One reason I am not a fan of frontier LLM models is that don't really support human-AI collaboration but promote straight displacement of work. In this case, I think student will use LLMs to just do all the work for them, similar to how high school kids used GPT3 when it was first released. Hopefully I am wrong and this technology forces us to try to evolve our critical thinking skills to be more innovative than what LLMs can produce.
In terms of the data viz process I think GPT is around the 50% percentile mark in terms of performance compared to the average knowledge worker. I am expecting GPT5 to be in the 80-90th percentile based on the scaling laws of AI.
Based on your experience with GPT with respect to data viz, where which level do you think the latest GPT model falls under? https://arxiv.org/pdf/2311.02462
Level 1: Emerging, equal or somewhat better than an unskilled human
Level 2: Component, at least 50th percentile of skilled adults , ex. Watson, LLMs, writing/simple coding
Level 3: Expert, at least 90th percentile of skilled adults, ex. grammarly, Dall-e
Level 4: Virtuoso, atleast 99th percentile of skilled adults, DeepBlue (chess)
Level 5: Superhuman, outperforms 100% of humans, ex. AlphaFold, AlphaZero, AlphaGo
Thanks for the comment! I don't think these examples demonstrate that LLMs do not support human-AI collaboration. There is nothing wrong with having an AI suggest ideas to you. What is wrong is to take the output and use it as it is without any critical thinking. Humans must still acquire all the skills and can't defer important decisions to machines. If anything, at the very least, because someone needs to be held accountable for these decisions.
agree with that assessment for the current frontier LLMs and the importance of us still developing critical thinking.
I am less certain that human in the loop AI systems with humans validation will last much longer given how quickly these models are advancing. Maybe my critical thinking skills are below average but for problems that don't require niche domain specific context, I think Claude 3 Opus is a already better critical thinker from a multi-disciplinary, system 2 thinking framework.
I agree that things can and should be automated, if possible. But sooner or later, a machine needs to do what it is told by someone and explain what it has done. I can't see how this part will be removed. There are many tasks where the goal is for the human to learn something. In those cases, the machine will need to provide explanation and education, at minimum. In any case, I am skeptical that we can create systems that we can trust completely, so verification will always be needed. But maybe I am completely off here!