I’ll warn you right now, this is a blog post about AI. If you’re sick of hearing about it, and definitely don’t want to hear my take- skip this post.
A wise man once told me that the only difference between a human and a machine was our ability to fail tasks we’ve done a billion times before.
Modern (post-2022 GPT-4 release) machine learning has yet-again taken a grasp on the world in a way that nobody could have anticipated. If you rolled your eyes at that statement because literally everyone is saying it, you’re definitely justified in doing so. But I need to point something out that I feel that is often missed in the great “here’s my new AI-powered product” game.
The speed and improvement of the large-language models created across the world is unparalleled in both scale and potential. We are sitting on the cusp of a massive change in the way we work, live and play. Even if we never improve LLMs past the state they are today, everything is still changed forever.
There are three teams of researchers both amateur and professional. Those that create models more powerful and performant than the previous generation, those working on making those models smaller and those that fine-tune these models. It is essential for these two groups to exist both in isolation and in unity, driving models with higher accuracy onto smaller devices like laptops and phones.
I want to put forward the totally obvious theory that generalised models are not the end-game to drive value for billions today. Lots of small (smol), fine-tuned models are the future.
Hot take: It’s not the general accuracy (benchmarks) of the model that matters, it’s the specialisation of the model you pick.
You are a civil engineer in Tokyo who primarily focuses on ensuring a new building meets specified building codes and needs to draft up proposal documents. Would you pick a model trained on millions of proposal documents from the Tokyo building commission in Japanese, or one trained on writing stories?
Something we often forget in the pursuit of perfection is that humans get away with a lot of errors simply by doing the same kind of pattern recognition that we’re accusing large language models of performing. Ironically enough, these kinds of errors in specialised fields or topics fades with the more data you give models.
It’s more important to focus on teaching models problem solving than domain knowledge. With problem solving, a model can self-iterate to find a confident answer based on logic driven by the iterative process. With domain knowledge, the model can only mimic an answer. It’s the difference between teaching the textbook and teaching the job.
As a human, if I give you excellent problem solving abilities (such as those taught through day-to-day interactions), you will be able to confidently solve most domain problems that come up in your life, even with no prior knowledge. Will you be right? Not all of the time.
Criticising the model for not having perfect knowledge of every field is pretty much the same as blaming a human for not knowing everything.
Remember, machines have interests too.