Conjurors of Content

Wizards of maths and stats

Modern AI technologies optimise processes against rapidly changing variables. They show up almost everywhere: healthcare, business, autonomous vehicles, personalised content.

They can transform practically every sector because they make existing processes dramatically more efficient. Those who master their deployment enjoy strong advantages over competitors. Big Tech eclipsed the rest of the economy in part through first-mover advantages in AI, backed by data, compute, and the engineers who knew what to do with both.

The comparison is electricity and motive power a century ago: power drills and tractors instead of mule-driven ploughs. Every business now needs to recognise what AI is bringing to its sector and plan accordingly.

One of the most immediately applicable areas of machine learning is generative AI. In one well-known approach, the generative adversarial network, two neural networks compete: one tries to produce plausible content, the other tries to spot fakes. Loop them together and you breed ever more convincing representations of almost anything: human faces, for example.

Generative techniques turn a simple sketch into a painting in the style of a great artist at the touch of a button. They restore damaged or obscured content, massively upscale low-resolution images and video, and transfer an aesthetic je ne sais quoi from one object onto another. Given a few examples, the model follows underlying patterns and correlations you would be hard-pressed to describe in words or mathematics. It can even transform a winter video into summer, or vice versa.

Generative AI is the closest thing to magic in the world today. It is already widely deployed to restore and upscale pre-HD content in movies, TV, and games, and it powers the video filters in Zoom and Snapchat. The earliest applications focused on visual content, but recent developments are about to unleash something larger.

GPT-3 (Generative Pre-trained Transformer 3) by OpenAI is the latest and largest of these generative models, though it works differently, as a transformer trained to predict text. It builds on previous work by taking it to a massive scale, ingesting almost the entire known internet, with a stupendous number of parameters (the adjustable strengths of the connections between its artificial neurons).

GPT-2 had 1.5 billion parameters; GPT-3 uses 175 billion. To the surprise of many researchers, the massive increase in compute made it dramatically more capable. The same model at 10 billion parameters completes maths problems at a D- level; at 100 billion, B-; at 175 billion, A+. Scale in compute turns out to matter as much as the abundance of data that powered deep learning a decade ago.

GPT-3 is accessible only via API for now, owing to the hardware requirements (it cost OpenAI around $5 million to compute). That is not a deal-breaker for business use; in fact, it makes it easier to start applying these techniques in minutes instead of weeks.

Hardware improvements will make it far cheaper for those who want to train their own models. The human brain has an estimated 100 trillion parameters, give or take, and we will see models of comparable complexity for the same $5 million before the end of this decade. I expect the same pattern of increasing capability to hold as parameter size grows further. That is as worrying as it is exciting.

Right now, GPT-3 can be applied to a startling range of creative endeavours. The same model translates poetry from Chinese to English, plays chess, solves maths problems, serves as a hilarious dungeon master, works out treatment regimens and medicine dosages. That is a massive amount of flexible capability. One student even used GPT-3 to write posts for a new blog. Unbeknownst to its readers, one of the posts reached the top of Hacker News.

It is still closer to human intuition than human intelligence per se, but it is capable within limits. GPT-3 is a significant step in AI, though not an intelligence panacea. Its multifunctionality is formidable, but it still lacks executive functions or logical reasoning, and it’s restricted to working with text. It’s adaptable, so long as humans clearly define the problem to be solved. One can think of it as a babbling savant genie, but even that is incredibly valuable. The next version will be even more flexible, so much so that entire creative industries may be made obsolete overnight. I strongly recommend that businesses in all sectors experiment with GPT-3 and similar generative AI technologies, and become familiar with their application. Those who embrace this new wave will be as well-positioned as Big Tech has been to reap the benefits of the previous wave of deep learning.

The Sorcerer’s Apprentices

Machine learning is one of the hottest careers going, and demand is likely to grow. Deep learning has emerged in the past ten years or so, enabling amazing new predictive processes that can find patterns within patterns, and make order out of chaos. This has transformed industry, but has had less immediate effect upon the office. That’s about to change, thanks to GPT-3 and a wider family of new models (large language models, transformers and diffusion models, sometimes described as foundation models). These are very large statistical models that can be dynamically reconfigured to solve thousands of different problems with nothing more than a simple natural language request, typically called a ‘prompt’.

With this new technology, ML is finally accessible to the masses, as it takes little skill beyond asking a simple question to get quick and reasonable help with almost any digital office task one can imagine. The latest models are even generating computer code, video, 3D models, virtual personalities, and music on demand from nothing more than a description of the desired output. One might think that ML skills will be less needed as a result. However, once the public sees what state-of-the-art ML can do, demand for machine learning that can optimise almost any problem we can conceive of will grow sharply.

Machine learning and statistics are related disciplines. Statistics is about analysing data, and constructing models to explain and predict phenomena, whereas ML is about creating technical information pipelines of automated data analysis that can construct their own internal models to make a prediction. Statistics is human-focused and easily explainable. ML is machine-focused and less explainable, but may be more powerful in circumstances where there are very complex patterns, perhaps with too many variables for a human being to manage.

Both occupations require a solid grounding in mathematics and statistics. Statisticians focus on procedure (how to clean up data so it can be used, and how to analyse it, usually in R); ML engineers focus on which model to apply and the code to implement it (usually Python).

However, the terms are often mixed up, sometimes through honest confusion, and sometimes through wilful misrepresentation. Because the term has been in use for so long, ‘AI’ can refer to anything from a hand-written chess algorithm to a sophisticated transformer that can turn a simple natural language request into a masterfully executed output. Often, rather basic data science is dressed up as ML or AI. On the other hand, companies often apply the jackhammer of AI to surgically split a peanut of a problem when good old-fashioned data science would be far cheaper and quicker. Data science holds up the modern economy far more than ML, but it remains an unsung hero. Data science is still often a prerequisite for ML, as it provides the grounding to help ensure accuracy and robustness in ML models, which can reduce the risks of an unethical outcome, such as disproportionate treatment or other statistical biases.

Already, academia has been plundered by industry for top ML talent, lured away by large salaries and generous research budgets, and new graduates can hardly come soon enough. There will be increasing demand for skills in building and applying foundation models in particular, as well as in other, more specialised areas such as machine vision, which helps embedded systems like robots understand the physical world with ever-greater precision. New search engines for prompts are also emerging, as the art of constructing new prompts to manifest unseen potential from existing models will be another hot commodity, like sorcerers figuring out the pronunciation of words written in a spellbook. ML is here to stay, and it is only getting stranger.