On this page · 6 sections ▾
Local inference means running a trained AI model on your own computer or device instead of sending requests to a company's servers. The model has to sit in memory while it works, so RAM is the main resource it needs. That is why a remark about RAM from OpenAI's CEO touched a nerve.
What is local inference?
Training an AI model is the expensive process of building it. Inference is the everyday part: you type a prompt, the model produces an answer. Cloud services do this in data centers. Local inference does the same work on hardware you own.
As The Verge reported, Sam Altman said on the Cheeky Pint podcast that many people want local inference and assume open-source models solve the problem of needing large data centers. India Today notes that the appeal includes privacy, offline access and more control over personal data.
Why does RAM matter so much?
A model is essentially a very large set of numbers. To produce text quickly, those numbers must be held in fast memory, not fetched from a slow drive for every word. DRAM, the type of memory in most PCs, is fast enough for that. The bigger the model, the more memory it needs, which is why a 32GB desktop can run some models and not others.
Data centers face the same limit at larger scale. They hold models in memory across many machines and serve many people at once.
What is the utilization argument?
Utilization means how much of a resource is actually working over time. Altman's claim, as Tom's Hardware quoted it, is that it is "quite terrible" from a global allocation perspective for everyone to have a share of the world's limited DRAM sitting in their own computer and being used 5% of the time. He said that in a supply-constrained environment you want maximum utilization in the cloud.
The logic is the same as sharing a tool. One power drill used by a whole neighborhood sits idle less often than a drill in every garage. In a data center, India Today explains, resources can be pooled and assigned to different users and workloads as demand changes.
What are the limits of that argument?
Several caveats apply. The 5% figure is Altman's own characterization, not a measured statistic from the sources. Tom's Hardware points out that OpenAI's main business is closed models run in data centers, so the argument also defends its business model. The argument also assumes that data center buildouts continue, and that RAM running around the clock is more valuable than RAM used for spreadsheets, browsing, gaming and occasional local AI.
