How AI video generators work

An AI video generator does not film anything, and it does not animate a scene the way a 3D program does. It starts from random noise shaped like a short clip and removes that noise step by step, using a trained network, until the frames look like footage that fits the prompt. The published systems from OpenAI, Google, Meta, Runway and Stability AI all describe a version of this process.[1][2][3][4]

If you have read our explainer on how AI image generators work, the denoising loop will be familiar. This article covers what video adds on top of it. Every frame has to agree with the frames around it, the amount of data grows with every second, and the model has to produce motion that looks physically right without ever having been taught physics. Those three demands explain most of what creators see: short clips, faces that drift between shots, objects that melt or merge, and lettering that falls apart.

From still images to moving ones: what video adds

A video is a stack of images, but generating one frame at a time and stringing the frames together does not work well. Each frame might look fine on its own while the sequence flickers, jumps or changes the subject between frames. Ho and colleagues at Google set out the goal in the first sentence of their 2022 paper:

“Generating temporally coherent high fidelity video is an important milestone in generative modeling research.”

— Ho and colleagues, Video Diffusion Models, abstract (2022).[5]

Temporally coherent is the key phrase. The model has to treat a clip as one object that extends through time, not as a series of separate pictures. Every architecture below is a different way to achieve this within a limited compute budget.

Three changes follow:

• The data is far larger. A few seconds of footage holds dozens of frames, and each frame is a full image.
• The model must learn motion. Training on video teaches what tends to move, how fast and in which direction. An image dataset does not contain that information.
• Errors build up over time. A slightly wrong hand in a still is one flaw. In video, the same hand has to stay correct for every frame it appears in.

Latent video: compress first, then generate

Image generators often work in a compressed latent space rather than on raw pixels. Video generators push this further by compressing across time as well as across the frame.

OpenAI's technical report on Sora describes a network that "takes raw video as input and outputs a latent representation that is compressed both temporally and spatially." Sora is trained on, and generates in, that compressed space, and a separate decoder maps the generated latents back to pixels.[1] Meta's Movie Gen paper describes the same arrangement: a temporal autoencoder maps images and videos into a "spatio-temporally compressed latent space" and decodes the result back into frames.[2] Google's Veo 3 tech report says video and audio are each encoded by their own autoencoders "into compressed latent representations in which learning can take place more efficiently than with the raw pixels or waveform."[4]

The compression is large. For its largest model, Movie Gen compresses video by a factor of 8 in each of height, width and time before generation begins.[2] The generator never touches most of the raw pixel data. It works on a much smaller block, and the decoder rebuilds the detail at the end.

Small details such as fingertips, the spokes of a wheel or the strokes of a letter take up very little of the compressed representation. The decoder has to reconstruct them from very little information. This is part of why fine detail is often the first thing to break. It is our inference from how the pipeline is built, not a measured result reported in these papers.

Spacetime patches and Transformers

Once a clip is compressed, the model needs a way to process it. Two broad designs appear in the primary literature.

Adding time to an image network

The earlier approach adapts an image model. Ho and colleagues turned an image diffusion network into a video network by keeping its spatial layers and inserting a temporal attention block after each spatial attention block, so information can pass between frames as well as within each frame. They note that this "factorized space-time" design is a common choice for computational efficiency.[5] Runway's Gen-1 paper extends an image architecture in a similar way, adding a 1D temporal convolution after each 2D spatial convolution and a temporal attention block after each spatial attention block.[6] Stability AI's Stable Video Diffusion also starts from an image model, Stable Diffusion 2.1, inserts temporal layers and trains on video in stages.[7]

Starting from an image model passes on its knowledge of appearance: what objects, textures and lighting look like. The new temporal layers then have to learn how those things move.

Treating the whole clip as tokens

The more recent large systems handle space and time together. OpenAI describes cutting Sora's compressed video into "spacetime patches which act as transformer tokens", in the way a language model handles text tokens (our explainer on large language models covers tokens). Sora is described as a diffusion Transformer: given noisy patches and conditioning such as a text prompt, it is trained to predict the clean patches.[1] Movie Gen likewise uses a Transformer backbone, trained with a related objective called Flow Matching, on patchified latent video.[2] Google's tech report says Veo 3 uses "a transformer-based denoising network" applied iteratively to Gaussian noise.[4]

The patch approach has two consequences for creators. First, shape is flexible. Sora's report explains that size can be controlled at generation time "by arranging randomly-initialized patches in an appropriately-sized grid", which is how one model can produce widescreen, vertical and square output.[1] OpenAI also reports that training on videos at their native aspect ratios, rather than square crops, improved framing. A model trained on square crops sometimes left the subject only partly in view.[1]

Second, the whole clip is generated together. Because the model works on the full block of patches, a frame near the end can be shaped by what happens at the start. OpenAI credits this "foresight of many frames at a time" with keeping a subject the same even when it briefly leaves the frame.[8]

AI Chronicle

How text and images steer a video

A prompt does not reach the video network as words. It is converted into numbers that condition each denoising step. The way video models are trained to read those numbers explains a lot about prompting.

Captions are rewritten in training, and often your prompt is too

Video datasets rarely come with good descriptions, so developers write their own. OpenAI trained a "highly descriptive captioner model" and used it to caption every video in Sora's training set, reporting that the descriptive captions improved text fidelity and overall quality.[1] Google says Veo 3's training clips were captioned "at different levels of detail, leveraging multiple Gemini models."[3] Runway says Gen-3 Alpha was trained with "highly descriptive, temporally dense captions", meaning captions that describe what happens at different moments in a clip, which it links to more precise key-framing.[9]

Because the models learn from long, detailed captions, a short user prompt is often expanded before generation. OpenAI used GPT to turn short prompts into "longer detailed captions."[1] Movie Gen describes an inference prompt rewrite for the same reason.[2] Google's Vertex AI documentation lists "prompt rewriting" as a Veo capability.[10] As a result, your words may not be the exact text the video model receives, and details the rewrite adds can show up in the clip.

Several encoders, one of them for lettering

Movie Gen combines three text encoders. Two supply meaning at the level of the whole prompt. A character-level encoder, ByT5, is used "only" for the part of a prompt that asks for written text to appear on screen.[2] That design choice shows that spelling is a separate problem from meaning, which matters later in this article.

Movie Gen also conditions on frame rate by adding a value such as "FPS-16" to the prompt during training, which it uses to control the length of generated videos.[2]

Images, frames and references

Text is a weak way to specify exactly how something looks. Video systems accept stronger visual inputs, and each type does a different job:

• Image-to-video. A still becomes the starting point. Sora's report shows animating images, and Stable Video Diffusion presents image-to-video as a core use.[1][7]
• First and last frames. Google's Gemini API documentation for Veo 3.1 describes generating a video from specified first and last frames.[11]
• Reference images. The same documentation says Veo 3.1 accepts "up to 3 reference images" of a person, character or product to preserve the subject's appearance.[11] Runway says Gen-4 can generate consistent characters from "a single reference image" without fine-tuning.[12] These are vendor capability statements, not independent measurements of how well appearance is kept. Our Runway review sets out what Runway documents about its current models and editing tools. Reference stills are often made in an image generator first; our comparison of Midjourney vs. Stable Diffusion covers the control each offers.
• Structure. Runway's Gen-1 research conditions on depth estimates from existing footage, so a new look can follow the shape and motion of the original. The paper also describes a guidance setting that gives explicit control over temporal consistency.[6]
• Extension and editing. Sora's report describes extending videos forward or backward in time and editing input videos from text.[1]

The table separates what each input can fix from what it leaves to chance.

Why clips are short: resolution, length and compute

Every vendor limit on duration and resolution traces back to one fact: the number of tokens grows with every frame and every pixel, and Transformer attention grows faster than that.

Ho and colleagues were direct about it in 2022:

“To manage the computational requirements of training our models, we only train on a small subset of say 16 frames at a time.”

— Ho and colleagues, Video Diffusion Models, section 3.1 (2022).[5]

Movie Gen gives the clearest numbers. Its 30-billion-parameter video model was trained with a maximum context of 73K video tokens, which corresponds to a 16-second video at 16 frames per second.[2] The paper explains where that figure comes from: a 768 × 768-pixel video with 256 frames, compressed 8 × 8 × 8 by the autoencoder and then grouped into 2 × 2 patches.[2] The arithmetic checks out. That gives 48 × 48 patches per latent frame across 32 latent frames, or 73,728 tokens. On our arithmetic, a single 768-pixel image under the same scheme comes to about 2,300 tokens, so the 16-second clip is roughly 32 times larger as an input. The paper notes that because of "the quadratic nature of self-attention", scaling to such long contexts "requires immense computation."[2] Movie Gen reports training on up to 6,144 H100 GPUs.[2]

Developers respond in a few ways, and each one shows up in products:

1. Generate at a base resolution, then upsample. Movie Gen generates at 768 pixels and uses a separate spatial upsampler to reach 1080p, which "lowers the overall computational cost."[2]
2. Keep each generation short, then extend. Ho and colleagues extended 16-frame samples into longer videos block by block.[5] Google's documentation describes Veo 3.1 as a model for "8-second videos (720p, 1080p, or 4k)", at 24 frames per second. Clips can be extended by 7 seconds at a time, up to 20 times, giving a combined video of up to 148 seconds.[11]
3. Make sampling cheaper. Movie Gen describes a 50-step sampling schedule designed to approximate a 250-step one.[2] Stability AI notes that video diffusion models "are typically slow to sample and have high VRAM requirements, and our model is no exception."[7]

These limits affect your schedule as well as your budget. Google's documentation gives request latency for Veo 3.1 as a minimum of 11 seconds and a maximum of 6 minutes at peak times, notes that higher resolution means higher latency, and says generated videos are deleted from the server after 2 days unless you download them.[11] (Documentation checked 16 October 2026; these specifications change between model versions.)

Headline durations also deserve care. OpenAI's 2024 report says Sora could generate "up to a full minute of high definition video."[1] OpenAI's Sora page now states that as of 26 April 2026, "the Sora product is no longer available."[8] Research capabilities, product limits and availability are three separate facts, and each needs its own date.

Why the typical failures happen

The same vendors that publish these systems also document their weak points. Read together with the mechanics above, the failures have understandable causes. A failure with a known cause can usually be planned around.

Physics: the model learns what usually happens, not why

None of these systems contains a physics engine. OpenAI describes Sora's 3D consistency and object permanence as emerging "without any explicit inductive biases for 3D, objects, etc."; in its words, they are "purely phenomena of scale."[1] What the model learns is which motions tend to follow which, across a very large amount of footage.

That works well for common motion and breaks down on rare or complex interactions. OpenAI's report is plain about it:

“it does not accurately model the physics of many basic interactions, like glass shattering.”

— OpenAI, Video generation models as world simulators, discussion (February 2024).[1]

The Sora page lists further weaknesses: a plastic chair not behaving as a rigid object, a cookie showing no bite mark after being bitten, and objects "morphing."[8] Movie Gen's evaluation notes that unusual prompts, such as people flying, may produce "static videos or those with only camera movement" because of limited training data.[2] Stability AI reports that its model sometimes generates "too little generated motion."[7]

What this means for creators: common, continuous motion such as walking, pouring, drifting clouds or a slow push-in is generally safer ground than collisions, breakage, liquids interacting with solids, or several characters handling the same object. Treat cause-and-effect beats as high-risk shots, and either film them or cover them with a cut.

Hands: small, fast and constantly changing shape

None of the primary sources we used gives a failure rate for hands in video, so treat this section as our explanation, not a finding. The mechanics suggest several reasons hands are hard. A hand occupies a small part of the frame and so a small part of a compressed latent. It changes silhouette quickly as fingers bend and overlap, and it is often partly hidden. Every finger must stay consistent across every frame. Movie Gen's human evaluators for image quality were told to look for "errors in the number of fingers or arms," and its motion-naturalness criterion for video explicitly includes "natural limb movements."[2]

What this means for creators: close-ups of hands doing detailed work, such as typing, tying knots or playing an instrument, are the shots most likely to need several attempts or real footage. Wider framing, simpler gestures and hands at rest lower the risk.

Identity drift: the face that changes between shots

Within a single clip, joint generation helps keep a subject stable, but vendors stop short of promising it. OpenAI says Sora is "often, though not always, able to effectively model both short- and long-range dependencies," and its product page lists people or animals spontaneously appearing, "especially in scenes containing many entities."[1][8] Google's model card for Veo 3 states that "maintaining complete consistency throughout complex scenes or those with complex motion, remains a challenge."[3]

Across separate clips the problem gets worse, because each generation starts from fresh noise. Nothing carries over from the last shot unless you supply it. Ho and colleagues showed one version of this when extending videos. A naive method produced segments that "appear to be uncorrelated samples", while their reconstruction-guidance method kept the extended video coherent.[5] Movie Gen reports that adding positional information to every Transformer layer reduced "distortion and morphing artifacts, especially in the temporal dimension."[2] Data matters too. Stability AI ran cut detection on its training data "to avoid cuts and fades leaking into synthesized videos."[7] Google, for its part, notes that Veo 3 has a bias toward "frequent camera cuts and dramatic camera angles."[4]

This is why reference images, first-frame anchors and vendor "character" features exist. They supply, as an input, the identity that fresh noise cannot remember. Our guide to keeping characters consistent in AI video turns that into a shot-by-shot method.

Text on screen: spelling is a different skill

Generated signs, labels and captions often come out as convincing shapes that do not spell anything. Google's Veo 3 tech report states that the model, "while much better than previous Veo models," is "still poor at generating text."[4] The research explains part of the reason. Stability AI found that early training data led its models to generate "excessive amounts of written text," so it filtered out clips with large text areas.[7] Movie Gen added a character-level encoder specifically for requested on-screen text.[2] Developers either remove text from training data or add dedicated machinery for it. Neither approach makes a model a reliable typesetter.

Video adds motion on top: a word must stay spelled correctly in every frame as the camera moves.

What this means for creators: put titles, prices, logos and captions in the edit, where you can check and change them. Ask the generator for blank signage or surfaces you can track text onto later.

Bias and safety are failures too

Not every failure is visual. Google's model card reports that Veo 3 "appears to skew towards lighter skin tones when race is not specified in the prompt."[3] When people appear on screen, specify the casting you actually want and review who the model produced. Do not accept its default.

A worked example: planning a 20-second teaser (hypothetical)

This is a hypothetical planning exercise. The business, shot list and numbers are invented to illustrate the mechanics above. We did not run these shots through any tool, and nothing here reports a result.

Imagine a fictional neighborhood bicycle workshop that wants a 20-second vertical teaser for social media: the street at opening time, a mechanic fixing a chain, a wheel spinning true, and an end card with the shop name and opening hours.

Read against the failure modes, that brief contains four risks: a sign with words on it (text), hands working a chain (hands and fine physics), a recurring mechanic (identity), and spokes in motion (fine detail in a compressed latent). One long prompt for the whole 20 seconds would ask the generator to handle all four at once.

A plan built around the mechanics splits the problem:

The cuts between shots are part of the plan. Each shot is short enough to fit a single generation on services that make clips of around 8 seconds, so nothing depends on extension holding together. The mechanic appears in only one generated shot, so there is no shot-to-shot identity to keep. Voiceover and music are planned as a separate track; our AI voiceover workflow covers drafting and reviewing that layer.

Budget the attempts too. Suppose the team allows four candidates for each of the three generated shots, at 8 seconds each. That is 12 generations and 96 seconds of generated footage to produce about 16 seconds of final edit. The ratio is an assumption made for planning, not a measured hit rate. It is still a better basis for a schedule than expecting the first take to work.

Review each kept shot frame by frame, not only at full speed. Check that the subject stays the same, that no extra people or objects appear, that nothing passes through anything else, and that the blank sign stays blank. Then export with provenance metadata intact if your tools support it, as covered below.

AI Chronicle

Provenance, briefly

A realistic clip shows what a model produced, not what a camera recorded. Some vendors attach provenance signals. Google says videos created by Veo are watermarked with SynthID, and its Vertex AI documentation lists Content Credentials (C2PA) as supported for Veo 3.1.[11][10] Runway said Gen-3 Alpha would ship with "C2PA provenance standards."[9]

C2PA's own explainer is careful about what this proves:

“provenance information alone cannot tell you whether the digital content is true, accurate or factual.”

— C2PA, C2PA and Content Credentials Explainer (version 2.2), FAQ 7.2.2.[13]

The explainer also notes that provenance metadata can be removed, which is why the standard pairs it with watermarking and fingerprinting.[13] Our guide to content credentials covers what a credential can and cannot establish, and our comparison of AI detectors vs. content credentials explains why a detector's guess is a different kind of evidence.

Questions readers commonly ask

Is a video generator just an image generator run once per frame?

No. Early research adapted image models by adding temporal layers, but the model still learns across frames. The large current systems generate the whole clip together as spacetime patches.[1][5][7]

Does the model understand physics?

Not in the sense of calculating forces. It learns statistical patterns of motion from footage. OpenAI describes object permanence and 3D consistency as emergent from scale, and in the same report lists basic interactions it gets wrong.[1] Expect common motion to look plausible and unusual interactions to fail.

Why can't I generate a two-minute shot in one go?

Token count grows with duration and resolution, and attention cost grows faster than token count. Developers therefore train and generate in short windows, then extend or upsample.[5][2] Where a service offers long output, it is typically assembled from shorter generations. Google's Veo extension feature works this way.[11]

Will a reference image keep my character identical?

It gives the model the identity information that fresh noise lacks, and vendors describe it as preserving appearance.[11][12] It is not a guarantee. Check every shot against the reference, and expect complex motion and new lighting to cause drift.[3]

Can I trust vendor comparisons between video models?

Treat them as vendor-reported. Movie Gen and Veo both cite human-preference evaluations run by the developers themselves.[2][3] Our explainer on what AI benchmarks actually measure explains how to read such claims. For your own work, judge the model on shots like the ones you plan to make.

The next step

The mechanics come down to a few working rules. Keep shots short. Supply identity as an input. Keep text in the edit. Film the shots that depend on hands or physical cause and effect. Identity is the rule that takes the most practice, so the next guide covers it in detail: how to keep characters consistent in AI video.

Sources and evidence note

This explainer relies on primary research papers, vendor technical reports, model cards and official documentation, all accessed 16 October 2026. Vendor documents establish what each developer says about its own system. They are not independent tests. Product specifications are dated and change between versions. The workshop teaser, its shot plan and the attempt budget are hypothetical. Statements about why hands fail and how compression affects fine detail are our inference from the documented architecture, and are marked as such in the text.

Related reading

• How to spot AI-generated content: signals and limits
• AI training and copyright: the important distinctions