Keeping the same characters across AI clips: universal elements instead of the first frame

Episode two of "Sheriff Potato" is done — you'll find it in full at the end of the clip: the tomato, the cucumber, the sheriff himself and a newcomer, Grandma Onion. But this article isn't about the plot. It's about what changed under the hood. I built almost all of episode one from the first frame. In episode two there are noticeably more characters, and I need each one to stay recognizably itself from scene to scene, with the story moving forward instead of falling apart. So I switched the canvas over to the second approach — references.

I covered the canvas itself in the previous part; here I'll take apart one of its joints: how to explain to a video model who is who.

First frame or references: the models make you choose

The "one frame → video" trick has a ceiling, and I hit it. As long as you work from the first frame, you can't bring an extra character into the scene: whatever isn't in the picture doesn't exist for the model.

And video generation systems, unfortunately, work on an either-or basis. Either you give the model a first frame, or a set of references — almost nobody lets you combine the two.

For the reference path I introduced a concept on the canvas that I called "universal elements". An element is a character, an object or a location. I take a picture from the footage, declare: here's an element, here's its name and role — and it plugs into the video node as one of the references. Scenes get built out of these blocks.

What the models can do, and at what price

To use any of this you have to understand how the specific video models are built — they can do different things.

Kling is the only model that takes a start frame, an end frame and elements at the same time, in a single request. A crazy thing — nobody else can do it: everyone else wants either a single start frame or a stack of references.

Grok accepts up to seven references at once, Gemini — up to ten. And both get along well with Russian audio: the characters speak Russian and the lip sync holds. Kling and Seedance I wouldn't trust with Russian speech. That said, Seedance itself is a really good model, currently among the best for effects. Its main drawback is the price.

Now about money, because it hurts. Eight seconds of Seedance — a dollar and seven cents. Gemini — eighty cents. Grok is insanely cheap: sixty-four cents for the same eight seconds.

And here's a little secret: I run my experiments in 480p. At the very least it keeps you from going bankrupt — while you're tuning the prompt and the references, resolution doesn't matter; what matters is that the model understands the scene.

The system prompt: telling the model who is who

Just handing the model a stack of pictures isn't enough. In the system prompt you have to specify which character is responsible for what — and every model has its own syntax for it.

Gemini Omni Flash, for example, understands ImageRef tags: zero, one, and so on down the list. I hint: the town square is the environment, it goes as ImageRef 0. And I get a perfect picture. As long as the model is left guessing who is who among the references, the result drifts. The moment you say "this character is in this picture", everything falls into place.

The second layer is storyboards — part three of the series was about them. With a storyboard the model gets not one picture of a hero but a clear understanding of how he looks from the front, the side, the back and above, the details of his face and his distinguishing marks. For the sheriff that's the badge, the boots, the holster. The more of those marks the model knows, the harder it is for it to draw a "roughly similar" hero instead of mine.

Scene six: I went overboard and added everyone

On scene six I decided to stress-test the system and, frankly, went overboard: I threw every character into a single request, plus the square as the environment. The model did assemble the scene.

But it didn't go without glitches. The text in the frame that it was supposed to read, it couldn't read properly, and artifacts crept into the small details. I show it in the video as is — I think honest bugs are more useful than a highlight reel of wins. One of the takes turned out so likable, though, that I seriously considered putting it into the episode.

I ran the same scene on Grok — I wanted to see how it would handle the bartender, not the sheriff. It did well; I got two interesting takes. I didn't keep iterating: I can't spend a lot of money on renders, the experiment's budget is finite.

Honestly about the channel — and a question for you

I don't consider myself a proper blogger. But the numbers have moved: my clips started reaching eight hundred impressions right away, and episode one set my personal record for viewer retention. For a channel I run on the side, that's unexpectedly nice.

Episode two is at the end of the clip, in full. And I need to figure out where to take this: would you want to build clips like these in a system like this yourselves? Tell me in the comments under the video. If there's enough interest — I'll open up access.