← Back to MAKING / LAB (EN) 🇯🇵 日本語版の記事を読む →
🎬 MV, 3D & Video AI 2026-11-17

I Tried to Make a Rock Singer Move with AI—and Ended Up Acting It Out Myself: A 2D Anime MV with Video AI (H3 MAX) × 3D Stick-Figure Capture

We set out to make a 2D anime MV for a rock band with video AI (fal.ai H3 MAX), but the character just would not move like a rock singer. Detailed prompts produced the wrong kind of motion, and free 3D motion data turned into a video-game walk cycle. After a lot of trial and error, I ended up performing it myself in front of my phone.

I tried to make a rock singer move with AI and ended up acting it out myself: Video AI (H3 MAX) × 3D stick-figure capture for a 2D anime MV

Hi, I'm Rey! I produce a 6-piece virtual rock band called WASHABI using AI.

When making anime MVs for the band with video AI (for example H3 MAX or H3 MAX Turbo on fal.ai), I kept running into the same problem: the characters just would not move like rock musicians.
(Note: For our usual cuts that are "animated from a start frame and an end frame," we use the Image-to-Video model H3 MAX Turbo. But when we pass three inputs at once—a guide video + a character reference sheet + the vocals—as in this MV, we use the Reference-to-Video model H3 MAX (minimax/h3-max/reference-to-video). The basics are covered in Image & Video AI Basics: Making Illustrations & Anime Videos of the Same Character with gpt-image-2.5 and fal.ai (H3) (JP).)

While making the one-shot MV for our new song "#035 Secret" (54.8 seconds, garage rock), we tried all kinds of approaches—"prompt instructions to the AI," "free 3D motion capture," and more—and learned this: "Don't build every scene the same way. The right answer is to choose the method by motion type: (1) sitting, (2) walking, (3) singing with the whole body!"

At first, we tried to leave everything to AI or free 3D motion data, but we simply could not get motion that felt like a rock singer. In the end, we arrived at this method: "From motion I acted out in front of my phone, build a 3D stick figure + a dark-blue shadow with thickness (Roku's body width added around the 3D bones) + a silver mic, place it in the guide of a blue 3D room, and hand that to H3 MAX."

💡 I had AI agents build the 3D stick-figure capture and the 3D room placement

This post includes steps like "extracting 3D joints (bones) from a phone video" and "placing the stick figure and a silver mic in a blue checkerboard 3D room built in Blender," but I didn't do any of it by hand.

My part was mainly to "air-sing along to the song in front of my phone" and to check the results. The Python scripts for skeleton extraction, beat sync, and 3D guide generation were written by AI agents (Claude Code, Codex, Antigravity, etc.) over many rounds of back-and-forth. At the end of the article you'll also find a starter prompt for AI agents.


1. What Can This 3-Scene Approach Do?

First, take a look at Chorus 1 of "#035 Secret," made with this method (Roku puts a foot up on the monitor, looks down, and shouts while riding the rhythm with her whole body). This kind of motion is really hard to get out of video AI.

Finished Chorus 1 (from #035 "Secret")

Roku's first chorus: her left boot up on the stage monitor, knees sinking to the shuffle beat, tossing her head and leaning back as she sings (how we composited the 3D background and neon lyrics is explained in Article ①).

In this post, we split the MV's motion into 3 scene types (sitting, walking, singing) and go through what happened in each one and how we solved it.

Motion design by scene type in 3 steps: Step 1 (seated scenes = leaving it to H3 MAX works) → Step 2 (walking scenes = 3D mocap looked like a game → solved with a stick-figure motion of my own acting) → Step 3 (singing scenes = prompt instructions gave a stiff, standing-still result → solved with a stick-figure motion of my own acting)
Motion design by scene type in 3 steps: Step 1 (seated scenes = leaving it to H3 MAX works) → Step 2 (walking scenes = 3D mocap looked like a game → solved with a stick-figure motion of my own acting) → Step 3 (singing scenes = prompt instructions gave a stiff, standing-still result → solved with a stick-figure motion of my own acting)
  • Step 1 (seated scenes): For scenes with no big movement, like singing while sitting on a chair in the verse, the most natural result came from not making a motion guide at all and leaving it to H3 MAX with just "a blue checkerboard room + a limbless shape that marks the position."
  • Step 2 (walking scenes): A plain walk works fine with H3 MAX, but matching the song's rhythm or adding changes partway through is hard with H3 MAX alone. Free 3D motion data turned it into "video-game locomotion," so we solved it with a stick-figure motion guide of my own acting.
  • Step 3 (full-body chorus singing): Left to H3 MAX, Roku just stood in place. Directing the motion in detail with prompts produced unnatural movement. And we couldn't find any "rock singing" data in 3D motion libraries at all. Handing over a stick-figure motion guide of me performing all-out produced the best rock motion!
📚 Related reading: H3 & Blender basics + the 3D room compositing article

2. Step 1 (Seated Scenes): For Verses Without Big Movement, "Leave It to H3 MAX" Works!

First up are Verse 1 (Roku sits at a round table in the audience area, chin in hand, and languidly starts singing) and Verse 2 (she sings while sitting at the bar counter).

For these "seated scenes with no big full-body movement," we didn't need to hand over human motion or a 3D skeleton at all.
In fact, when we tried to make H3 MAX trace detailed motion keyed onto the joints of a 3D mannequin, the shoulders and elbows went stiff. "Placing only a simple limbless shape (a marker for where she sits) in the blue checkerboard room and leaving all the acting to H3 MAX and the vocals" gave far more natural acting!

Here is the source video for the final Verse 1 (blue checkerboard background) that H3 MAX generated from "just a position marker."

Step 1 result: Verse 1 generated by leaving it to H3 MAX (blue checkerboard source video)

We didn't pass any motion guide, yet as the camera circles around her, H3 MAX naturally performs a detailed sequence—"chin in hand → looks up and rests both arms on the table → shrugs and turns away with a pout → looks down at her glass"—with lip-sync matched to the vocals!


3. Step 2 (Walking Scenes): Walking to the Rhythm — "H3 MAX Only → 3D Mocap Failed → My Own Acting"

Next is the interlude (Roku walks across the stage to the guitar riff and the "Ah~ ha~ ah~!" vocal ad-lib). Here we went through three rounds of trial and error.

① A plain walk works with H3 MAX alone, but syncing to the rhythm or adding changes mid-walk is hard

Just like the verse, we first gave H3 MAX a guide with no motion guide—only "the 3D camera move + a limbless cylinder moving along"—and let it walk.

Step 2-①: Interlude with H3 MAX walking on its own, no motion guide

If she just walks at a steady pace, H3 MAX alone draws it beautifully. But locking her stride to the song's shuffle beat, or adding dynamics like "throwing in a provocative ad-lib mid-walk to land a hit," was hard to get by leaving it to H3 MAX.

② Generic 3D motion data (CMU) turned it into "video-game motion"

Our next idea: "What if we pour free, publicly available 3D motion capture (the CMU Graphics Lab Motion Capture Database) into a 3D model, line up the foot contacts and the moment of the jump exactly with the song's beats, and have H3 MAX redraw it? That should look cool, right?" So we experimented.

First, here is the motion video with the generic motion data applied to a 3D model and adjusted to the song's beats.

Step 2-②a: Motion video with generic 3D motion data (CMU) applied to a 3D model

A 3D video where we put a run-and-big-jump motion clip (CMU `49_04`) on a 3D model and aligned the takeoff and landing to the song's beats frame by frame.

And here is the result of using this 3D motion as a guide and having H3 MAX redraw it as Roku.

Step 2-②b: Roku video generated by H3 MAX from the 3D motion above

The takeoff and landing land within 1 frame of the song's beats, but the arms-spread jumping pose and the upper-body balance looked, no matter how you see it, like "movement and jump animations from a 3D action game"...

Generic 3D motion data is a record of "everyday walking and sports movements," not rock band stagecraft. So even with AI painting 2D anime art over it, the "game character moving around" feel never went away.

③ So I Acted It Out Myself! (Stick-Figure Capture Guide → Done!)

So I propped up my phone, played the song, and performed the walk to the rhythm myself, fully in character as Roku!

(1) Film my own acting on a phone → (2) convert it into a 3D model (stick figure + dark-blue shadow) and change the camera angle → (3) hand it to H3 MAX to redraw as a 2D anime Roku
(1) Film my own acting on a phone → (2) convert it into a 3D model (stick figure + dark-blue shadow) and change the camera angle → (3) hand it to H3 MAX to redraw as a 2D anime Roku

The phone video never leaves my computer. On my local PC, it is first converted into a 3D model (bone lines + joint points + a silver mic in the right hand), then given "a dark-blue shadow with thickness matched to Roku's body width (capsule-shaped thickness added to the 3D bones)," and placed on the walking path in the blue checkerboard 3D room.
(Note: With only thin stick-figure lines, H3 MAX tends to mistake them for "lines in the background" or "a pattern on the clothes." On the other hand, if you add a shadow thicker than the real body, it fills the extra area with "black haze" or "mysterious poles." The key is to pass a dark-blue shadow around the 3D bones that exactly matches Roku's body width. Because it goes through 3D once, you can change the camera angle freely, and no live-action footage or real body silhouette ever appears!)

First, here is the "stick figure + dark-blue shadow guide video in the blue checkerboard room" built by capturing my motion.

Step 2-③a: Interlude stick figure + dark-blue shadow guide captured from my motion (blue checkerboard room)

A guide video where the 3D bones extracted from my acting on the phone (left side = red, right side = light blue, spine = white) are given Roku's body-width "dark-blue shadow" and a silver mic in the right hand, then placed in the blue checkerboard 3D room.

And here is the "final interlude source video (blue checkerboard background)" that H3 MAX generated from this motion!

Step 2-③b (result): Final interlude source video generated by H3 MAX from my motion

The game-like feel from the 3D mocap is completely gone. She keeps time with her shoulders on the shuffle beat and walks while singing provocatively—a groove only a rock band has!


4. Step 3 (Singing Scenes): The Most Important Part, the Chorus — "H3 MAX Stands Still → Prompts Failed → My Own Acting"

Then comes the biggest climax of the MV: "Chorus 1 (Roku puts a foot on the monitor, looks down, and sings hard with her whole body)." Here, too, we tested things step by step.

① Left to H3 MAX as-is, the video has no big movement

First, just like the verse in Step 1, here is the Chorus 1 source video where we only placed a position marker and left the acting to H3 MAX.

Step 3-①: Chorus 1 source video with H3 MAX singing on its own, no motion guide

It adds pointing and hand gestures, but her feet stay planted and she sings politely with only her upper body. Against the explosive rock energy of the chorus, we couldn't shake the "just standing there" feeling.

② Directing the motion in detail with prompts still gives strange movement

"Then what if we drop the figure guide and direct rock-style motion thoroughly in words (the prompt)?" With that idea, we passed a camera guide of just the blue room and sent H3 MAX a passionate prompt like this.

The prompt we actually sent (excerpt):
*She is a wild rock singer with a heavy, low groove, full of attitude. On the first words she drops low with her knees bent and her head bowed over the microphone. She kicks one boot up behind her, hops and stamps on each 'naku', throwing her head. Then she stands wide and sings hard into the microphone, one hand in her hair, and throws her free arm out to the side on the last words.*
('naku' is a word in the Japanese lyrics.)

Here is the video that came back from H3 MAX.

Step 3-②: Chorus 1 source video with rock motion directed in detail by prompt

There is motion, but it's so "not it"...

③ 3D motion data has nothing like "a rock singer singing"

So what about the free 3D motion data (CMU, etc.) we tried in Step 2? General-purpose motion databases have actions like "running, walking, jumping, boxing," but we couldn't find any data for "the motion of a rock vocalist with one foot on a stage monitor, bouncing the knees to a 103 BPM shuffle beat while gripping the mic and shouting."

④ I Acted It Out! (Beat Sync with 3 Claps & Converting to a Low-Angle Shot Looking Up via a 3D Stick Figure → Done!)

That's when I made up my mind: "The feel of a rock chorus can only come from a human riding the song and acting it out!" So I performed Chorus 1 in front of my phone, too, air-singing all-out.

There were two tricks here:

1. Sync to the song's beat with millisecond accuracy using "3 claps" at the start:
- In the bar right before the performance (beats 1, 2, 3 before the chorus), I clapped three times toward the camera—"clap, clap, clap"—before starting to sing. This let us lock the frame times of the claps to the song's beats with millisecond accuracy.

2. Converting the video into a 3D model (3D stick figure) once lets the camera swing freely to an "ultra-low angle looking up," even from a front-facing phone shot:
- The phone shot was taken from the front at eye level. But because the video is first converted into a 3D model (stick figure + a dark-blue shadow at Roku's body width) and placed on top of the stage monitor, we can shoot it with a 3D camera looking up from an ultra-low angle just above the floor and still get a three-dimensional guide of her looking down with her left foot on the monitor. (Technical note: we use Google's pose estimation library MediaPipe to extract 3D joint coordinates from the phone video, and place them scaled to Roku's height of 1.48 m. Also, so that the small rock sways don't get flattened, we keep the joint smoothing width to just "1 frame.")

Step 3-④: Comparison from the Chorus 1 "stick figure + dark-blue shadow guide" to "Roku redrawn by H3 MAX"
Step 3-④: Comparison from the Chorus 1 "stick figure + dark-blue shadow guide" to "Roku redrawn by H3 MAX"

Here are the Chorus 1 motion guide videos built by capturing my motion (first half: looking up at the monitor / second half: full-body action)!

Step 3-④a: Chorus 1 first-half guide captured from my motion (stick figure + dark-blue shadow)

A 3D stick figure + dark-blue shadow guide, from a low angle up to a face close-up, with the left foot on the stage monitor, swaying to the beat while looking down.

Step 3-④b: Chorus 1 second-half guide captured from my motion (stick figure + dark-blue shadow)

A full-body guide: the foot comes down from the monitor, the knees sink deep, a hand goes to the head, and on the last line "Tsukareru desho!" she leans back and shouts.

And now, take a look at the "final Chorus 1 source videos (blue checkerboard background)" that H3 MAX generated from this motion!

Step 3-④c (result, first half): Chorus 1 first-half source video generated by H3 MAX from my motion

Her boot planted firmly on the corner of the monitor speaker, swaying her upper body to the beat while looking down—a rock camera shot achieved without a single frame of lag!

Step 3-④d (result, second half): Chorus 1 second-half source video generated by H3 MAX from my motion

From the hand-to-the-head gesture to the leaning-back shout, a human rock groove turned straight into Roku's 2D animation!


5. Copy-Paste Ready! AI Agent Prompt for "3D Stick-Figure Capture × H3 MAX 2D Anime MV Production"

The prompt below is a starting point for having an AI agent (Claude Code, Codex, Antigravity, etc.) set up a pipeline that turns an acting video you shot on your phone into a 3D stick figure + dark-blue shadow guide and animates it with video AI (H3 MAX). Adapt it to your footage and song, and review the result of each step as you go.

# For AI Agents: Scene-by-Scene Motion Design (Sitting, Walking, Chorus Singing) & 3D Stick-Figure Capture × H3 MAX Control Spec

You (the AI agent) will switch guide methods according to how intense the motion is in each MV scene. From an acting video the user shot on a smartphone or similar, extract a "3D skeleton (bones and joint points)" with MediaPipe, add a "dark-blue shadow (capsule shapes)" around it that matches the character's body width, project it through the camera of a blue checkerboard 3D room in Blender, and generate lively 2D character animation with video AI (fal.ai `minimax/h3-max/reference-to-video`).

## Step 1: Seated scenes (small acting)
1. Do not use a 3D skeleton or live-action motion. Place only a "limbless shape that marks the position" in the blue checkerboard 3D room.
2. Leave small gestures (chin on hand, gaze, arm movement) and lip-sync to the prompt and the vocals (`reference_audio_urls`).

## Step 2: Walking / moving scenes (interludes, etc.)
1. Do not use generic 3D motion capture (it looks like a game). Place the 3D joints extracted from the user's acting video (the `world` coordinates in `pose_*.json`) on a walking path in 3D space (Pchip interpolation), rotate them toward the direction of travel, and ground the feet at floor height.
2. Add dark-blue (e.g., `RGB(10, 30, 150)`) capsule shadows around the 3D bones to match the character's body width, draw an 18 cm silver microphone from the right hand (joint 16) toward the mouth (joint 0), and project it with the blue checkerboard 3D room camera (`cameras_full.json`).

## Step 3: Full-body, high-energy singing scenes (chorus, shouts)
1. **Privacy protection (never reveal the real person or a live-action silhouette)**: Read the original live-action video only from a local temporary folder, and never copy or send it to external APIs or public folders. Visualize using only the capsule shadows, bone lines, joint points, and silver microphone computed from the 3D bones.
2. **Beat sync with clap markers**: Detect the frame times of the 3 claps at the start of the acting video (e.g., on beats 1, 2, 3 of the bar before the chorus), and align the offset to the song's beat times with millisecond accuracy.
3. **Support for low-angle shots looking up**: Scale the 3D world joints to the character's height (e.g., 1.48 m to the nose) and pin the left foot (joints 27 and 31) to the ridge of the stage monitor, so that an ultra-low-angle, looking-up 3D camera projection is possible even from a smartphone video shot from the front.
4. **Keep smoothing to a minimum (1 frame)**: Convert 30 fps to 24 fps by time interpolation, and limit Gaussian smoothing of the joints to a width of 1 frame on each side (`sigma=1.0`) (at 2.5 frames or more, the small rock sways and groove of 0.4 seconds or less get flattened).
5. **Guide length and splitting**: For a single guide longer than 4 seconds, H3 MAX plays the guide back slowly and lag accumulates, so split it into pieces of about 3 seconds (73 frames) and use the last good frame of the previous video as the next start frame.

6. Related Guides (#035 "Secret" Making-Of Series & Detailed H3 / Blender Guides)

Detailed setup and usage of H3 MAX (fal.ai) and Blender 5.2 introduced in this article, plus the 3D room compositing and songwriting stories behind "#035 Secret," are covered in the articles below. Be sure to check them out!

🎛️ Interactive Suno v6 Sound & Prompt Catalogs (5 Categories · 107 Styles · 131 Audio Previews)

Compare 131 Suno v6 audio clips across Synth/Keys, Vocals, Guitar, Bass, and Drums—and copy tested 4-Block prompts or save ★ bookmarks in one click.

🎹 Synth & Keys (28) 🎤 Vocal & Singing (24) 🎸 Guitar (20) 🎸 Bass (19) 🥁 Drums & Percussion (16)

🎛️ More MAKING / LAB Guides in English

All English Guides →