Video to video

Making a new clip that takes its motion, layout or camera from an existing video.

In plain words

A source video can guide a new clip in several ways:

  • Restyle by partial denoise: encode the video into a latent and redraw it with denoise below 1, as image to image does for pictures. Motion stays, the look changes; too low a denoise changes little, too high loses the motion.
  • Control signals: turn each frame into a depth map, a pose skeleton, edges or line art, and let the model follow them while the prompt sets the look. Wan uses VACE or Wan Fun Control models for this.
  • Pose transfer onto a character from a picture: see Pose animation.
  • Camera re-shoot: ReCamMaster redraws the scene along another camera path.

Where you meet it in a graph

  • Load Video (file from input) → Get Video Components gives images (frames), audio and fps.
  • Preprocessors from the comfyui_controlnet_aux pack (Depth Anything, DWPose, OpenPose, Canny, Lineart) between the frames and the control input.
  • WanVideoWrapper: WanVideo VACE Encode (input_frames, ref_images, strength, start and end percent); built-in ComfyUI has WanVaceToVideo for the same idea.
  • Resize and batch-cut nodes (ImageScale, ImageFromBatch) to match the clip's size and length.

Values and what they change

  • Control strength about 1; lower for more freedom.
  • Ending the control early (vace_end_percent 0.5) keeps the layout but lets the model finish the detail.
  • The new clip cannot be longer than the source frames you give.

Typical mistakes

  • Source frames of another size or length than the clip.
  • Raw frames when you wanted only the motion: the colours and clothes come along. Use depth or pose.
  • Using someone's video without the right to: see Content rules.

How the app helps

Lessons VACE: motion from another video, Another camera: ReCamMaster, Video guided by another video: Wan 2.2 Fun Control, and the tools of Load, cut and save video: VideoHelperSuite, A larger, sharper video: SeedVR2, Remove an object from a video: ProPainter and Smoother motion: more frames between frames.

See also

This page is imported from the application repository's wiki (docs/wiki/concepts/Video-To-Video.md, revision b8e3f71). The same guide is built into the app: Help ▸ User guide.

To refresh it, run npm run import and then npm run build in the website folder.