So I built an agentic video editor, the video editor I built is a one shot agent that takes a long form live stream video and outputs a condensed fast paced edited video, with the length of the final output based on the user specifications. The way it works is quite simple: the agent is given an index of funny/important events inside of the video, the agent then sequentially uses the look tool to look into 10-180 second segments of the video and then after these looks updates the edit decision list (what sections to include). The agentic workflow is quite simple but getting the agent to follow the workflow is incredibly annoying and frustrating. But the end product is an autonomous editor that can edit a single video in 1/10 of the time and cost of an editor. With the biggest limitation being the scope of the editing possible.
Agent Break Down
Before diving into the architecture and nuances of this system I will explain the base technologies used. To build the index for looks I use a VLM (video language model), VLM’s take in video and text and return text great for video understanding, specifically I used the gemini flash models. The videos that are being sent in are downsampled to 1 fps and 480p. I built my own agent harness with pure python inspired by mini-swe agent. The agent itself is just a loop of reasoning and action until the final result is achieved. It’s important to note that agents are stateless. All of the conversation history is just appended and is part of the next LLM call.
Why use an agentic loop? Originally when I was designing and creating this I followed a workflow procedure that was published in another paper, where a series of LLM calls is responsible for creating the index and one for creating a storyboard and another for retrieving clips. This paper uses a complete text based semantic index for EDL decisions, so when the EDL is being edited there is no video context. Maybe this is reasonable but I thought of two problems with this approach: that this is wasteful and potentially lossy and that they only account for 20 second windows in the index. The 20 second window is limiting because what if the most optimal edit would be in between those 20 second videos, potentially making the video slower paced. So instead I trusted my intuition and built an agentic loop. The benefits of this system is that when making edits to the EDL the model can have the video that it’s editing in its context window, it can choose how much video context it needs, and it can iterate to correct editing mistakes or imperfections. The biggest negative is the added complexity of building a video agentic loop.
Tools
The agent is given two tools: bash commands and looks. The bash commands are identical to those used in coding agents, but used for different purposes. Where the model can get information about the video using ffmpeg commands like ffprobe, but also use pregiven scripts to render the video and make whatever script it thinks would be helpful. These commands are the glue that makes the system autonomous. The next tool the agent has access to is look, where the model is given the context of a certain section of the video. To reduce the size of the context window and save cost, the look tool call expires after a certain amount turns, which the model determines. This is a crucial part for the agentic approach. And it’s also important to note that video is put into the context window temporarily so no long term information is added to the agent’s history, this is necessary to reduce cost, reduce context window size, and stop video data from being lost in the middle.
Index
The agent is also fed an index to provide high level scaffolding and understanding. The index is very similar to the paper, with mine only using coarse segment analysis. The indexer goes through the video in 15 minute chunks, where the first 15 minute chunk establishes the characters and general plot. And each subsequent 15 minute chunk gets both important/funny events and plot events. The plot events are appended to a plot timeline and given to the next 15 minute call. Once this is done an index describing funny events and plot details is created for the entire video.
Workflow
The workflow as stated early is pretty simple and is inspired by an actual editors workflow: first have an understanding of the video and its important moments from the index, then look through each of the potential clippable moments from the livestream using the look tool, then sequential after each look added best cuts to the EDL.
Despite the simplicity of the workflow a herculean effort is needed to get the agent to actually follow it. The biggest issue is index collapse where instead of looking through the video to intelligently make edits the agent simply finds the funniest/interesting moments from the index and stitches together those timestamps never using the look command. Resulting in horrendous choppy edits which make no sense. To avoid index collapse I did three things, first extremely explicitly telling the agent to never use the index to make EDL decisions, implementing guardrails so the agent cannot edit EDL unless it has an unexpired look, and third gave explicit example workflow’s and how to edit the EDL.
Another important engineering hurdle was to incentivize the agent to create intelligent edits that aren’t just good moments but a sequence of clips where each clip needs to make sense in the context of the last clip and the next clip. So to improve flow I added an entry and exit field inside of the json of each clip.
{"src": "...", "in": "...", "out": "...", "note": "why this clip earns its place",
"entry": "what the viewer needs to know at the in-point, and where the previous clip left them",
"exit": "what state/energy this hands to the next clip"}Results
The result is a well put together rough cut of however long the user wants the edit to be mimicking an edited youtube video. In particular the agent’s output have good general video understanding when making edits, something one of the biggest roadblocks in such systems.
In this example I gave the agent first hour of the VOD
Limitations and Future Work
The limitations can be separated into two categories, problems inherent with a single shot agentic editor and problems with the current scope of the project. For the latter, the agent only has access to live stream video, but if multiple streamers have unique perspectives editors will often stitch different cameras and streams together. Another issue is lack of other edits like zooms or music, and broll. For the former a single shot agentic editor is good for someone who doesn’t want to edit and just wants a final output, especially good for efficiency and mass uploading, but will always have little human input. Meaning edits will always lack the human touch and nuance that current models simply aren’t able to implement. This can show up in simple context editions like is lying right now he doesn’t have that car, leading to a less cohesive and poorer quality edit than an expert editor could do.
For future work I will add music, basic edits like zooms and crops, and some basic broll for context. Long term this technology would be very useful to smaller streamers and creators who don’t have the budget for editors or want to spend the time editing a video, I would love to monetize it for such people.
Conclusion
After implementing strict guardrails I create a one shot agentic system that can autonomously create a well edited narrative cohesive youtube-like video. Implemented using a well crafted index made from a series of VLM calls, that feeds into an agentic loop that using well designed tools creates an edit decision list. To avoid index collapse I used extensive guardrails such as schema formatting, explicit prompt instructions, and agent hooks. In the end making a system that can output a narrative cohesive video of any size from an original video.