📽 openpilot Replay Clipper

Make compact, shareable openpilot replay clips with UI overlays, driver debugging, 360 renders, and more.

Capture and develop clips of openpilot from comma.ai's Comma Connect.

Ready to make a clip? Start with Quick Usage.

The clipper can produce clips of:

• ui, the comma.ai openpilot UI, including desired path, lane lines, modes, etc.
• ui-alt, a telemetry-present alternate UI render family with explicit compositions:

  • device keeps one main camera view and adds telemetry alongside it

  • stacked_forward_over_wide shows the forward/road view above the wide view. Adds a cyan rectangle to show what part of the scene the model is fed with.

  • stacked_wide_over_forward same as above but the wide view is above the forward/road view

    

• driver-debug, a driver camera replay/debug layout

  • Replays the driver camera, draws a coarse driver-face box estimate, and adds a large telemetry footer with driver monitoring state, awareness, distraction, and pose/model values.

    

• Camera-only forward, wide, and driver renders with no UI
  • Concatenate, cut, and convert the raw, low-compatibility, and separated HEVC files to one fairly compatible HEVC MP4 or super-compatible H.264 MP4 for easy sharing.
• 360 video
  • Rendered from Wide and Driver Camera. Uploadable to YouTube, viewable in VLC, loadable in 360 video editing software such as Insta360 Studio or even the Insta360 mobile app, and accepted by any video players or web services that take 360 videos.
• 360-ui
  • A 360 video with the openpilot HUD and driving path composited into the wide-camera half before spherical conversion; the header image above is a 360-ui example. It is much slower than plain 360 because it has to replay the UI overlay frame-by-frame.
• forward_upon_wide and 360_forward_upon_wide
  • Forward video is automatically projected onto the wide video using logged camera calibration. Not perfect, but much better aligned than the old manual overlay.
  • 360 Forward Upon Wide scales and renders the final result at a higher resolution to assist in reframing the 360 video to a normal video if that's what you want.

Common features across render types:

• Debug/reporting context can be branded into clips, including route id, platform, git remote, branch, commit, git's dirty state, and a running route timer.
• Rendered clips are designed to be shareable in places like the comma.ai Discord's #driving-feedback and/or #openpilot-experience channel, reddit, Facebook, or anywhere else that takes video.
• Clip options have configurable target file sizes for platforms like Discord that limit uploads.
• Clips can optionally include the qcamera AAC audio track when the route has one uploaded.
• Outputs are packaged into practical MP4s instead of the raw, low-compatibility camera files from the device.

The clipper is deployed on Replicate:

https://replicate.com/nelsonjchen/op-replay-clipper

Replicate is an ultra-low-cost pay-as-you-go compute platform for running software jobs. Replicate is a great way to run this clipper as it's fast, easy to use, and you don't need to install anything on your computer or even deploy anything yourself. Just enter in the required information into the form, and Replicate will generate a clip. Expect to pay about ~$0.01 per clip but not even need to put in any payment details until you've reached a generously large level of usage.

If you want to run the clipper yourself, be aware that this codebase is not primarily geared toward polished local installs across every possible setup due to level of code and support needed. Users going that route are welcome to fork the codebase and let a coding agent work through what is needed for their machine. Doing that ahead of time for everyone is not realistic because local GPU, OS, driver, video codec, and openpilot build setups vary wildly. Good luck to those folks, genuinely.

Warning

comma devices should not be used as primary dashcams, especially because crash-related power cutouts can lose footage.

They are still great as a backup dashcam, openpilot, and for other purposes though.

Terminology

• Route - (comma terminology) A drive recorded by openpilot. Generally from ignition on to ignition off. These look like "fde53c3c109fb4c0/000002ae--7da67a8960".
• Segment - (comma terminology) A 60 second chunk of a route. These look like "fde53c3c109fb4c0/000002ae--7da67a8960/0" for the first segment, "fde53c3c109fb4c0/000002ae--7da67a8960/1" for the second segment, and so on.
• Clip - (clipper terminology) A portion of a route that you want to render and share. These look like "fde53c3c109fb4c0/000002ae--7da67a8960/90/120" for a clip from 1:30 to 2:00.
• Smear - (clipper terminology) A portion of time before the start of the clip that is included in the rendering to help with UI state initialization. This is needed because renders may need some state that may require past data to be sent to get it to the correct state at the beginning of the clip. Smearing helps ensure the UI is in the correct state at the start of the clip, but it also means you may need to upload additional data before your desired start point for renders with UI or renders that use recorded telemetry data like driver-debug or forward-upon-wide-type renders for auto-alignment.

Requirements

• A comma.ai device paired to a comma Connect account you can access.
• Free GitHub account to log into Replicate with

Non-Requirements

• A comma lite or prime subscription.
  • Clipping was a comma connect prime-only feature but was removed for refurbishment on October 15, 2023 and has not returned. This is a free and open source tool to do the same.

Quick Usage

We assume you've already paired your device and have access to the device with your comma connect account.

0. (Optional) Bookmark first, clip later. A bookmark is not required to make a clip, but it makes the moment much easier to find later. When something interesting happens and it is safe to interact with the device, bookmark it right away. Bookmarking drops a yellow marker on the route timeline and helps preserve the nearby route segments.

   • On comma 4 / mici, swipe left from the right edge of the onroad UI.

     

   • On the C3 or C3X tici/tizi UI, tap the onroad UI to reveal the sidebar, then tap the bookmark flag button.

     

   • In comma Connect, bookmarked moments show up as yellow markers on the route timeline.

     

1. Visit comma connect at https://connect.comma.ai and select a drive or route.
   • 
2. Scrub the seek bar to the time you want to clip.
   • In this example, I've scrubbed to a time where I had some issue with the steering wheel shaking and I want to make a clip of it to share on the comma discord for feedback.
   • 
3. Now I need to select the portion of the drive or route I want to clip. Here's a GIF of what that UI looks like
   • See how I drag and select a portion.
   • You can see me make a few mistakes but pressing the left arrow (←) in the top-left corner lets me re-expand and try to trim again.
   • The clipper has a maximum length of 5 minutes. Try to select a portion that's less than that. Try to aim for 30 seconds to a minute though as everybody has short attention spans.
   • 
4. Once satisified with the selected portion, prepare the route and files for rendering.
   • First, while you are viewing the selected portion, make sure all files for the portion are uploaded. Select "Upload All" under the "Files" dropdown if you haven't already and make sure it says uploaded. You may need to wait and your device may need to be on for a while for all files to upload. Consider bringing in your device and leaving it on a charger in a good wifi area if you have trouble with uploads.
     • The clipper only works with high-resolution files and needs all files that are part of the clip to be uploaded.
     • 
     • 
   • Make sure the route has "Public access" under "More info" turned on. You can set this to off after you're done with clip making.
     • 
     • 
5. Copy the URL in the address bar of your browser to your clipboard. In the case above, I've copied the URL of "https://connect.comma.ai/fde53c3c109fb4c0/000002c4--4347572a9e/2391/2409" to my clipboard.
   • When you were adjusting the selected portion of the route in a previous step, it was changing those last two numbers in the browser address bar URL which is the start time and end time respectively.
   • "Share This Route" button if it is present will work too. Choose "copy to clipboard" or similar.
6. Visit https://replicate.com/nelsonjchen/op-replay-clipper
7. Under route, paste the URL you copied in the previous step.
   • 
8. Tweak any settings you like.
9. Press Boot + Run or Run.
10. Wait for the clip to render. It may take a few minutes.
11. Once done, you can download the clip. If you want, turn off "Public access" on the route after you're done.

    • Here's a generated clip with ui-alt:

      

    • You can reupload this file onto Discord. Be aware of Discord's file size limits. Discord Free users should target 9MB file sizes for rendering to slip in under the 10MB limit.

Driver Related Clip Usage

If driver camera footage is uploaded, you can make driver-related clips such as driver-debug and 360/360-ui. These can be very helpful for debugging driver monitoring issues or just sharing funny or interesting driver camera moments.

You'll need to turn on driver camera recording in openpilot to use these features. No data, no video!

Driver Debug

openpilot's Driver Monitoring isn't perfect. The driver-debug render type is designed to help view or observe these moments by compositing the driver camera footage with a large telemetry footer showing the driver monitoring state, awareness, distraction, and pose/model values. It also draws a coarse driver-face box estimate on the driver camera to help see where the model thinks the driver's face is along with passenger seat boxes when passengers are detected.

If you are having driver monitoring issues, report it to the #openpilot-experience channel in the comma discord. You don't need to share a driver-debug clip showing the issue but it does help to make sure you've included the data they are looking for.

Anonymization

Driver-backed renders can optionally anonymize people visible in the driver camera. This applies to driver, driver-debug, 360, 360-ui, and 360_forward_upon_wide. It does not affect road-only renders such as ui, ui-alt, forward, wide, or forward_upon_wide.

On Replicate, use the anonymizationProfile setting to choose what happens to the driver and passenger seats:

• none leaves the driver camera unchanged
• driver unchanged, passenger hidden keeps the driver as-is and hides the passenger
• driver unchanged, passenger face swap keeps the driver as-is and face-swaps the passenger
• driver face swap, passenger unchanged face-swaps the driver and leaves the passenger as-is
• driver face swap, passenger hidden face-swaps the driver and hides the passenger
• driver face swap, passenger face swap face-swaps both visible seats

When a profile uses passenger hidden mode, passengerRedactionStyle controls the look: blur, silhouette, black_silhouette, or ir_tint. Face-swapped seats use FaceFusion with the bundled donor bank, and the hosted model uses an automatic same-tone donor selection pass before rendering.

Anonymized outputs burn a bright banner into the driver video, such as PASSENGER BLURRED or DRIVER SWAPPED, PASSENGER BLURRED, so viewers can tell the clip was altered.

This is best-effort privacy tooling, not a guarantee. Check the output before sharing it. Reflections, clothing, voice/audio, route context, the car interior, or imperfect detection can still identify people or places. If the passenger must not be visible, prefer a hidden-passenger profile over passenger face swap.

Reframing 360 and 360 Forward Upon Wide to a normal video

360 videos are cool but sometimes you want a normal video pointing at a specific direction or directions from that data.

https://github.com/user-attachments/assets/08b51cee-f357-4afc-87f2-4c4d0f6aedba

With 360 videos, it is possible to reframe the 360 video so it is a non-360 video to a normal video pointing at a specific direction.

The best current way to do this is to use a 360 video editor like Insta360 Studio to reframe the video to a normal video. Simply load the 360 video into the editor and reframe the video to the desired direction. A more through description of this functionality can be found at their site.

The Insta360 mobile apps also allow using the phone's movement and swipes for a more natural reframing as well. That is also described at their site

https://github.com/user-attachments/assets/deea7b78-61ee-43be-8a29-38319114c083

There may be alternative software that'll do it and I will take pull requests to add them to this README, but this is the best way I know how to do it and it is free.

The 360 Forward Upon Wide rendering option scales input videos and renders the final result in a much higher 8K resolution to assist reframing with a high resolution forward video. The normal 360 option just glues the videos together. 360-ui keeps the normal 360 output shape but overlays the openpilot HUD/path onto the wide side first, so expect it to cost more time than plain 360.

If wanting to use 360 Forward Upon Wide, test with the non-360 Forward Upon Wide option first so you can quickly sanity-check the route's automatic camera alignment before paying for the larger 360 output.

Gallery

Demonstration of speed or longitudinal behavior of openpilot with model-based longitudinal is nearly impossible or hard without this clipper. This video is of a good model based long behavior at highway speeds.

https://user-images.githubusercontent.com/5363/202886008-82cfbf02-d19a-4482-ab7a-59f96c802dd1.mp4

Cars can have bugs themselves. Here's my 2020 Corolla Hatchback phantomly braking on metal strips in stop and go traffic probably from the radar. Perhaps a future openpilot that doesn't depend on radar might be the one sanity checking the radar instead of the other way around currently. And another example of that in Portland.

https://user-images.githubusercontent.com/5363/219708673-4673f4ff-9b47-4c57-9be3-65f3ea703f3f.mp4

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/1e59844b-46f8-4289-bea9-511db2718549

This is a video of a bug report where openpilot's lateral handling lost the lane.

https://user-images.githubusercontent.com/5363/205901777-53fd18f9-2ab5-400b-92f5-45daf3a34fbd.mp4

Lane cutting?

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/d0ab3365-b5ef-4e05-84ee-370b88e8af02

Nav-assisted follow the road instead of taking the side road.

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/8f970c76-21d1-4209-b0e1-3eb6989feea8

Copying the car in front to get around someone waiting for the left turn

https://github.com/nelsonjchen/op-replay-clipper/assets/5363/9f845b8d-e4aa-4ab3-8785-8d09b83c9d8b

Search up the readme for 360 stuff! It's pretty cool.

https://github.com/user-attachments/assets/deea7b78-61ee-43be-8a29-38319114c083

Limitations

• Data from old or forked versions of openpilot may not render well. YMMV.

Advanced Usage

JWT Token Input

There is a JWT Token input field. This is for users who do not wish to set a route to be "Public access". There is a major catch though. The JWT Token is valid for 90 days and is irrevocable in any way. Password changes from SSO account logins like in Comma Connect will not invalidate the token. Addtionally, it is not granular, meaning it will give access to all routes for the user if leaked.

If you share a JWT Token with anyone, they will be able to access all your routes for 90 days with no possibility of revocation from you. This is why it's not recommended to use this feature unless you know what you're doing compared to the "Public access" method which is much easier to revoke access to.

Tokens can be obtained from visiting https://jwt.comma.ai/ and logging in with the same comma connect account type. Tokens should be about 181 characters or longer.

Replicate can queue up jobs to run in parallel

After you run something, just use your browser to "Duplicate" the tab, change the settings for the next thing, and press Run. Replicate will queue up jobs and if necessary, even scale up to run multiple jobs in parallel. Very cool!

Running Locally, Deploying Yourself, and Other Advanced Topics

Most people should use the hosted Replicate flow above. The local/developer path is intentionally more advanced because GPU, OS, driver, codec, and openpilot build setups vary wildly.

• Local CLI, smoke tests, driver anonymization, and local hosted-model helpers: docs/running-locally.md
• Replicate deployment flow: docs/deploying-to-replicate.md
• BIG UI runtime patching and headless rendering background: docs/runtime-patching-and-ui-rendering.md
• Upstream/openpilot/Cog modification inventory: docs/upstream-modifications.md
• Pre-promotion smoke checklist: docs/prod-readiness-checklist.md

Credits

UI

The real MVP is @deanlee for the replay tool in the openpilot project. The level of effort to develop the replay tool is far beyond this project. This tool builds on that replay work to make clipping videos practical.

https://github.com/commaai/openpilot/blame/master/tools/replay/main.cc

Video-only

A lot of the FFmpeg commands is based off of @ntegan1's research and documentation including a small disclosure of some but not all details by @incognitojam when @incognitojam was at comma.

https://discord.com/channels/469524606043160576/819046761287909446/1068406169317675078

@morrislee provided original data suitable to try to reverse engineer 360 clips.

Bookmarking Handout

Share this with people who need a quick visual reminder for how to bookmark moments before clipping.