I have detailed in a previous post how I create my “car/cab ride” videos: a Mobius Maxi 4K is placed on a flat car, and either pulled or pushed by the train using a custom “3D-printed rod” draw-bar connector:

The only issue with this approach is the extremely visible rod on the camera, even after cropping a 1080p out of the 4K video. I’ve had many comments that it’s quite distracting; I fully agree; it’s been bugging me too:

Could there be a way to go from this:	to that?

I’ve been thinking about that for a while. I can do better. There are a number of “object removal” techniques in NLEs these days, but none that I know to be available in “community” editions of these softwares, and I’ve not been impressed by the trial versions I’ve seen -- for example there are a number of workflows that start by “lasso/select the object to be removed in the frame”, and that’s already a no-starter as I’m not going to select that draw-bar in every single frame.

Instead it looks to me like an algorithm could “easily” find that bar in the image because it has some very distinguishable characteristics. For starters, it’s a rather uniform color which always contrasts with the rails or engine background. And then it’s mostly vertical (but not quite, especially in turns), and finally it always reaches to the bottom of the image.

I’ve looked into a number of solutions on how to detect the draw-bar. My initial choice was OpenCV but I didn’t see a detector that would work for my case; the second choice was AI since that’s the killer buzzword these days, and I don’t see how I can reliably train a model to detect that.

However, good old school image analysis seems obvious here: if I can track at least one point that is on that gray rod, then a simple line-scanning algorithm can detect the left/right borders, followed by another simple flood-fill-like algorithm to follow that vertically. A few experiments also showed that if I work in HLS, the luma channel is all I need, and I can boost the contrast to the max to make the borders even more recognizable. In essence, I have a very dedicated boundary detection algorithm.

The next step is to do the “removal” part… It’s nice to detect the draw-bar, but what should it be replaced with? Here we can take a big shortcut: the background is mostly the track, with rails on each side and horizontal ties. See where I’m going here? If we can detect the space between the rails, we merely need to get the pixels on the left and right of the draw-bar and do a simple fill with some blending, horizontally, line by line. So that’s what I tried and the results were very promising from the start. Here’s an example of the track-filling algorithm showing how good that naïve simple method can get:

Can you spot where the draw-bar used to be?

As for the implementation and usage, I decided to go with DaVinci Resolve since it’s my current de-facto NLE. Turns out that DaVinci Resolve integrates Eyeon’s Fusion since version 16, and that Fusion has an excellent plugin system called Fuses with a very easy to use Fuse SDK. Fuse scripts are written in Lua, which one learns in seconds, and they run very fast with LuaJIT.

The result is a fairly simple DaVinci “Fuse” script that I wrote and refined over two weeks:

https://bitbucket.org/alf-labs/lightworks/src/master/fusion/fuses/RalfCamCarRodRemoval.fuse

Performance is actually better than I expected. Since I’ve never used Lua nor the Fuse SDK before, I’m sure there are things that could be improved.

To use this, I first set a “Tracker” in Fusion to track the top part of the rod draw-bar. That defines an origin point in the rod that moves with it in the video, then my algorithm detects the rod and replaces it. There are a few parameters I can tweak.

The result is pretty good, keeping in mind I started this project a couple weeks ago.

There are a few limitations:

• One is turnouts… obviously turnouts have at least one rail crossing the track so that makes a somewhat odd effect, but it’s acceptable.
• The other thing that I did not think of is… tunnels. Turns out it’s really hard to detect an object when it becomes black on top of black, who knew! That just needs to be solved by carefully editing these sections out.
• Finally there’s the part where the rod draw-bar connects to the engine. I’ve tried removing it but that creates unsightly artifacts, so for now I’m keeping it. I’ll figure that one out in version 2.0 of the plugin.
• Another week of work, and after some more research and trial-and-error, I managed to remove the draw-bar coupler from the video using a combination of DaVinci Fusion “Clone Paint” tool and a Color Node with an image tracker.

Here’s a new render from one of my older videos, applying the new effect plugin.

New version:

Old version:

---