Filming trains with the video car pulled by my “extended” coupler leaves that coupler visible on screen. It’s a tad distracting.

One question I had is: Can I write a Fusion plugin that would “erase” this object?

Let’s look at the obvious part: how would I erase this object?

We can split the problem in two: how to detect the object, and how to erase it.

First, the “coupler” appears as a gray-ish rectangle. It always touches the bottom edge of the screen and extends upwards. It’s roughly centered horizontally, but can move left and right as the train takes on curves. If we had to modify a movie frame by frame, one optimization would be on a given frame the coupler will very close to its position in the previous frame.

The coupler is more or less a vertical rectangle.

If we were operating as a plugin in Fusion, ideally we could use a 2d ui crosshair to define its initial position, but we would also have to deal with the fact it moves slowly from frame to frame, and tracking would be ideal.

If we were operating as command-line script without UI, the initial position could be given as parameters.

Its width is constant. It tapers as it gets farther from the camera, but overall we could parametrize the max/min width we are looking for -- e.g. what looks like a vertical gray box of W pixels wide and at least H pixels height starting from the bottom of the screen, roughly centered?

If we have a UI giving us a starting pixel position, we can use a basic flood algorithm to detect the boundaries.

When looking at a still image capture, the coupler edges are naturally fuzzy. We cannot expect a stark transition between the coupler and its background. However we can parametrize this, as typically the fuzzy border will have a predictable width.

Although the coupler is mostly of the same gray color, that color can vary depending on illumination, not to mention tunnels, and there’s bound to be shade variation across the length of the object.

One option is to look at the image in HSL -- the coupler being gray should not have a hue.

Assuming we can detect the coupler, how do we “erase” it?

One suggestion is to treat the coupler as a mask of width W:

• Shift the image horizontally left and right by W pixels.
• Merge this over the coupler using an averaging algorithm.
• Using the luminance as the coefficient to merge it.

Assuming this works for one image, the obvious concern is how will it look once applied to a 30 fps movie? We may need to do some temporal averaging -- that is apply a percentage of the last frame to smooth the output.

Implementation wise, why kind of framework would be used?

One option is to make a standalone processing app, either in Java/Kotlin/C++, with its own movie decoding + UI rendering. The input movie decoding would take advantage of either ffmpeg or vlc as libraries or avcodec on linux. Similarly this can write a movie result on the fly.

In this mode, we would convert a raw footage upfront -- cam files ar 10 minutes long.

The other option is to use a Fusion plugin as the framework -- in this case Fusion provides the input and the output. Debugging seems potentially more challenging this way. The Fuse SDK doc indicates the Lua code can be reloaded “live” to ease debugging. This has the side benefit that we can judge performance immediately, and that we can take advantage of the fuse UI widgets to e.g. select the initial position to look at or to select some parameters (width, blending, etc).

In this mode, we could apply the tool to only the segments being rendered in the final production rather than on the entire raw footage.

One thing I don’t see in the Fuse SDK is a way for a “tool” to incrementally operate across frames in an animation. We want to do some basic tracking of the coupler position, and it would be nice if it could be automated. The other option is to simply animate the initial position, and/or to use a planar tracker to define it.

---