Greenscreen keying on the edge

Hello again,

I´ve gained some experience with Greenscreen Effects in Kdenlive. The Quality is pretty good.
But there is a little nasty side effect. There are some small greencolourmixed edges on my masked layer, and I don´t want to cut them.

It would be nice if someone could give me a tip, on how to colourcorrect only the edges of an alpha masked layer.

---- FYI ----
If someone is working the first time with Green/Bluescreen here are some Tips
I try to give my best in English :)

Add multiple Keying Effects for more Precision.
Use a neutral Background for Tests to see even a single Pixel changing.
Start with the darkest colour value and invert the Subtraction to control your Result.
End with the brightest an biggest continues canvas.

This might help you to have always the outline of keyed Objects in Focus.

With virtual Regards,

Can you post an example image of your problem? (a still would do)
For example, the selection on a gray background or similar.

Hello Marko,

thank you offering your help.
Here is the latest result of my rendered Image.

You can see a green edge on the sleeve of the shirt. That is the problem :)
Both are nearly clean keyed. It is a still Image in the back.

Thank you.


Looks like spill surpression is required.

I've downloaded the pic, and pixel-peeked it in Gimp. Must say that the key is quite clean - at least on a still, probably looks worse on a moving image?

By the shirt sleeve, you mean on the left woman, her right side?
The right side woman seems to be keyed quite cleanly.

I see worse keyspill on the chair's arm rests in the middle?

Apart from that, i guess a very slight key softening (alpha blur) would be nice, see for example the chin of the left woman. As far as I know, there's no suitable tool currently in Kdelive.
I'll add that (small blur) to the alpha operations plugin, when I'll find some time (don't hold your breath ;-)

As for the keyspill, I'll think about what could be done. For example Cinderela has an option to reduce the saturation close to the edges.
But that wouldn't help much on the arm rests (tried it out in Gimp).
So maybe "growing" the color and luma outwards near the edges?
Any suggestions or hints?

I also have/had plans to work on a new or improve one of the current greenscreen filters in terms of edge behavior but unfortunately I currently don't have much time.
Marko, you might want to have a look at this paper:
Since your select0r already uses the 3D method also employed by the "TZI keyer" from the paper it would be extremely easy to add a softness area. The auto spill removal presented in the paper also shouldn't be a big deal.

Hmm, I have some Ideas how to do an effective spill suppression with Kdenlive. But this will get a bit complicated not at last because of Kdenlive's missing ability to automatically do alpha operations between tracks without transitions.
I will check that out and write a tutorial as soon as I manage to find a good technique ;)

Blender has greenscreen plugin and nodes, could look at that code, RGB & YCC options. There's also a seperate spill supression node, if I get time I can dig out the code for info.

thanks for the link, I'll see what I can do in that direction.

OK, I've read the TZI paper (and some references from it).

For the "spill mop up", I'll probably write a separate plugin, so it will be usable with any color keyer.

Later, I'll add the softness control to select0r, and maybe the 3D cube and hue histogram scopes.

Also, it would be nice to have some way to work around the DV chroma subsampling. The TZI uses simple blurring, but I would like to do something a bit smarter, like matched to the particular subsampling (420, 411...).
Or even smarter, like cross filters? I think this should also go into a separate plugin, as a polished-up chroma is desirable for other purposes besides keying.

But hey, I am promising too much....


Checked with pr0file, and Kdenlive/MLT already seems to do some chroma smoothing/interpolation, so I can at least postpone that :-)

I have written a plugin based on the spill suppression method described in the TZI paper above (Till's post), namely de-saturation of the key color.

I haven't pushed it in the GIT, because I am not sure about it's performance. If anybody is interested, I have put the source code and a Kdenlive effect description file here:

if somebody wants to test it. Compilation instructions are in the comments, then just copy the .so file to /usr/lib/frei0r-1 or wherever you have your plugins. The .xml file goes to ...share/apps/kdenlive/effects or some similar path.

I am testing it on the TZI test videos (, and I suggest you use the same, so that we can talk about the same thing. I have a small screen, so I use "pr0be" as a magnifier, to check the edges, and to pick the color.

What I found out is, that while it works quite well on spills that fall on a dark or neutral object, like black hair (clip_1.2.avi, "girl with glass"), it is more or less useless, when the spill falls on a colored or bright object (clip_1.1.avi, "water flow"), because the key color gets mixed with the object color, creating a gradient, which is hard to cover without hurting other things.
Also, when the edge of a bright colorful object is desaturated, you get a gray rim instead of a key spill.

I'll try to find other algorithms, suggestions welcome.


BTW, these spills, if not too wide, can be suppressed with the existing "alpha operations" plugin, operation "shrink soft".

Also, with selector in RGB mode with blue screen (the above test videos) increasing the B tolerance helps on the hair.

Here is a (rather long) way with onboard Kdenlive effects. I know that this may not be the best way, but it is the first idea I had withour taking a look at the blender despill's source code ;)

What we want to do is to extract (in our HSV system) the hue and value into one channel - I call it the color channel from now - and the value channel into an other channel.
Then we copy the color channel (so we have 2 of them), extract the greens in color channel 1, invert the hsv value and then do a subtractive color mixing with color channel 2. Now let's call the result color channel 3. By mixing Color Channel 1 and two we got a color channel where the greens are compensated out and so turned into greys. We prefer this method to simply desaturating the greens because by desaturating the greens we would simply take all pixels that have a defined amount of green and make them grey, but if it is for example a yellowish pixel that represents a red object that is affected by green spill and so it looks yellow, instead of subtracting the green from the yellow to get nearer to the object's original color (that's what we try to do) we would simply make it ... well grey.
Can't subtract all the green components from our picture. We will have to define some kind of a filter for that.

Unfortunately I do not know a way neither to split the HSV channels nor to separate the greens then. So what I want to do is a quick and dirty workaround.

1. Duplicate the Clip in the Timeline three times and place them one below the other on the tracks Video 1 (now 1), Video 2 (2) and Video 3 (3).

2. 2 gets a Greyscale effect. 3. gets in this order:

Bezier Curves
Bezier Curves
Hue Shift

3. In the first Bezier Curves you select the red channel and pull it completely down. Same with the Blue on Bezier Curves 2.
Now we have done some kind of green-extraction. Now we set the Hue Shift to 180 Degree. Looks sweet now, eh?
Now we have basically the layer whichs color want to subtract from Video 1.
With the Levels and Saturation effect (maybe you want to use an other Curves effect instead. Feel free to do so if it gives you the best results) we can now control the strenght in relation to the intensity of the green (but unfortunately we can not get a control of the Saturation-Gradiation because we are seeing an image that displays a combination of Saturation and Value. But it's okay to work with this for our purpose. Let the levels and Saturation effects as they are for a moment.

4. Now, we have the problem that there is no way to only mix the Color channel of Video 1 and 3. The best way there is (at least I have seen. I'm sure there are more possibilities in Kdenlive) is to mix Video 1 and Video 3 together with an alpha value of 50%. But because Video 3 does only contain the green channel, the overall image gets darker. Because this we have created Video 2 which has a greyscale (value) version of Video 1. Place a transition between Video 2 and 3.
Then place a composite transition between 1 and 2, change "with track" to Video 3 and change the Opacity to 50.
Now change the transition type of the one between 2 and 3 to screen.
Now you should see that the Green turned to grey. The screen transition is not perfect for this purpose. All we want to do is to light up 3 a bit.

Play with the Saturation and Levels of Video 3 and with the Opacity of the Composite transition to get the best results. Maybe add a curves effect to Video 2 to control the brightness. Note: The result will be undersaturated because of the way we are working with our color channels ;) But I think you can get rid of the spill with this method.

The key color is not just one of the primaries, and keyspillm0pup does not work by simply reducing the amount of that primary.
It has its own small color keyer, and is well capable of desaturating greens without affecting yellows.

The problems I encounter are basically different:

1. The key reflection on a bright object usually creates a long color gradient from key to object color, so if you specify a broad enough color subspace, other things will get caught in.

2. The edges are often dark. The color space gets very tight in the dark corner, so again, it is hard to do a good selection.

So what I'll try to do next in keyspillm0pup, is to add an edge detector in the alpha channel, and gate the selection with that.

I don't think the acrobatics described above would help very much. There is only one color space, and RGB, HSV, etc. are just different coordinate systems that we put onto this single color space.
NOTE: with some of the Hxx systems there will be some metric distortions, because of the simplified formulae (for fast computation), but these amount to a few percent and can be neglected in our case.

It is hard to keep the key colored background evenly illuminated. This causes its brightnees to variate, but (mostly) not the hue. That is the motivation to use Hxx spaces to specify keys, and make the tolerance along the luminance axis wider, while keeping the chroma tolerance tight.
Select0r can do that.
Actually the same could be done by allowing rotation of an RGB subspace, but I guess that would be rather unintuitive for the user.

Some keyers allow complicated selection subspaces, like 128 sided polyhedrons, to precisely adapt to the key variability.
I think select0r is even more versatile than that, with its ability to combine simple shapes. What it currently lacks is a "softness" parameter, to define the width of a transition zone.
I plan to add that in the future, either a single softness parameter, or a separate one for each axis.

However, even the most maleable color selection subspace can not solve the problems in the transition areas, where we have a continous gradient between the key and the foreground object colors. This happens where the key color reflects from the sides of the object, on the edge pixels that are partially exposed to object and partly to background, on transparent and fuzzy objects, and with motion blur.

So in principle, any sophistication in the generation of the alpha channel itself, is not enough for perfect compositing.
In the transition areas, one MUST correct the foreground object color too.

Suggestions and links welcome!

@yellow: if you can dig out that Blender code, I'll be very thankfull!

Here's the old outdated YCbCr greenscreen plugin code for Blenders VSE, out of date now but worked well from memory.

For Blender 2.5x RGB color spill node, tarball source link:

Then navigate through the rabbit holes to blender-2.62 -> source -> blender -> nodes -> composite -> nodes

Thanks, I'll have a look at it.
I am already studying some other papers I found on the net.

Still working on this...

Checked the Blender code, but it uses the old "Vlahos" style keying, where the key is one of the primary colors. In Kdenlive (Frei0r) we have more flexible keying, that can use any color as the key, so I had to find other ways to clean the spill.

In the end, I decided to simply move the colors radially away (in color space) from the key color, and it seems to work quite well. Besides that, I'll include the possibility to move towards a "target" color, to desaturate, and to change the luma.

The other thing is how to find the pixels that need cleaning. I decided for three methods: similarity to key color, partial transparecy, and proximity to the edge.

I already have working code for most of this, but since there are so many possible combinations, I am scratching my head about what the parameter set should be, to give the biggest flexibility.

The key spill mop up plugin is now available in frei0r, if you compile with script, it will be available under "Misc".
However, without the effect description file, it is mostly useless, as all parameters are not shown, and defaults won't be set.

So, to use it, put the attaced file (remove "_.txt") into kdenlive's effect directoty.

The README for the keyspillm0pup is this:


After some experimentation with chroma keying, it looked to me that
there is no single method of key cleaning, that works in all
situations, like keyspill on bright, keyspill on dark, etc.
So I included several cleaning options, which can be used alone
or in combination.
In short, it offers three ways of pixel selection (masking), that
can be combined with four types of color changing operations.
The three selection / masking modes are based on
- similarity to key color
- transparency
- closeness to the edge
and the four things that can be done to the selected pixels are
- move away from the key color (De-Key)
- move towards an target color (Target)
- desaturate
- luma (brightness) adjust.


Color difference masks are based on the color of the image, and do
not depend on the alpha from the preceding keying, except for
ignoring the 100% transparent areas, to increase speed.

Transparency and Edge masks are based on the alpha channel from the
preceding keying operation.
Transparency masks will affect only the parts that are neither 100%
opaque nor 100% transparent, based on the alpha values from the
preceding keying operation. The effect will be proportional to the
NOTE: if a "hard key" was used in the preceding keying, there will be
no areas that T operations could affect.
Edge maskss will affect only pixels close to the edge, with the effect
diminishing away from the edge. The outer edge is the edge of the
fully opaque part, where the alpha from the preceding keying
is 1.0 (255).
NOTE: the edge masking algorithm is not yet what I would like it
to be. I will have to look some more into this, and improve it, so
consider it a "temporary solution" that will change in the future.

All masks can be further pruned with two parameters: an "hue gate",
which will limit the mask to hues close to the key hue, and an
"saturation threshold", which will limit the mask to areas with
color saturation above a threshold.


This plugin can be cascaded, but it is not possible to get the same
color based mask in the second instance, because the colors will be
changed by the first instance. To enable two operations with the
same mask, each plugin instance can do two opertions.
With transparency and edge masks, cascading is a bit easier.
If the hue gate and saturation threshold are not used, transparency
and edge masks can be exactly the same in cascaded plugins.


Key color:
This should be the same or similar to the key color used for the
preceding keying operation.

Target color:
This is only used when "Target" operation is used with one of the
masks. The colors in the affected areas will be moved towards this
color, according to the "Amount" parameter.

Mask type:
Selects the type of mask that will determine where the color
altering operations will occur.

For the color difference mask, the range of colors around the key,
that will be 100% affected.
For the transparency mask, the "amplification".
For the edge mask, the width of the affected area.

For the color difference mask, the range of colors outside of
"Tolerance", that will be gradually less affected.
No function for the transparency and edge masks.

Hue gate:
Reduces the mask according to difference from key hue, to prevent
change to pixels that are within the mask, but not polluted by key.

Saturation threshold:
Reduces the mask according to color saturation, to avoid affecting
the neutral areas.

Operation 1:
Selects which of the four possible operations will be done on the
mask-selected pixels. Apart from no operation, there are four
possibilities: De-key, Target, De-saturate and Luma adjust.

Amount 1:
The amount of the selected operation 1, how much the colors will

Operation 2, Amount 2:
Enable a second operation to be performed with the same mask.

Show mask:
This will show the selected mask as a greyscale image, to help with
fine tuning of the masks. Shoud be OFF for the final render.

Mask to Alpha:
Copies the active mask to the alpha channel.
For all normal spill cleaning operations, this should be OFF.
By seting it ON, the keyspillm0pup itself can be used as a keyer,
or to generate some special effects.

Thank you, Marko for your magnificent contribution! To increase awareness of this fantastic plugin you have made, I have created a video tutorial on Youtube showing it in action! Excellent work!


The xml description is now in git and the effect is available in the "alpha manipulation" group. So if you use the latest development versions of frei0r and Kdenlive no further action is required.

Thanks again Marko for this amazing effect.

I don't see it, and I'm using the build script, which as far as I'm aware compiles the latest versions of Kdenlive and fri0r...

Under "alpha manipulation" I only see:
alpha gradient
alpha operations
alpha shapes
blue screen
color selection

...has it moved or changed name?

It comes under Effects - Misc

I have created an alphabetical index of effects and transtions - so I can find them when I forget where they live.