Greenscreen keying on the edge

Can you post an example image of your problem? (a still would do)
For example, the selection on a gray background or similar.

Looks like spill surpression is required.

I also have/had plans to work on a new or improve one of the current greenscreen filters in terms of edge behavior but unfortunately I currently don't have much time.
Marko, you might want to have a look at this paper: http://www.tzi.de/fileadmin/resources/publikationen/tzi_berichte/TZI-Ber...
Since your select0r already uses the 3D method also employed by the "TZI keyer" from the paper it would be extremely easy to add a softness area. The auto spill removal presented in the paper also shouldn't be a big deal.

Blender has greenscreen plugin and nodes, could look at that code, RGB & YCC options. There's also a seperate spill supression node, if I get time I can dig out the code for info.

OK, I've read the TZI paper (and some references from it).

For the "spill mop up", I'll probably write a separate plugin, so it will be usable with any color keyer.

Later, I'll add the softness control to select0r, and maybe the 3D cube and hue histogram scopes.

Also, it would be nice to have some way to work around the DV chroma subsampling. The TZI uses simple blurring, but I would like to do something a bit smarter, like matched to the particular subsampling (420, 411...).
Or even smarter, like cross filters? I think this should also go into a separate plugin, as a polished-up chroma is desirable for other purposes besides keying.

But hey, I am promising too much....

---------------------------------

Checked with pr0file, and Kdenlive/MLT already seems to do some chroma smoothing/interpolation, so I can at least postpone that :-)

Here is a (rather long) way with onboard Kdenlive effects. I know that this may not be the best way, but it is the first idea I had withour taking a look at the blender despill's source code ;)

What we want to do is to extract (in our HSV system) the hue and value into one channel - I call it the color channel from now - and the value channel into an other channel.
Then we copy the color channel (so we have 2 of them), extract the greens in color channel 1, invert the hsv value and then do a subtractive color mixing with color channel 2. Now let's call the result color channel 3. By mixing Color Channel 1 and two we got a color channel where the greens are compensated out and so turned into greys. We prefer this method to simply desaturating the greens because by desaturating the greens we would simply take all pixels that have a defined amount of green and make them grey, but if it is for example a yellowish pixel that represents a red object that is affected by green spill and so it looks yellow, instead of subtracting the green from the yellow to get nearer to the object's original color (that's what we try to do) we would simply make it ... well grey.
Can't subtract all the green components from our picture. We will have to define some kind of a filter for that.

Unfortunately I do not know a way neither to split the HSV channels nor to separate the greens then. So what I want to do is a quick and dirty workaround.

1. Duplicate the Clip in the Timeline three times and place them one below the other on the tracks Video 1 (now 1), Video 2 (2) and Video 3 (3).

2. 2 gets a Greyscale effect. 3. gets in this order:

Bezier Curves
Bezier Curves
Hue Shift
Levels
Saturation.

3. In the first Bezier Curves you select the red channel and pull it completely down. Same with the Blue on Bezier Curves 2.
Now we have done some kind of green-extraction. Now we set the Hue Shift to 180 Degree. Looks sweet now, eh?
Now we have basically the layer whichs color want to subtract from Video 1.
With the Levels and Saturation effect (maybe you want to use an other Curves effect instead. Feel free to do so if it gives you the best results) we can now control the strenght in relation to the intensity of the green (but unfortunately we can not get a control of the Saturation-Gradiation because we are seeing an image that displays a combination of Saturation and Value. But it's okay to work with this for our purpose. Let the levels and Saturation effects as they are for a moment.

4. Now, we have the problem that there is no way to only mix the Color channel of Video 1 and 3. The best way there is (at least I have seen. I'm sure there are more possibilities in Kdenlive) is to mix Video 1 and Video 3 together with an alpha value of 50%. But because Video 3 does only contain the green channel, the overall image gets darker. Because this we have created Video 2 which has a greyscale (value) version of Video 1. Place a transition between Video 2 and 3.
Then place a composite transition between 1 and 2, change "with track" to Video 3 and change the Opacity to 50.
Now change the transition type of the one between 2 and 3 to screen.
Now you should see that the Green turned to grey. The screen transition is not perfect for this purpose. All we want to do is to light up 3 a bit.

5.
Play with the Saturation and Levels of Video 3 and with the Opacity of the Composite transition to get the best results. Maybe add a curves effect to Video 2 to control the brightness. Note: The result will be undersaturated because of the way we are working with our color channels ;) But I think you can get rid of the spill with this method.

Here's the old outdated YCbCr greenscreen plugin code for Blenders VSE, out of date now but worked well from memory.

http://sites.google.com/site/paprmh/greenscreen1.3.c?attredirects=0

http://sites.google.com/site/paprmh/greenscreen

For Blender 2.5x RGB color spill node, tarball source link: http://download.blender.org/source/blender-2.62.tar.gz

Then navigate through the rabbit holes to blender-2.62 -> source -> blender -> nodes -> composite -> nodes

Still working on this...

Checked the Blender code, but it uses the old "Vlahos" style keying, where the key is one of the primary colors. In Kdenlive (Frei0r) we have more flexible keying, that can use any color as the key, so I had to find other ways to clean the spill.

In the end, I decided to simply move the colors radially away (in color space) from the key color, and it seems to work quite well. Besides that, I'll include the possibility to move towards a "target" color, to desaturate, and to change the luma.

The other thing is how to find the pixels that need cleaning. I decided for three methods: similarity to key color, partial transparecy, and proximity to the edge.

I already have working code for most of this, but since there are so many possible combinations, I am scratching my head about what the parameter set should be, to give the biggest flexibility.

The README for the keyspillm0pup is this:

DESCRIPTION

After some experimentation with chroma keying, it looked to me that
there is no single method of key cleaning, that works in all
situations, like keyspill on bright, keyspill on dark, etc.
So I included several cleaning options, which can be used alone
or in combination.
In short, it offers three ways of pixel selection (masking), that
can be combined with four types of color changing operations.
The three selection / masking modes are based on
- similarity to key color
- transparency
- closeness to the edge
and the four things that can be done to the selected pixels are
- move away from the key color (De-Key)
- move towards an target color (Target)
- desaturate
- luma (brightness) adjust.

MASKS

Color difference masks are based on the color of the image, and do
not depend on the alpha from the preceding keying, except for
ignoring the 100% transparent areas, to increase speed.

Transparency and Edge masks are based on the alpha channel from the
preceding keying operation.
Transparency masks will affect only the parts that are neither 100%
opaque nor 100% transparent, based on the alpha values from the
preceding keying operation. The effect will be proportional to the
transparency.
NOTE: if a "hard key" was used in the preceding keying, there will be
no areas that T operations could affect.
Edge maskss will affect only pixels close to the edge, with the effect
diminishing away from the edge. The outer edge is the edge of the
fully opaque part, where the alpha from the preceding keying
is 1.0 (255).
NOTE: the edge masking algorithm is not yet what I would like it
to be. I will have to look some more into this, and improve it, so
consider it a "temporary solution" that will change in the future.

All masks can be further pruned with two parameters: an "hue gate",
which will limit the mask to hues close to the key hue, and an
"saturation threshold", which will limit the mask to areas with
color saturation above a threshold.

CASCADING

This plugin can be cascaded, but it is not possible to get the same
color based mask in the second instance, because the colors will be
changed by the first instance. To enable two operations with the
same mask, each plugin instance can do two opertions.
With transparency and edge masks, cascading is a bit easier.
If the hue gate and saturation threshold are not used, transparency
and edge masks can be exactly the same in cascaded plugins.

PARAMETERS:

Key color:
This should be the same or similar to the key color used for the
preceding keying operation.

Target color:
This is only used when "Target" operation is used with one of the
masks. The colors in the affected areas will be moved towards this
color, according to the "Amount" parameter.

Mask type:
Selects the type of mask that will determine where the color
altering operations will occur.

Tolerance:
For the color difference mask, the range of colors around the key,
that will be 100% affected.
For the transparency mask, the "amplification".
For the edge mask, the width of the affected area.

Slope:
For the color difference mask, the range of colors outside of
"Tolerance", that will be gradually less affected.
No function for the transparency and edge masks.

Hue gate:
Reduces the mask according to difference from key hue, to prevent
change to pixels that are within the mask, but not polluted by key.

Saturation threshold:
Reduces the mask according to color saturation, to avoid affecting
the neutral areas.

Operation 1:
Selects which of the four possible operations will be done on the
mask-selected pixels. Apart from no operation, there are four
possibilities: De-key, Target, De-saturate and Luma adjust.

Amount 1:
The amount of the selected operation 1, how much the colors will
change.

Operation 2, Amount 2:
Enable a second operation to be performed with the same mask.

Show mask:
This will show the selected mask as a greyscale image, to help with
fine tuning of the masks. Shoud be OFF for the final render.

Mask to Alpha:
Copies the active mask to the alpha channel.
For all normal spill cleaning operations, this should be OFF.
By seting it ON, the keyspillm0pup itself can be used as a keyer,
or to generate some special effects.

The xml description is now in git and the effect is available in the "alpha manipulation" group. So if you use the latest development versions of frei0r and Kdenlive no further action is required.

Thanks again Marko for this amazing effect.

It comes under Effects - Misc

I have created an alphabetical index of effects and transtions - so I can find them when I forget where they live.

http://userbase.kde.org/Kdenlive/Manual/Effects_And_Transitions