So … this all started when a friend, Melanie, was working on her final paper about Light Graffiti and showed me the video below around mid-September.
Light Painting, also known as Light Graffiti or (partially) Lumasol, is so easy with a photo camera. You take a dark room, use a low shutter speed (several seconds), and paint in the air.
But what about video? You cannot decrease the shutter speed to one frame per minute. Therefore it is not possible.
This really made me think a lot. Actually for several weeks.
After a few minutes of research, I found that the effect used to create the video seems to be kept secret. (At least I couldn’t find any technical information about it.) After some additional research, I read that they actually did this effect in hardware (i.e. programming some camera chip) — but this was after I’d already written this blog post. (Actually, I’m not sure whether it is truely in hardware; some effects suggest that it was done in post as well, like in the part where you only see the light but not the painter.)
Nevertheless, the idea was fascinating, so I decided to start writing an effect frei0r filter that would do the same. It couldn’t be that hard.
That is what I thought at the beginning.
Warning: The first part of this post will only be about the design of this effect. Scroll down if you only want to learn how to use it. If you want to see real-world results first, watch this video done with kdenlive.
For quick navigation …
- Background information: Detecting the light source, Getting a realistic look
- How-to-use information: Summary: How does the effect work and How to apply the effect, and Light Sources
Detecting the light source
My first idea was simple. Light sources are generally brighter than the environment, so I could simply remember every pixel which once had its red, green, or blue value on 255 (this is the brightest possible value) in a light mask. The light mask is then drawn over the current frame. Take the next frame, search for all 255 values, add them to the light mask, and paint it over this frame again.
Obviously this did not work. Due to camera noise several pixels just became colored without me even having started painting yet. So why not additionally taking a look at the sum of the color values. If the sum is big enough, then the pixel is really close to white and not just noise.
I mean, painting with light is cool, but it should not be white only. This is kind of a dilemma now: The green is close to 255, but «close to 255» does not work due to noise. Using bright colors only (i.e. the ones very close to white) does not work either since then the green, or generally colour, is lost. Perhaps taking a look at the saturation can help?
As it seems this is a very bad idea. Huge compression artifacts, no relevant information. (If one cannot even distinguish the light source from the rest by eye, how should the filter do it?) It could not be worse.
Is there still some hope? Certainly yes. There is something we did not make use of yet. The camera needs to stay steady, as you often want to paint around objects to obtain a 3D effect. Like in the introduction video at the beginning. So, why not remember what is background and then assume pixels becoming much brighter than the background to be originated in a light source?
That’s it! Except for some minor details like the hand and some other patches being grey (this could easily be fixed, but I prefer fixing details when the important parts are running) this works really well.
What followed now was a long way of tweaking and tuning, adjusting, testing. Repeatedly ;) After several hours I finally got it. I could reliably distinguish between foreground (light) and background. This via a combination of the absolute brightness of the light source and the brightness difference with respect to the background. I could also vary the light’s brightness!
At this point in time I started writing this blog entry. And when I reached this point, I was at least as disappointed as you when I compared the two above images. All hours spent on tweaking made detecting the light source reliable, but the light itself looks just wrong. So I put everything away for some days to meditate about the important parameters, and finally, after two additional hours of programming …
Making the light look realistic
There were a couple of points I considered when trying to get good-looking lights. Mainly based upon observations I made with light painting pictures. Which were:
- The light source adds light to the image. If it stays at the same position, it keeps on adding light there. If it stays at the same position for some time, it will become overexposed. Green lights will eventually become white.
- A faint trail of a coloured light source is colored as well. Actually, the saturation is bigger than for rather overexposed lights. This is the contrary of the previous point; the effect distinguishes between underexposure and overexposure.
And that was it already — simulating a camera sensor as good as possible.
Using the Light Graffiti effect
Summing it up: How the Light Graffiti effect works
I hope you are at least as confused as I was (if you read the first part). Therefore a quick overview about how the effect really works now.
First, the video should start with just the background. Without the painter in it, since he could hide brighter spots in the background, which would then (incorrectly) be recognized as light source. This is the background image I will refer to later again. It is remembered by the effect.
The background should not be too bright. The darker it is, the easier it is to detect the light source (which can be anything that is bright) and the better light colours you will get. But too dark is not good either; Painting on black will most of the time look boring.
Now comes the difficult part — at least for the painter. You need to remember the picture you painted in your brain, in order to get good shapes and connected trails. When painting, the picture ''you'' see will always just look like this:
i.e. you can only see where the light source is at this very moment.
One thing to pay attention at is that you don’t draw too quickly. When shooting at 25 fps, each frame will ideally fill exactly 1/25 of a second. Unfortunately the camera usually isn’t fast enough, so one frame might actually be recorded during only 1/50 part of a second, followed by a gap of another 1/50 of a second, where nothing is recorded. That means that moving a small light source with too high speed will create no straight line but a dashed line in the final video. Just try it out, and you will understand the problem :)
(If you can manually adjust the shutter speed, set it to a value as low as possible.)
After applying the Light Graffiti effect, the video might look like that:
In the background this works by storing a Light Mask which stores every pixel that might be a light source. This light mask is then painted over the video. The principle is somewhat related to an overhead projector: You paint on a transparent layer.
Detecting the light source mainly works via two parameters: The minimum brightness of the light source, and the minimum brightness increase relative to the background. Relative means that if the brightness of a pixel changes to 60 %, then the effect does not recognize it as a light source if the background was at 55 % brightness already, but it does recognize it if the background was at, say, 10 % only. (5 % difference in the first case vs. 50 % difference in the second case.)
An additional alpha map remembers how often a pixel has been visited already, like a counter; Repeatedly painting on the same spot will eventually make the mask white even if you were painting with a red or green light source.
The light mask would look like this for our example:
Now we have three components: The original video, the light mask, and the background image. This opens the door to further possibilities (the Light Graffiti effect can do this as well):
- Painting the light mask only, with a transparent background
- Painting the light mask over the background image instead of the actual video, resulting in the person painting in the video not being visible at all
- Making the light faint away after some time, such that light sources leave a light trail behind them
Can you spot these features?
How to use the Light Graffiti effect
First of all some bad news. You most likely cannot use this effect yet. I’ve added it to the frei0r repository yesterday, so if you are not compiling frei0r and kdenlive yourself, you will have to wait for the next release. (Good news is that this should not take too long anymore.)
This being said, let’s visit the effect interface as it looks today, to see how to set the parameters correctly.
(You will notice the new (i) icon which Till added for this release – it will provide you exhaustive explanations of all important parameters.)
The first step is to navigate to the beginning of the clip you applied the Light Graffiti effect on and to activate and again deactivate the «Reset» checkbox. This makes sure that the effect really uses the first frame — or the one you are seeing right now in the render monitor — as background image, and not something between.
Second step: Catch the light source. Usually this is possible with the Brightness Threshold and the Difference Threshold only. To get them right, check the Show brightness statistics box first, and move the timeline cursor until your light source appears in the image. This might look like that:
You will notice that the image is grayscale, except for some parts which are blue. Everything that is not blue will not be recognized as light source. So the Brightness Threshold needs to be adjusted until the whole light source is blue.
Now other parts of the image are blue as well, but that is where the second parameter jumps in. Deactivate the stats checkbox and activate the one for the difference statistics. Now some parts will be green:
Adjust the Difference Threshold until the light source is completely green.
The good thing is that here some other patches are green as well — but they do not overlap with the blue patches from before. And only patches which are both green and blue will be fetched in the light map.
That’s already it! You saw the result of this in the above video.
I should tell you now that not all videos are equally suitable for this effect. The darker the environment is, the better — since then the light source itself does not need to be extremely bright (which usually means plain white, which is a little boring. Color makes it look interesting!). But in many cases it works amazingly well.
There is a bunch of other parameters you can adjust:
- Sensitivity — The higher you set it, the brighter lights will be.
- Dimming — Makes lights in the light mask fainter after some time.
- Background weight — Draws the background from the beginning of the video, such that the painter is invisible.
- Transparent background — Makes everything except for the light mask transparent. See the last seconds of the video above.
- Nonlinear dimming — Changes the way dimming is accomplished. May look more natural.
Usually I try to provide interesting links in my articles. For this Light Graffiti effect this is a little bit difficult — as far as I know there is no other Light Graffiti effect available yet. But the techniques and the light sources used for shooting a Light Graffiti video are the same as for ordinary Light Painting images. And there one can find valuable information on the web.
One of the best collections of light painting articles I’ve found so far is at DIYPhotography: Light Painting. Fireworks, cold cathodes, LEDs, and so on.
Some other links not mainly related to light sources:
- Looking for ideas? Check out 25 Spectacular Light Painting Images and Flickr: Light Painting images.
- Finally there are other possibilities to use light painting in video: Making Future Magic: light painting with the iPad and Light Warfare.
Writing this effect involved hours of testing, adjusting, and thinking. But that is over now. (Mostly — there are always details that can still be improved.) So now it is your turn to make some cool videos! (And share them here :)) Remember, when shooting:
- The environment must be darker than the light source. This does not necessarily mean that it should be black. If you can live without lots of colors, then the background/environment can actually be quite bright. Rule of thumb: None — try yourself what works for you and what does not :)
- Starting the shot with a plain background and afterwards walking into the video to start painting makes life much easier. On the one hand to determine which is background and which is the light source, on the other hand if you want to keep the background and make the painter invisible.
- If you can manually adjust it, shoot with the lowest possible shutter speed. Because the shorter it is, the bigger the gaps (which you certainly noticed in the fireworks part in my video) of a fast moving light source will become.
- Light Graffiti looks very easy. LOOKS! :D
That’s it for today — thanks for reading!
For questions, your own light graffiti videos, hints on something that is unclear in this article, and so on, please drop a comment.
And to read my other articles (I’m pleased if you do so) click here :)
Simon A. Eugster (Granjow)