Working with many GB of data in one project?

Hi,

I'm not sure if this is the place to ask for such tips and tricks, but...

How do other users handle projects with lots of HD clips? Like for example all the material from one holiday, maybe 5 hours of HD raw material that you want to cut down to one hour? If I load 4 GB in 228 file into kdenlive at once, my system stays usable. Not bad, so far. I can use folders for the clips inside kdenlive so that I don't get too confused. The timeline, on the other hand, can't be segmented or structured, like in chapters for example. Daydream: "Minimize this chapter" in the timeline, or even "unload from RAM" without deleting it from the project. Speaking about RAM: I'm not sure how much kdenlive can load on my less than average hardware. I tried 6,6 GB in 465 files. No fun, kdenlive needs half an hour to load them. 9 GB of data in swap space. Playback (project or clip monitor) not in realtime anymore. Everything I try to do is very slow. I guess I won't try that again.

Any suggestions?

TIA

Chamo

What version of Kdenlive do you have?

Waynevid

Turf off making of thumbnails waveforms of audio if you don't need them. That takes a lot of time, especially with AVCHD.

Hi,

I'm using kdenlive 0.7.7 with mlt 0.5.4. Video output is via VDPAU in mlt.
Thanks for the hint, KoRnholio8. I tried it out and it seems to help a little. The 6,6 GB in 465 files now loaded in 10 minutes, and responsiveness is better. The playback in the project or clip monitor is still not smooth, though.
I think the limit is somewhere with the loading of the clips into the project tree and with creating the thumbnails there.
Hm, what does "Maximum number of clients reached" on the command line mean?

I guess I'll just go on with "only" 2,7 GB in 141 files. It's not that bad. ;-)

Chamo

I'm sorry if I go off topic, how do you get output via VDPAU in MLT? Does it means that you have playback/monitor with GPU acceleration in Kdenlive? How can I test it for myself?
I'm running Ubuntu Lucid 64-bit, Kdenlive 0.7.7, I have an NVIDIA 8500GT, default ffmpeg with VDPAU enabled and SMPLAYER using VDPAU correctly leaving CPU usage around 5% even with 1920x1080 videos. AMD Athlon 64 X2 6000+.
I never needed extra power in Kdenlive working with SD miniDV but now I'm waiting for a Canon HV10 (tape HDV) I got from ebay and I already know It will be harder when I'll start adding Frei0r plugins ecc. ...
Thank you,
Ignazio

I just finished working on a project with AVCHD files decoded to DHxHD with total size of the clips at about 50gb. I tried setting up VDPAU, but failed. The project however did load in approximately 10-15 seconds from a USB 2 external drive. I'd say your main problems would be AVCHD, VDPAU and the sheer number of small files.

Hi,

so the size of the project (the Gigabytes of the clip material) is not an issue in kdenlive. The number of AVCHD clips is. That seems to be related to the "Maximum number of clients reached" message. But what clients, and how is this number defined? Could I increase this number?

About VDPAU support and how to get it: Please see the other thread. That's about all I know about this.
http://www.kdenlive.org/forum/recommended-way-working-avchd-content#comm...

Chamo

Thank you. I 've read the other thread you linked. That's probably why you have such great performances with AVCHD on a single core machine. I use Ubuntu and installed everything through the package manager. I have 8500GT and an Athlon dual core 3GHz, so, maybe, if I was able to compile everything by myself I could reach even better results... I don't know how to do, I'm frightened to damage my system because I'm a (3 years...) newbie...
Anyway, that's really amazing!
Ignazio

to see if you have vdpau working, in a terminal run:
$ melt some.mts -debug 2>&1 | grep vdpau

and see if you get a bunch of vdpau_get_buffer and vdpau_release_buffer messages.

There are still major performance problems with seeking on AVCHD. Also, playback is not low cpu as mplayer since Kdenlive must pull the uncompressed images from video memory back into main memory for additional processing. mplayer can just keep them in video memory and tell vdpau when to show them.

Thanks for the tip, I tested it!
The terminal gave no messages at all. The video opened up in a window and played but no message in the terminal :-(

I uninstalled Kdenlive from the repos and I followed the instruction on this site and compiled MLT and Kdenlive by myself and ffmpeg with this script: http://www.webupd8.org/2010/04/script-to-automatically-download-and.html
I have everything up and running but no VDPAU in MLT (which I also read should be automatically detected)....
This is a killer feature that could help me staying with my dual core for other two years and Kdenlive is my favorite video editor!

See if it is enabled in FFmpeg. When you run 'ffmpeg -codecs' it should show a h264_vdpau. If not, then check to see if you have libvdpau-dev installed by looking for /usr/include/vdpau/vdpau.h or "locate 'vdpau/vdpau.h'"

I'm pretty sure I've installed libvdpau-dev from synaptic and I know that smplayer+vdpau is still working good but when I'll come back home (now I'm at work) I'll check FFmpeg (since I didn't compile it manually) and I'll look for vdpau.h.
Thanks, I'll let you know.
Ignazio

vdpau.h is there :-)

And that's what returned the command ffmpeg -codecs:

Hi,

the option "codecs" seems to be unknown to ffmpeg. I guess Dan meant "formats":

ffmpeg -formats | grep vdpau

should give you something like

FFmpeg version SVN-r20373, Copyright (c) 2000-2009 Fabrice Bellard, et al.
built on Feb 27 2010 22:27:14 with gcc 4.3.4
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-static --enable-shared --cc=x86_64-pc-linux-gnu-gcc --disable-debug --disable-network --enable-libmp3lame --enable-libvorbis --enable-libx264 --enable-libxvid --disable-indev=v4l --disable-indev=v4l2 --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-x11grab --disable-altivec --cpu=athlon64 --enable-gpl --enable-version3 --enable-postproc --enable-avfilter --enable-avfilter-lavf --disable-stripping --enable-hardcoded-tables
libavutil 50. 3. 0 / 50. 3. 0
libavcodec 52.37. 1 / 52.37. 1
libavformat 52.39. 2 / 52.39. 2
libavdevice 52. 2. 0 / 52. 2. 0
libavfilter 1. 4. 1 / 1. 4. 1
libswscale 0. 7. 1 / 0. 7. 1
libpostproc 51. 2. 0 / 51. 2. 0
D V D h264_vdpau H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (VDPAU acceleration)
D V DT mpeg1video_vdpau MPEG-1 video (VDPAU acceleration)
D V DT mpegvideo_vdpau MPEG-1/2 video (VDPAU acceleration)
D V D vc1_vdpau SMPTE VC-1 VDPAU
D V D wmv3_vdpau Windows Media Video 9 VDPAU

Cheers

Chamo

-formats or -codecs depends on your version of ffmpeg. New versions now only show muxers and demuxers for -formats.

I see, Dan. So Ignazios and my versions seem to be a little older, still working fine for me.

I would like to come back to my question in post #6:
What does the "Maximum number of clients reached" message mean that I get when I try to work with too many (>200) AVCHD clips? And is there anything I can do to increase that number of clients?

Cheers,

Chamo

I do not know where that error message comes from. I searched the Kdenlive code base for "clients" and "Maximum number" and neither of them yielded any results. Also, searching MLT and FFmpeg code bases for "clients" yielded nothing either.

I have this issue too.

234 files in AVCHD 1080
Total: 11GB

I was trying to join them all and got the attached after 7 hours of encoding.

kdenlive-0.7.8-1pclos2010
ffmpeg-0.6-1pclos2010
mlt-0.5.10-1pclos2010

In another thread,
http://www.kdenlive.org/forum/yet-another-crash-openening-project

I thought it might be related to VDPAU, but we disable VDPAU when rendering because only one process can access it at time and Kdenlive UI is first.

I used to get the same error "max number of clients reached". I believe the tmp file that is created must be massive. I moved my tmp and project folder over to a huge drive dedicated to video editing and have not had those errors again. I would have about 10 gig free on the other drive and it would still fail with the above error. I was also working with transcoded avchd files.