0002232: high memory and CPU usage

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0002232

Kdenlive

File Loading

public

2011-07-12 03:56

2011-07-14 23:14

Reporter

igloo

Assigned To

j-b-m

Priority

normal

Severity

major

Reproducibility

always

Status

assigned

Resolution

open

Platform

amd64

Debian Linux

OS Version

unstable

Product Version

Recent git

Target Version

Fixed in Version

Summary

0002232: high memory and CPU usage

Description

Memory usage and time were always high for project loading, but having
just upgraded various things and started using the build script from
    http://www.mltframework.org/twiki/bin/view/MLT/BuildScripts#Kdenlive [^]
they're now even worse (I changed "-DCMAKE_BUILD_TYPE=debugfull" to
"-DCMAKE_BUILD_TYPE=full" in the hope of improving things; I'm not sure
if that made any difference or not). Loading one of my projects now takes:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25703 ian 20 0 5787m 3.0g 13m S 0 82.7 2:54.19 kdenlive

I've set severity to "major" as this level of memory usage is already inconvenient, and if it gets much high then it's going to cause me some headaches. The high startup time is also bad for my productivity. I'm not
really sure what to do with "priority", so I left it as default.

Anyway, I thought I'd do some measuring.

m100.kdenlive is a kdenlive file with 100 images in the clip list
(which actually all happen to be copies of the same image); there
is nothing in the timeline. m1, m200, m300, and m400 are similar,
but with 1, 200, 300 and 400 copies.

mt100.kdenlive is the same, except each clip appears once in the
timeline. Likewise, mt200, mt300 and mt400.

Here is what top has to say about each of them, after just running
"kdenlive foo.kdenlive" and waiting for kdenlive to stop using CPU:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

27940 ian 20 0 803m 62m 29m S 0 1.7 0:00.69 kdenlive
28335 ian 20 0 794m 116m 32m S 0 3.1 0:01.46 kdenlive m1.kdenlive
27976 ian 20 0 1391m 462m 32m S 0 12.4 0:59.38 kdenlive m100.kdenlive
28006 ian 20 0 1612m 491m 32m S 0 13.2 1:57.31 kdenlive m200.kdenlive
28017 ian 20 0 1822m 517m 31m S 0 13.9 2:55.66 kdenlive m300.kdenlive
28064 ian 20 0 2048m 549m 32m S 0 14.8 3:51.25 kdenlive m400.kdenlive
28089 ian 20 0 1327m 449m 32m S 0 12.1 1:00.51 kdenlive mt100.kdenlive
28101 ian 20 0 1612m 481m 32m S 0 13.0 1:58.59 kdenlive mt200.kdenlive
28161 ian 20 0 1773m 508m 32m S 0 13.7 2:56.07 kdenlive mt300.kdenlive
28231 ian 20 0 1997m 534m 31m S 0 14.4 3:53.33 kdenlive mt400.kdenlive

So initial conclusions:

* Being in the timeline doesn't seem to make much difference.

* kdenlive can process about 2 images per CPU second. What's it
  doing? Is it necessary? Could it be done in the background?
  (The file in question is an 8.4M 4288x2848 photo).

  For comparison, timing catting 100, 200, 300 and 400 copies to
  /dev/null takes:
      100: 0.00s user 0.30s system 34% cpu 0.893 total
      200: 0.00s user 0.68s system 4% cpu 15.518 total
      300: 0.01s user 1.06s system 3% cpu 30.428 total
      400: 0.02s user 1.42s system 3% cpu 37.234 total
  (the disk is a USB external drive, so IO takes a while)
  and md5summing the copies takes:
      100: 2.13s user 0.50s system 18% cpu 14.325 total
      200: 4.30s user 1.01s system 29% cpu 17.890 total
      300: 6.43s user 1.58s system 49% cpu 16.342 total
      400: 8.38s user 2.20s system 23% cpu 44.326 total

* I'm not too sure what to make of the memory usage yet.

I can put my test files online if that would be useful.

$ kdenlive --version
Qt: 4.7.3
KDE Development Platform: 4.6.4 (4.6.4)
Kdenlive: 0.8.1 (rev. 5788)

Tags

No tags attached.

Build/Install Method

Manual build from SVN

Attached Files

hickstead.kdenlive [^] (333,427 bytes) 2011-07-14 23:14

Relationships

Notes
(0007081) igloo (reporter) 2011-07-12 18:21	Tried to make audio thumbnails for the above project, and I had to kill kdenlive as it was using too much memory: 412 ian 20 0 8135m 2.9g 10m S 1 81.3 3:33.25 kdenlive And I have an even bigger project still to do!

(0007082) j-b-m (administrator) 2011-07-12 21:17	Currently having a look at this issue and made a few tests with a project containing 45 large jpeg images. When the images are not used in timeline (only in project tree), I get the following memory usage, depending on the default image producer: pixbuf: 445 M qimage: 585 M avformat: 234 M When the images are added to the timeline, loading the project shows the following memory usage: pixbuf: 225 M qimage: 225 M avformat: 270 M So it looks like when the images are not in timeline, the producer is taking more memory, or the cached data is not managed the same way... Not sure yet where is the problem, but there is definitely room for improvement, will try to understand it...

(0007085) igloo (reporter) 2011-07-14 23:14	I don't know if this hints at (part of) the problem, but if I strace loading my real project and then look at how many times each file is opened: strace -f -o out kdenlive hickstead.kdenlive grep open out \| sed -e 's/^[0-9]* open("//' -e 's/".//' \| sed 's#//#/#g' \| sort \| uniq -c \| sort -n Many are opened only once: 1 /home/ian/ext1/igloo/unsorted_raw/saturday-our-cam-2x/PRIVATE/AVCHD/BDMV/STREAM/00012.MTS 1 /home/ian/ext1/photos/Hayley/IMG_2050.JPG 1 /home/ian/ext1/wedding/6_hickstead/kdenlive/thumbs/0483ae14c108895aeb92dbdb2abd34a3.png 1 /home/ian/ext1/wedding/6_hickstead/kdenlive/thumbs/0483ae14c108895aeb92dbdb2abd34a3_0.png many more opened twice (all image proxies): 2 /home/ian/ext1/wedding/6_hickstead/kdenlive/proxy/0ab8ac1231505aa68f0e9f8ecda7101c.png many three times (images and video proxies): 3 /home/ian/ext1/photos/Judy/P1000372.JPG 3 /home/ian/ext1/wedding/6_hickstead/kdenlive/proxy/4064696aaecbfa1747f5b1c1ffaa1cd3.ts 3 four times: 4 /home/ian/ext1/paul/2011-05 Wedding - 1/tony/MVI_1759.AVI 4 /home/ian/ext1/paul/2011-05 Wedding - 1/tony/MVI_1771.AVI 4 /home/ian/ext1/wedding/6_hickstead/kdenlive/proxy/e66a3449f9b3cea259430a5b20e28aa6.png 1 each of six, nine and twelve times times: 6 /home/ian/ext1/wedding/6_hickstead/kdenlive/proxy/3f2a78f914804483322f38c7306a7ced.ts 9 /home/ian/ext1/wedding/6_hickstead/kdenlive/proxy/f3e9a69faf3109347936aef56c4e659a.ts 12 /home/ian/ext1/wedding/6_hickstead/kdenlive/proxy/c9cd61324aea753ef87be35c3b372cc3.ts I don't have any of the "thumbnail" options on. The number of times something is loaded doesn't seem to be something obvious like the number of times it appears in the timeline. For example, all the photos should be in just once. I'm a little confused as to why clips have the number of producers tags they do in the .kdenlive file, too. It doesn't look like it's related to the number of times they're in the timeline. I'll attach the project file in case it is of any help.

Issue History
Date Modified	Username	Field	Change
2011-07-12 03:56	igloo	New Issue
2011-07-12 18:21	igloo	Note Added: 0007081
2011-07-12 21:17	j-b-m	Note Added: 0007082
2011-07-12 21:17	j-b-m	Assigned To	=> j-b-m
2011-07-12 21:17	j-b-m	Status	new => feedback
2011-07-14 23:14	igloo	Note Added: 0007085
2011-07-14 23:14	igloo	Status	feedback => assigned
2011-07-14 23:14	igloo	File Added: hickstead.kdenlive