Kdenlive   bug tracker Home page

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002867KdenliveFile Loadingpublic2012-12-10 18:062012-12-10 23:35
Reporterkeremhd 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionopen 
Platform64 bitOSUbuntuOS Version12.04
Product VersionRecent git 
Target VersionFixed in Version 
Summary0002867: Very high memory usage on startup with empty timeline and lots of clips
DescriptionI can reproduce this behavior on 64 bit system. On 32bit ubuntu 12.04, I arrive at a crash (due to address space limitations I guess).


I have an empty timeline with only imported clips in the project. Project is created from scratch. Every clip has a proxy.

1100 1920x1080 60p clips (28mbit/s, around 10 hours, 110gb)
1200 1280x960 MP4 clips (GoPro recording, 15-20mbit/s, around 160gb)

I don't use audio thumbnails. For all files, there are proxy and thumbnail PNG files.

ByPass Codec Verification setting is enabled.

Timeline is empty, on an empty project: only the clips are loaded, proxies&thumbnails are created, project is saved, kdenlive is restarted.


The issue:
Opening the project file on an 64-bit computer takes around 0000010:0000015 minutes (6 virtual cores, 6gb ram, virtualized on a 8-core 8gb ram machine). After loading window disappears, and clips are added to the display, the clips are grayed out for a long time. Memory usage increases from %5 to %87 during this period (around 5-6 minutes). Using "lsof", I can see it is enumerating all the clips. After it is finished, clips are enabled, but memory usage stays the same.

On a 32-bit Ubuntu 12.04, when examining memory with "top", I see that it crashes after 2200mb of "virt" memory (and 1500mb of "res" memory) is used, I guess due to 32-bit address space.


I guess that, instead of using the PNG files for clip thumbnails, it tries to load each clip through MLT. The following is the KDE crash report for 32-bit version:


Thread 2 (Thread 0xac223b40 (LWP 13563)):
[KCrash Handler]
0000007 0xb551e580 in ?? () from /lib/i386-linux-gnu/libc.so.6
0000008 0xb5cdc5d5 in mlt_frame_clone (self=0x807ae170, is_deep=1) at mlt_frame.c:1049
0000009 0xb5cf657a in mlt_cache_put_frame (cache=0x3e34fae8, frame=0x807ae170) at mlt_cache.c:563
0000010 0xaf7471b2 in producer_get_image (frame=0x807ae170, buffer=0xac222a7c, format=0xac222b34, width=0xac2227d0, height=0xac2227d4, writable=0) at producer_avformat.c:1757
0000011 0xb5cdaff8 in mlt_frame_get_image (self=0x807ae170, buffer=0xac222a7c, format=0xac222b34, width=0xac2227d0, height=0xac2227d4, writable=0) at mlt_frame.c:475
0000012 0xae315484 in filter_get_image (frame=0x807ae170, image=0xac222a7c, format=0xac222b34, width=0xac2227d0, height=0xac2227d4, writable=0) at filter_crop.c:77
0000013 0xb5cdaff8 in mlt_frame_get_image (self=0x807ae170, buffer=0xac222a7c, format=0xac222b34, width=0xac2227d0, height=0xac2227d4, writable=0) at mlt_frame.c:475
0000014 0xae305c0c in filter_get_image (frame=0x807ae170, image=0xac222a7c, format=0xac222b34, width=0xac2227d0, height=0xac2227d4, writable=0) at filter_deinterlace.c:246
0000015 0xb5cdaff8 in mlt_frame_get_image (self=0x807ae170, buffer=0xac222a7c, format=0xac222b34, width=0xac2227d0, height=0xac2227d4, writable=0) at mlt_frame.c:475
0000016 0xae31caeb in filter_get_image (frame=0x807ae170, image=0xac222a7c, format=0xac222b34, width=0xac222998, height=0xac22299c, writable=0) at filter_rescale.c:217
0000017 0xb5cdaff8 in mlt_frame_get_image (self=0x807ae170, buffer=0xac222a7c, format=0xac222b34, width=0xac222998, height=0xac22299c, writable=0) at mlt_frame.c:475
0000018 0xae317397 in get_image (frame=0x807ae170, image=0xac222a7c, format=0xac222b34, width=0xac222998, height=0xac22299c, writable=0) at filter_fieldorder.c:34
0000019 0xb5cdaff8 in mlt_frame_get_image (self=0x807ae170, buffer=0xac222a7c, format=0xac222b34, width=0xac222998, height=0xac22299c, writable=0) at mlt_frame.c:475
0000020 0xae31d5d8 in filter_get_image (frame=0x807ae170, image=0xac222a7c, format=0xac222b34, width=0xac222b2c, height=0xac222b30, writable=0) at filter_resize.c:262
0000021 0xb5cdaff8 in mlt_frame_get_image (self=0x807ae170, buffer=0xac222a7c, format=0xac222b34, width=0xac222b2c, height=0xac222b30, writable=0) at mlt_frame.c:475
0000022 0xb5cc3c45 in Mlt::Frame::get_image(mlt_image_format&, int&, int&, int) () from /home/kerem/kdenlive/20121210/./lib/libmlt++.so.3
0000023 0x082b46c6 in KThumb::getFrame (frame=0xb84b7d70, frameWidth=64, displayWidth=64, height=36) at /home/kerem/kdenlive/src/kdenlive/src/kthumb.cpp:219
0000024 0x083180c0 in Render::processFileProperties (this=0xaa61380) at /home/kerem/kdenlive/src/kdenlive/src/renderer.cpp:813
0000025 0x08335783 in QtConcurrent::VoidStoredMemberFunctionPointerCall0<void, Render>::runFunctor (this=0xc526c50) at /usr/include/qt4/QtCore/qtconcurrentstoredfunctioncall.h:209
0000026 0x0810e672 in QtConcurrent::RunFunctionTask<void>::run (this=0xc526c50) at /usr/include/qt4/QtCore/qtconcurrentrunbase.h:134
0000027 0xb5d5b39b in ?? () from /usr/lib/i386-linux-gnu/libQtCore.so.4
0000028 0xb5d68de0 in ?? () from /usr/lib/i386-linux-gnu/libQtCore.so.4
0000029 0xb5ca0d4c in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
0000030 0xb54d7d3e in clone () from /lib/i386-linux-gnu/libc.so.6
Steps To ReproduceCreate an empty project.
Add lots of clips.
Wait for proxy/thumb generation (maybe not needed?)
Save the project, after restarting kdenlive, and loading the project, you can trace 2-3 passes over the project files with "lsof -p <pid>". On the last pass, the memory increases with each clip.
TagsNo tags attached.
Build/Install MethodBuild Script
Attached Files

- Relationships

-  Notes
(0008716)
keremhd (reporter)
2012-12-10 18:48
edited on: 2012-12-10 18:51

may be related: I also have around 1500 photos on the project. They don't have proxies (creating proxy creates the same dimensioned image with size 4-5mb). Maybe it is these pictures taking RAM space on load?

And if it would mean anything, on 64-bit machine after loading is complete, virtual memory is 10.1gb (includes memory mapped files?) and resident memory is 4.8gb

(0008720)
vpinon (administrator)
2012-12-10 23:01
edited on: 2012-12-10 23:01

I also noticed on a big project for few month (about 300 HD source clips) that only opening the project takes several GB RAM.

Seeing this report I just learned to launch "valgrind --tool=massif [path]/kdenlive [project]" : it tells me 80% of the RAM is eaten by "mlt_pool_alloc" itself called 4 times by "QtConcurrent::RunFunctionTask<void>::run()"
-> "Render::processFileProperties()"
-> "KThumb::getFrame"
-> "Mlt::Frame::get_image"

So maybe there is MLT resource to close/free after multithreaded operation (I didn't enable the multithread option in kdenlive preference, I guess it is in the clip manager loading clips where parallel operation is enabled).

I'm not enough at ease with MLT code to fix it kickly, if a guru around finds the fix, I would love to test!

(0008726)
keremhd (reporter)
2012-12-10 23:35

@vpinon the trace you provided coincides with the crash stack trace on 32bit. There are lots of calls, but it passes through deinterlace, scale, etc. filters on MLT, and at the top there's the KThumb::getFrame call:
 0x082b46c6 in KThumb::getFrame (frame=0xb84b7d70, frameWidth=64, displayWidth=64, height=36)

With dimensions 64x64 (or 64x36? I'm not sure). So as I understand, the reason it goes through MLT is to fetch the icons on the file list


Why not use the png thumbnails instead (at Render::processFileProperties() I guess, but I didn't check what it does), the thumbnails are already created in the thumbnail directory. Then only the viewed/rendered clips would cause this behavior (this might be a temporary solution, but it would reduce the problem to a memleak, much easier to manage for users)

kerem@pulbiber:/media/D_DRIVE/kdenlive/thumbs$ identify ffefb20ab2214f907246c7373b1e7512.png
ffefb20ab2214f907246c7373b1e7512.png PNG 64x36 64x36+0+0 8-bit DirectClass 2.97KB 0.000u 0:00.010

It should also reduce initial load times for large projects

- Issue History
Date Modified Username Field Change
2012-12-10 18:06 keremhd New Issue
2012-12-10 18:48 keremhd Note Added: 0008716
2012-12-10 18:48 keremhd Note Edited: 0008716 View Revisions
2012-12-10 18:51 keremhd Note Edited: 0008716 View Revisions
2012-12-10 23:01 vpinon Note Added: 0008720
2012-12-10 23:01 vpinon Note Edited: 0008720 View Revisions
2012-12-10 23:35 keremhd Note Added: 0008726


Copyright © 2000 - 2014 MantisBT Team
Powered by Mantis Bugtracker