Kdenlive   bug tracker Home page

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000575KdenliveFile Loadingpublic2009-01-09 22:582009-06-18 01:51
Reportersatupe 
Assigned Toddennedy 
PrioritynormalSeveritycrashReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version0.7.1 
Target VersionFixed in Version0.7.3 
Summary0000575: Memory Issues in large projects
DescriptionIn past versions, I have been able to import large numbers of pictures (.jpg) to create slide shows. In 0.7.1, I run out of memory after after about 3 gig of pictures are imported (on my system),KDENLIVE crashes. For whatever reason, memory is never swapped. I can work around the problem by either using ImageMagic to shrink the jpg files or importing fewer pictures. However, this would be a big issue if I didn't have 5 gig of physical memory - or - if I were to import large dv files:)

I am unable to connect to my Canon camcorder via IEEE firewire. I can capture using Kino, so I know the issue is not the dreaded "Ubuntu Group Setings". I am a relative novice concerning DVGRAB (and Ubuntu!) so this may be a setup issue.
Additional InformationSystem: BE2300 AMD Processor
Memory: 5 gig physical, 4.7 gig swap
Disk space: 1400 gig
Linux Dist. Ubuntu 8.04 Hardy
TagsNo tags attached.
Build/Install MethodBuild Wizard
Attached Files

- Relationships

-  Notes
(0002096)
administrator (administrator)
2009-01-10 17:26

Same here, it seems like slideshows consume an increasing amount of memory with time. That might be a bug in MLT, will investigate the issue later.
(0002097)
madsdyd (administrator)
2009-01-10 23:47

Acknowleding as jb can reproduce.

Might be good to know if this is a 32 bit or 64 bit system? My guess is 32 bit.

Also, regarding previous versions; is that 0.7.0 or earlier than that?
(0002117)
satupe (reporter)
2009-01-13 22:32

Good guess - it's a 32 bit system with an AMD BE2300 dual core 64 bit processor. I've had issues finding 64 bit video software in the past:)

As far as the earlier version, I can't remember. I updated from Ubuntu 7.04 to Hardy 8.04, Intrepid 8.10, and then back to Hardy 8.04. The version that imported .jpeg files was the version that came from the 3rd party software channel in Gutsy 7.04, which is LONG GONE. I only work with family videos in the winter so I am not much help here. I do remember that there were "sound issues" for NTSC rendering (cracking and popping) with .dv videos. Also, you had to import .ogg files for audio to work correctly for slide show creation - if that helps. Your team has fixed ALL of the sound issues in this version:)

I also found a couple of minor bugs. Rendering using the setting Raw DV - DV NTSC 4:3 produces a .dv file of 0 size. I did not test DV NTSC 16:9. The DVCPRO25 and DVCPRO15 4:3 works, which is what really counts for me.

Your team rocks! I am using kdenlive to render raw dv and wma files as well as jpeg pictures to import into qdvdauthor. Thank you for your efforts.
(0002158)
satupe (reporter)
2009-01-16 23:39

I found the old version of KDENLIVE that loads large numbers of pictures. I had screwed up the configuration with the old version somehow and had 5 kdenlive windows floating around, but I finally was able to import a ton of pictures with no memory errors. The new version does import faster, at least until it runs out of memory. Here's the old version:

KDENLIVE 0.5 SVN20071228-0

It calls libmlt0.2.5 (at least in Ubuntu)

Just for kicks, I copied the 0.2.5 MLT folder from /usr/bin/share to my home directory where the new version of KDENLIVE was installed. I renamed the "new" mlt and pasted in the 0.2.5 version. Still ran out of memory:(
(0002177)
administrator (administrator)
2009-01-22 00:09

I made some tests and found a strange memory problem related to MLT's pixbuf producer. Could you try to open the file fezzik.dict that is located in $INSTALL/share/mlt

You should see some lines with:

*.jpg=pixbuf,qimage,sdl_image
*.jpeg=pixbuf,qimage,sdl_image

Try to remove the reference to pixbuf:

*.jpg=qimage,sdl_image
*.jpeg=qimage,sdl_image

Of course if you import png images, edit the *.png line

After that, start Kdenlive and let me know if it changes something in the memory usage...
(0002181)
zappa (reporter)
2009-01-22 15:35

Hi,
I have the same problem. During importing a single jpeg file in the project, kdenlive crashes. I applied the solution (revisited) described before and now kdenlives works fine.

I edited the file "/usr/lib/mlt/fezzik.dict" removing the word "pixbuf" as described before in the previous note.

Grazie! ;)
(0002182)
satupe (reporter)
2009-01-22 16:53
edited on: 2009-01-22 16:57

First, I renamed fezzik.dict to verify I was dinking with the right file. I opened KDENLIVE and it would not import clips - good sign. It's fun to be on the testing side rather than the developing side!!!

Test Procedure:
a) baseline memory usage on my system is runs approx. 320 MB.
b) I closed KDENLIVE, modified fezzik.dict, and reopened KDENLIVE for each test.
c) All jpg files in test were converted to 1600x1200 pixels (approx. 600 kb/picture). I am able to import a much smaller portion of the original 6 megapixel pictures.
d) I selected 540 jpg files to import in tests.
e) Only *.jpg files were imported. DV, MOV, WMA, AVI, VOB all import fine.

1) I modified the line *.jpg in fezzik.dict to "*.jpg=qimage,sdl_image". KDENLIVE crashed do to memory overflow.

2) I modified the line *.jpg in fezzik.dict to "*.jpg=pixbuf". I was able to import 540 "1600x1200" jpg files (approx. 600k average) into KDENLIVE, using 2.9 gig of 3.5gig available physical memory.

3) I modified the line *.jpg in fezzik.dict to "*.jpg=qimage". KDENLIVE crashed when I attempted to load the same 540 jpg files.

4) I modified the line *.jpg in fezzik.dict to "*.jpg=sdl_image". I was able to import 540 "1600x1200" jpg files (approx. 600k average) into KDENLIVE, using 2.3 gig of 3.5 gig available physical memory. *.jpg=sdl_image is most efficient setting on my system. Still, there is a memory leak on my system.

5) I removed the lines *.jpg=... and *.jpeg=... from the fezzic,dict file. I was able to import the jpg files with the same results as step 4. Why?

6) Bewildered, I renamed fezzzic.dict file and was unable to import clips, so step 5 still baffles me unless its info is cached somehow:-)

7) I imported all 699 jpg pictures via KDENLIVE 0.5 SVN20071228-0, and memory usage remained at baseline levels, but it was SLOWWWWWWWW:) I was able to import all 699 original (before compression) pictures.

(0002218)
ddennedy (developer)
2009-01-27 07:51

This is fixed in MLT SVN r1328 without having to give qimage a higher priority (fix in gtk2 pixbuf producer). Thank you to all for your diligent testing to help isolate the problem.
(0002245)
madsdyd (administrator)
2009-01-27 17:28

Thanks dan.

Satupe could you please confirm that the recent commit in MLT works for you too?
(0002248)
satupe (reporter)
2009-01-27 18:40

Sorry Dan, I don't know what to do here. Old people, what can you say?? Could you send me some brief instructions and links? Thanks for all your work.
(0002249)
administrator (administrator)
2009-01-27 18:50

You need to update your MLT to the latest svn, then recompile & install. If you are using the Kdenlive Builder Wizard to compile, you just need to run it again, it will update MLT, MLT++ and Kdenlive to the latest svn revision, then recompile.

Then, try to see if the problem is still here with the newly build version...
(0002258)
satupe (reporter)
2009-01-27 21:11

I'm still having the same problem. I totally deleted all kdenlive directories and performed a fresh install. The kdenlive builder seemed to have the new code (at least the dates were new). I'll provide you a link to the screenshots.

kdenlive builder screenshot => http://satuperail.googlepages.com/Screenshot-KdenliveBuilderWizard.png [^]

System Monitor Screenshot => http://satuperail.googlepages.com/Screenshot-SystemMonitor-1.png [^]
(0002265)
ddennedy (developer)
2009-01-28 05:08

I can no longer reproduce this. I am testing with 1250 JPEGs of various sizes in a single directory in a slideshow clip. kdenlive never breaks 100MB resident size when this is the only clip, added to the timeline, and played back. Of course, attempting to add all of them as individual clips is a problem. Are you adding them as a slideshow clip?
JB, are you still able to reproduce the problem?
(0002268)
administrator (administrator)
2009-01-28 09:50

The problems seems solved for me too.
Satupe, are you sure you are running the Kdenlive version installed by the Wizard ? Do you start it using the kdenlive_start script ?
(0002271)
satupe (reporter)
2009-01-28 15:42
edited on: 2009-01-28 19:45

As a retired developer, I feel your frustration. As an end user, I feel frustration because I don't know what I'm doing - LOL. It's hard for me to determine which libraries I should delete and which ones I should keep in order to make a "fresh install". I'm an old mainframe guy, so I know less than nothing about Ubuntu! Here's a link to my latest kdenlive crash log. I hope it helps. => http://satuperail.googlepages.com/kdenlive_crash_log.txt [^]

Just got in from cleaning the driveway - had a foot of snow this morning in central Indiana, USA! I imported 700 jpg files successfully via "import slideshow" rather than "import clip" and selecting the 700 jpg files. The memory increased from baseline during the import but went back to baseline after the import finished. HOWEVER, kdenlive crashed when I attempted to place the slideshow onto the time line. Here are some screenshots:

Memory usage => http://satuperail.googlepages.com/slideshow_memory_usage.jpg [^]

Successful Import => http://satuperail.googlepages.com/Screenshot-Untitled-DVNTSCWidescreen.png [^]

Here's the crash report => http://satuperail.googlepages.com/kde_crash_timeline.txt [^]

(0002273)
ddennedy (developer)
2009-01-28 20:07

Satupe, in your claim above "7) I imported all 699 jpg pictures via KDENLIVE 0.5," are you saying that you were able to import 540 pictures as individual clips and not as a slideshow?

Also, FYI, as long as you run kdenlive_start from the builder wizard, you should be isolated from other MLT and ffmpeg installs.

JB needs to take a look at that crash log as it is appearing somewhere in Qt and kdenlive whereas I am responsible for MLT.
(0002276)
satupe (reporter)
2009-01-28 20:47

KDENLIVE 0.5 build....

You are correct. With the KDENLIVE 0.5 build, there was no limit to the number of individual clips (jpg files) I could import. In fact, I have imported so many individual clips with the 0.5 build, I had to cut clips out of the timeline in order to have the rendered "Slide Show" fit onto a DVD. Memory WAS NEVER an issue with the KDENLIVE 0.5 build. Rendering NTFS video was the big weakness of the 0.5.

New KDENLIVE build

The new kdenlive build renders NTFS BEAUTIFULLY (and fast). But, I'm having issues importing "jpg pictures", and getting DVGRAB to work. The DVGRAB issue is not important to me - I just use Kino (not even I can crash Kino!).

Issue 1: I cannot import large numbers of jpg files by the "import clips" option and selecting files individually (as described in previous posts).

Issue 2: I can import large numbers of jpg files by choosing the "import slideshow" option, but then I cannot move the imported slide show to the timeline (as posted in previous post).

I have imported HUGE DV files from Cinelerra and LIVES (with no memory issues) into the new kdenlive build and rendered the dv file with no issues. I have even imported WMV and WMA with no issues.

KDENLIVE is SOOOO CLOSE to being a GREAT video application. Thank you for your efforts.
(0002280)
ddennedy (developer)
2009-01-29 06:34

I just committed some changes to MLT to allow it to load a large number of pictures as individual clips as opposed to a slideshow. However, this does not work in kdenlive yet.

Basically, the problem was caching. Anytime you request an image from the picture, it is cached in memory because often the same picture is repeated over and over again - at least for a second or two. There was a recent change in MLT to validate a picture prior to accepting it, and that caused it to cache. So, just asking to MLT to load many pictures prior to actually using them caused a problem, and that is what I just fixed. Now, as each picture is accessed during playout or render, the image will be cached and not released. This is by design for a random access project such as a video editor. There is a trick in MLT when you know that you are doing sequential processing whereby you can have pictures that have already been processed closed to keep memory usage low. Even kdenlive 0.5 and MLT 0.2.5 should have used a large amount of memory to render a long video with many photos as this caching is not new. Simply, the caching prior to actual usage was the new bug that was fixed.

In short, it is now possible to theoretically make a change in kdenlive to let you import 1000 hires pictures into the Project Tree. However, it would not fix memory consumption if you add all of them to the timeline and play the entire timeline.

So, is loading many into the Project Tree but only adding a fraction of them to the timeline a typical use case we should support? Because that seems feasible.

And I suppose with even more changes I can add a parameter on the image reading modules in MLT to prevent caching altogether.
(0002281)
madsdyd (administrator)
2009-01-29 06:42

Dan, in advance, excuse my ignorance, but couldn't the cache be purged in a RLU fashion? Of course, it requires that a mechanism to retrieve the image again, should it have been purged, is implemented too.

As you say, unless there is a usecase to support it, might not be worth it.
(0002282)
ddennedy (developer)
2009-01-29 07:18

I do not know what a RLU is, and google is not my friend today. However, I conjecture that you refer to something "smart." Let's brainstorm this, but here are the constraints:
There is a separate producer object for each picture file.
Each of these objects is not aware of the others.
There is nothing periodically running in these objects when no images are being requested.
We can not introduce a thread in each object to flush itself.
Something external to the object could call a virtual function on the service to ask it to flush. That requires changing the interface. The playlist property autoclose=1 (hinted at above) is similar to this but closes the producer instead of asking it to purge.
Perhaps I could still close the producer, replace it in the playlist with a surrogate, and then when a surrogate is re-accessed, I can recreate the producer.

It is fairly complex. Certainly, a simple property that tells the image producer to not cache is easier, no? However, solving this in a "smarter" fashion would also solve the problem of project with many clips each of which has potentially many decoding threads (which I have mentioned to you in the past).
(0002285)
madsdyd (administrator)
2009-01-29 10:32

Dan, I am sure you know what a LRU cache is, but here is a short overview: http://en.wikipedia.org/wiki/Cache_algorithms [^]

As you know, I am not familiar with the internals of MLT. But, I look at this line:

"Perhaps I could still close the producer, replace it in the playlist with a surrogate, and then when a surrogate is re-accessed, I can recreate the producer."

With the needed "disclaimers": This might be where you should add the cache: A level of indirection between the entity that (using the playlist?) gets the threads/producers. Instead it speaks to the "thread/producer cache", and asks for the thread/producer. If not available, it must create it, and insert it into the cache. When inserted, the cache purges the least-recently-used thread/producer and replaces it with the new entry. Parameters on the cache could be max-number-of-entries, or, if possibly, max-storage-allocated.

LRU can be implemented in a number of ways, using timestamps or just ints. The cache might be able to be designed such that it transparently (re)creates the resource requested, becoming a sort of cached factory. It might be designed to never purge some types of producers, etc.

The solution of never caching is of course simpler. But, I reckon it sort of ruins the intention of caching images?

Hope the above makes sense

Regards

Mads
(0002286)
madsdyd (administrator)
2009-01-29 10:33

Oh, and just noticed, sorry, I mistyped in my post: LRU, *not* RLU, typo, sorry, missed that.
(0002289)
satupe (reporter)
2009-01-29 14:59

This is a dumb end user comment, but would it be possible to allow the end user to cache/not cache the import of files? Perhaps this could be done automatically by checking the amount of memory being used - or - the number of jpg files being imported? Linux has very few video applications that creates a 2 hour slideshow dvd.

Cinelerra does very well at importing jpg pictures in 4x3 format, but it can't handle mixing 4x3 and 3x4 formats. I cannot use ManSlide due to my Nvidia video card. LIVES is great for capturing video from screens, but it's terrible for editing. Kdenlive is my only option for making slide shows in Linux. I don't want to be forced back to Adobe Premiere on WINDOWS:-)
(0002341)
ddennedy (developer)
2009-02-03 10:23

I have started making a very simple Least Recently Used cache in MLT. It is coming along quite nicely and should be done in a few days.
(0002379)
ddennedy (developer)
2009-02-09 09:24

FYI, this change is working good and safe now in my local copy testing outside of kdenlive. Memory usage for inigo never goes over 500MB, but it could if I was using even higher res images - I am using mult-megapixel photos, but nothing over 5 Mpix and which are interspersed with low res pics too. Making it work safely took a fair amount of time. I just need to update some docs and do final tests in kdenlive. I hope to get this into the repository within 24 hours.
(0002393)
ddennedy (developer)
2009-02-10 07:36

This is fixed in MLT SVN r1347. Please upgrade and test when you get a chance. More than likely you need to clean the kdenlive sources after you update and before running cmake.
(0002407)
satupe (reporter)
2009-02-11 13:45
edited on: 2009-02-11 13:46

I have some good news, and some bad news. The good news is you have resolved the "import memory issues":-) I was impressed how quickly the jpeg files imported into kdenlive.

Now for the bad news - I cannot move the imported pictures to the kdenlive timeline, not even one picture. Here is a link to the log file => http://satuperail.googlepages.com/KDENLIVE_log_report_2_11_2009.txt [^]

(0002408)
ddennedy (developer)
2009-02-11 18:35

Hmm, how strange. I did not reproduce it on 2 different systems/distros. It is a problem somewhere in Qt, which I do not know so well. Is this only happening for pictures and not video?
(0002414)
satupe (reporter)
2009-02-12 02:16

I cannot move any filetypes to the timeline. Tried dv,mpeg,avi,png,bmp,and jpg. All failed - even if only one file (4k) was imported via "clip import".
(0002415)
ddennedy (developer)
2009-02-12 03:23

Your message is confusing. If you start kdenlive and load just a .dv file, can you add it to the timeline?
(0002419)
ddennedy (developer)
2009-02-13 07:11

I tested this on a third system, my x86-64 system running Arch Linux, and that works as well.
(0002421)
satupe (reporter)
2009-02-13 16:47
edited on: 2009-02-13 16:50

***************
Testing status: Sorry, I am an end user:-)
A. Importing large numbers of jpg files

KDENLIVE now allows me to import 1000 pictures without memory errors. YEAH!

B. Moving clips from the project tree to the timeline in KDENLIVE

KDENLIVE will not allow me move clips from the project tree to the timeline. Please view the log file in my last post. This is a link to the log file -> http://satuperail.googlepages.com/KDENLIVE_log_report_2_11_2009.txt [^]

C. DV Capture via Canon ZR500 camcorder

I can now import DV from my Canon ZR500 camcorder if I manually play with the camcorder. If you are the same DDennedy (Dan Dennedy) as the Kino man, steal the DV record code from Kino and put it into kdenlive - LOL.

D. Current testing procedures:

In an attempt to localize the problem, I am testing kdenlive by importing a single clip and attempting to move the single clip to the timeline.


These are the steps I have taken thus far:

1. I cleared all directories pertaining to kdenlive
2. I reinstalled kdenlive as per these directions => http://linuxcrypt.net/?p=179 [^]
3, The KDE installer completed successfully, with the updated code.
4. I started kdenlive by typing /~/kdenlive/bin/kdenlive_start
5. kdenlive started successfully, and I imported a single jpg file.
6 A single thumbnail appeared in the "Project Tree" and the picture appeared in the clip monitor.
7. kdenlive crashed when I attempted to move the clip from the project tree to the timeline. Here is the link to the console report => http://satuperail.googlepages.com/Single_jpg_import_console.txt [^]
8. I restarted kdnelive
9. I imported a single dv clip
10. A single thumbnail appeared in the "Project Tree" and the video appeared in the clip monitor. I was able to play the video in the clip monitor successfully.
11. kdenlive crashed when I attempted to move the clip from the project tree to the timeline. Here is the link to the console report => http://satuperail.googlepages.com/Single_dv_import_console.txt [^]

I have tested kdenlive as describe in the steps above for avi clips, mov clips, png clips, and wmv clips with the same results. Bottom line, kdenlive will not allow me to move a clip of any type from the project tree to the timeline.

(0002422)
ddennedy (developer)
2009-02-13 21:38

Yes, I am the same ddennedy as the Kino developer. kdenlive is using dvgrab, which already shares a lot of code with Kino.

The problem now is separate from this issue. This issue will not be marked as resolved, and I would appreciate you open a new issue for the crash on adding to the timeline. The logs you provided do not containing any debug information. When you launch kdenlive from kdenlive_start, it should prompt if you want to run in the debugger. You should choose to run in the debugger, crash it, and send a more detailed log in a new issue. Thanks for testing the resolution to this issue.
(0002423)
ddennedy (developer)
2009-02-13 21:39

MLT SVN r1347

- Issue History
Date Modified Username Field Change
2009-01-09 22:58 satupe New Issue
2009-01-09 22:58 satupe Build/Install Method => Build Wizard
2009-01-10 17:26 administrator Note Added: 0002096
2009-01-10 23:47 madsdyd Note Added: 0002097
2009-01-10 23:47 madsdyd Status new => acknowledged
2009-01-13 22:32 satupe Note Added: 0002117
2009-01-16 23:39 satupe Note Added: 0002158
2009-01-22 00:09 administrator Note Added: 0002177
2009-01-22 15:35 zappa Note Added: 0002181
2009-01-22 16:53 satupe Note Added: 0002182
2009-01-22 16:53 satupe Note Edited: 0002182
2009-01-22 16:55 satupe Note Edited: 0002182
2009-01-22 16:57 satupe Note Edited: 0002182
2009-01-27 07:51 ddennedy Note Added: 0002218
2009-01-27 17:28 madsdyd Note Added: 0002245
2009-01-27 18:40 satupe Note Added: 0002248
2009-01-27 18:50 administrator Note Added: 0002249
2009-01-27 21:11 satupe Note Added: 0002258
2009-01-28 05:08 ddennedy Note Added: 0002265
2009-01-28 09:50 administrator Note Added: 0002268
2009-01-28 15:42 satupe Note Added: 0002271
2009-01-28 19:39 ddennedy Status acknowledged => assigned
2009-01-28 19:39 ddennedy Assigned To => ddennedy
2009-01-28 19:45 satupe Note Edited: 0002271
2009-01-28 20:07 ddennedy Note Added: 0002273
2009-01-28 20:47 satupe Note Added: 0002276
2009-01-29 06:34 ddennedy Note Added: 0002280
2009-01-29 06:42 madsdyd Note Added: 0002281
2009-01-29 07:18 ddennedy Note Added: 0002282
2009-01-29 10:32 madsdyd Note Added: 0002285
2009-01-29 10:33 madsdyd Note Added: 0002286
2009-01-29 14:59 satupe Note Added: 0002289
2009-02-03 10:23 ddennedy Note Added: 0002341
2009-02-09 09:24 ddennedy Note Added: 0002379
2009-02-10 07:36 ddennedy Note Added: 0002393
2009-02-10 07:36 ddennedy Status assigned => feedback
2009-02-11 13:45 satupe Note Added: 0002407
2009-02-11 13:46 satupe Note Edited: 0002407
2009-02-11 18:35 ddennedy Note Added: 0002408
2009-02-12 02:16 satupe Note Added: 0002414
2009-02-12 03:23 ddennedy Note Added: 0002415
2009-02-13 07:11 ddennedy Note Added: 0002419
2009-02-13 16:47 satupe Note Added: 0002421
2009-02-13 16:50 satupe Note Edited: 0002421
2009-02-13 21:38 ddennedy Note Added: 0002422
2009-02-13 21:39 ddennedy Note Added: 0002423
2009-02-13 21:39 ddennedy Status feedback => resolved
2009-02-13 21:39 ddennedy Fixed in Version => Recent git
2009-02-13 21:39 ddennedy Resolution open => fixed
2009-04-15 13:07 madsdyd Fixed in Version Recent git => 0.7.3
2009-06-18 01:51 xzhayon Status resolved => closed


Copyright © 2000 - 2014 MantisBT Team
Powered by Mantis Bugtracker