0000607: Title clip files (.kdenlivetitle) are stored in ASCII and not UTF-8

(0002190)
Granjow (developer)
2009-01-24 19:27

$ file AC_moves.kdenlivetitle
AC_moves.kdenlivetitle: ASCII text

(0002194)
Granjow (developer)
2009-01-25 14:36

Yeh :) I guess it works.

(0002234)
madsdyd (administrator)
2009-01-27 13:02

Acknowledged.

And, the fix in svn rev. 2950 fixes this.

It also breaks all old .kdenlivetitles that contains 8 bit chars, but I guess we have to live with that.

I am not resolving this issue though, as I want your comment on the fact that title clips are still saved in ascii in the .kdenlive files?

At least from my inspections. Isn't that a problem too? Is it related to this issue or something else?

Regards

Mads

(0002242)
Granjow (developer)
2009-01-27 14:20

Are they? I took a look at my .kdenlive file here, and the unicode characters are displayed correctly.

(0002244)
madsdyd (administrator)
2009-01-27 17:27

Please test again - I have tried it twice now. When saving the individual clip, it works as expected, when saving to .kdenlive, it saves like this:

HEJ <E6><F8> <E5><C6><D8><C5>

and, it should say "HEJ æøåÆØÅ" - the latter should be multibyte in utf8.

I guess you agree it should be consistent with the clip saving?

(0002251)
Granjow (developer)
2009-01-27 20:02

I've uploaded a test file where I've put your characters and some other unicode characters (?, ?, ?, ?). They are still displayed as they were after reloading, and when searching for one of the characters in the .kdenlive file, I find them.

Could you upload a file from your machine?

(0002252)
Granjow (developer)
2009-01-27 20:04

Oh crap, mantis doesn't seem to understand unicode, lol. It were the unicode characters 1234, 2345, 5432, 4321. See here http://decodeunicode.org/u+1234 [^] and so on.

PS: The file has been saved in unicode nevertheless, I've checked that.

(0002259)
madsdyd (administrator)
2009-01-27 22:42

Hmm. Inspecting the attachment, it *does* appear that your saved file contains utf8 in the xmldata part of the kdenlive_producer for the clip.

Mine does not. Sigh.

..

Found the reason. Files are saved using QTextStream, which uses QTextCodec::codecForLocale(), which on my system fails to supply an UTF-8 locale, but defaults to Latin1. (See also 0000308).

Try this patch:

Index: src/kdenlivedoc.cpp
===================================================================
--- src/kdenlivedoc.cpp (revision 2990)
+++ src/kdenlivedoc.cpp (working copy)
@@ -19,6 +19,7 @@

#include <QCryptographicHash>
#include <QFile>
+#include <QTextCodec>

#include <KDebug>
#include <KStandardDirs>
@@ -925,7 +926,9 @@
         KMessageBox::error(kapp->activeWindow(), i18n("Cannot write to file %1", path));
         return false;
     }
+ kDebug() << "For QTextStream, default locale is: " << QTextCodec::codecForLocale()->name();
     QTextStream out(&file);
+ out.setCodec( QTextCodec::codecForName( "UTF-8" ) );
     out << sceneList.toString();
     file.close();
     return true;

This makes the chars become utf8 on my system.

I expect that your default locale is utf-8 not?

Now, as we claim in the kdenlive files preprocessing directive that it is utf8, we really should not rely on the users current locale to encode/decode the kdenlive files. So, I sort of think we should apply this patch.

What do you think?

Oh, and AFAICT, reading the files rely on QFile and QDomDocument which appears to magically do-the-right-thing, and read either the latin1 or utf8 chars as one would expect.

(0002260)
madsdyd (administrator)
2009-01-27 23:00

Oh, that reference to 0000308, should be 0000408, of course...

(0002269)
administrator (administrator)
2009-01-28 10:01

Hi. We could use something like the following:

QFile f(fileName);
f.open(QIODevice::WriteOnly);
f.write(doc.toString().toUtf8());
f.close();

However I want to make sure that this does not have side effects. What if you have some clips in your project that have accents or white space in clip name or path (for example: "/home/user/vidéo test.mpg"). I am a bit worried that inigo needs local encoding to read the .kdenlive file for rendering.

Mads, could you try my patch, then create a project with such a clip and put it in timeline. Can you successfully re-open it in kdenlive. Can inigo play the project file ?

(0002270)
Granjow (developer)
2009-01-28 10:08

@madsdyd, yes, my default locale is utf-8. I guessed it depended on the locale but I couldn't find the place where the file was written ;)

(0002633)
xzhayon (developer)
2009-04-05 09:04

jb forced titles to be saved in utf8. don't really know if this affects also .kdenlivetitle files, but i guess so
feedback needed

(0002635)
Granjow (developer)
2009-04-05 10:15

It is working here. Madsdyd?

Issue History
Date Modified	Username	Field	Change
2009-01-24 19:27	Granjow	New Issue
2009-01-24 19:27	Granjow	Build/Install Method	=> Manual build from SVN
2009-01-24 19:27	Granjow	Note Added: 0002190
2009-01-25 14:36	Granjow	Note Added: 0002194
2009-01-25 14:36	Granjow	Status	new => assigned
2009-01-25 14:36	Granjow	Assigned To	=> Granjow
2009-01-25 15:59	Granjow	Status	assigned => feedback
2009-01-25 16:00	Granjow	Resolution	open => fixed
2009-01-27 13:02	madsdyd	Note Added: 0002234
2009-01-27 13:02	madsdyd	Status	feedback => acknowledged
2009-01-27 14:20	Granjow	Note Added: 0002242
2009-01-27 17:20	madsdyd	Relationship added	has duplicate 0000434
2009-01-27 17:27	madsdyd	Note Added: 0002244
2009-01-27 20:01	Granjow	File Added: unicodetest.kdenlive
2009-01-27 20:02	Granjow	Note Added: 0002251
2009-01-27 20:04	Granjow	Note Added: 0002252
2009-01-27 22:42	madsdyd	Note Added: 0002259
2009-01-27 23:00	madsdyd	Note Added: 0002260
2009-01-28 10:01	administrator	Note Added: 0002269
2009-01-28 10:08	Granjow	Note Added: 0002270
2009-04-05 09:04	xzhayon	Note Added: 0002633
2009-04-05 09:04	xzhayon	Status	acknowledged => feedback
2009-04-05 10:15	Granjow	Note Added: 0002635
2009-04-06 08:44	administrator	Relationship added	parent of 0000746
2009-07-07 11:28	Granjow	Status	feedback => resolved
2009-07-07 11:28	Granjow	Fixed in Version	=> 0.7.3
2009-07-07 11:29	Granjow	Status	resolved => closed

Notes
(0002190) Granjow (developer) 2009-01-24 19:27	$ file AC_moves.kdenlivetitle AC_moves.kdenlivetitle: ASCII text

(0002194) Granjow (developer) 2009-01-25 14:36	Yeh :) I guess it works.

(0002234) madsdyd (administrator) 2009-01-27 13:02	Acknowledged. And, the fix in svn rev. 2950 fixes this. It also breaks all old .kdenlivetitles that contains 8 bit chars, but I guess we have to live with that. I am not resolving this issue though, as I want your comment on the fact that title clips are still saved in ascii in the .kdenlive files? At least from my inspections. Isn't that a problem too? Is it related to this issue or something else? Regards Mads

(0002242) Granjow (developer) 2009-01-27 14:20	Are they? I took a look at my .kdenlive file here, and the unicode characters are displayed correctly.

(0002244) madsdyd (administrator) 2009-01-27 17:27	Please test again - I have tried it twice now. When saving the individual clip, it works as expected, when saving to .kdenlive, it saves like this: HEJ <E6><F8> <E5><C6><D8><C5> and, it should say "HEJ æøåÆØÅ" - the latter should be multibyte in utf8. I guess you agree it should be consistent with the clip saving?

(0002251) Granjow (developer) 2009-01-27 20:02	I've uploaded a test file where I've put your characters and some other unicode characters (?, ?, ?, ?). They are still displayed as they were after reloading, and when searching for one of the characters in the .kdenlive file, I find them. Could you upload a file from your machine?

(0002252) Granjow (developer) 2009-01-27 20:04	Oh crap, mantis doesn't seem to understand unicode, lol. It were the unicode characters 1234, 2345, 5432, 4321. See here http://decodeunicode.org/u+1234 [^] and so on. PS: The file has been saved in unicode nevertheless, I've checked that.

(0002259) madsdyd (administrator) 2009-01-27 22:42	Hmm. Inspecting the attachment, it does appear that your saved file contains utf8 in the xmldata part of the kdenlive_producer for the clip. Mine does not. Sigh. .. Found the reason. Files are saved using QTextStream, which uses QTextCodec::codecForLocale(), which on my system fails to supply an UTF-8 locale, but defaults to Latin1. (See also 0000308). Try this patch: Index: src/kdenlivedoc.cpp =================================================================== --- src/kdenlivedoc.cpp (revision 2990) +++ src/kdenlivedoc.cpp (working copy) @@ -19,6 +19,7 @@ #include <QCryptographicHash> #include <QFile> +#include <QTextCodec> #include <KDebug> #include <KStandardDirs> @@ -925,7 +926,9 @@ KMessageBox::error(kapp->activeWindow(), i18n("Cannot write to file %1", path)); return false; } + kDebug() << "For QTextStream, default locale is: " << QTextCodec::codecForLocale()->name(); QTextStream out(&file); + out.setCodec( QTextCodec::codecForName( "UTF-8" ) ); out << sceneList.toString(); file.close(); return true; This makes the chars become utf8 on my system. I expect that your default locale is utf-8 not? Now, as we claim in the kdenlive files preprocessing directive that it is utf8, we really should not rely on the users current locale to encode/decode the kdenlive files. So, I sort of think we should apply this patch. What do you think? Oh, and AFAICT, reading the files rely on QFile and QDomDocument which appears to magically do-the-right-thing, and read either the latin1 or utf8 chars as one would expect.

(0002260) madsdyd (administrator) 2009-01-27 23:00	Oh, that reference to 0000308, should be 0000408, of course...

(0002269) administrator (administrator) 2009-01-28 10:01	Hi. We could use something like the following: QFile f(fileName); f.open(QIODevice::WriteOnly); f.write(doc.toString().toUtf8()); f.close(); However I want to make sure that this does not have side effects. What if you have some clips in your project that have accents or white space in clip name or path (for example: "/home/user/vidéo test.mpg"). I am a bit worried that inigo needs local encoding to read the .kdenlive file for rendering. Mads, could you try my patch, then create a project with such a clip and put it in timeline. Can you successfully re-open it in kdenlive. Can inigo play the project file ?

(0002270) Granjow (developer) 2009-01-28 10:08	@madsdyd, yes, my default locale is utf-8. I guessed it depended on the locale but I couldn't find the place where the file was written ;)

(0002633) xzhayon (developer) 2009-04-05 09:04	jb forced titles to be saved in utf8. don't really know if this affects also .kdenlivetitle files, but i guess so feedback needed

(0002635) Granjow (developer) 2009-04-05 10:15	It is working here. Madsdyd?