From john.maguire on March 02, 2010 10:46:53

Labels: OpSys-All Component-MusicLibrary

From davidsansome on March 03, 2010 03:51:19

Status: Accepted
Labels: Milestone-0.2

From kadaimx on March 20, 2010 17:29:30

This defect is still present in build 383.

The main problem of this is not only the CPU consumption, but the heavy usage of HD
also, because Clementine appear to check every-file tags and look closely if they
have been touched.

A way to solve the problem is using the mtime in the database... but there can be
another way, that is having saved in a configuration the time of the last library
scan (when it started) and then, when starting Clementine (or doing a manual
update/scan) just make a quick search (using find command, having as result only the
changed files since the last scan) and just update those files... saving time.

Now, the command find is not present in each system, and even, it may not be all
acurrate if the dates and process are handled with care.

From davidsansome on March 21, 2010 09:29:59

After the first time you add a directory to the library, Clementine will only open a
file and look at the tags again if the mtime has changed since the last time it
looked. This means it has to "stat" each file on startup.

Clementine has to look at the mtime of each file - it can't skip a directory if the
mtime of the directory itself hasn't changed - because audio players don't typically
touch the directory itself when they change the tags in a file inside, and they
certainly don't touch the parent directories.

I've made the library scanner run at a lower IO priority on linux now ( r393 ), and the
whole thread should run at a lower CPU priority too. Hopefully this should make it
less intrusive.

I'll leave this issue open for the start/pause scan menu items.

Labels: -Milestone-0.2 Milestone-0.3

From kadaimx on March 25, 2010 00:56:11

Observations on the current way the library gets scanned:
The library scans itself almost completely even if only a file was added, making it a
long process to get updated.

Changes on library get queued, but making almost a scan for each change.
Explanation:
Let's say that we add a file to the library and then we wait Clementine to update the
library.
Once it does start to update it, we add another file to the library... and some
minutes later, we add a third file.
The library then will be scanned a total of 3 times consuming a lot of time.

The scan process would, in any case, have a smart way to queue changes or detect
them... for example, updating only the relevant parts of the modified library, or
awaiting to build another update job once the current one is finished, if is not
possible to append the changes to the current update.

This is a issue when managing large libraries.
Size around ~34000 files.

From davidsansome on April 01, 2010 10:02:07

I've had another think about this and I've come up with a much better library scanner.
It will now take a little bit longer to scan the library the very first time, but
after that rescanning the library on startup should take less than a second. Also, if
a file changes it will only rescan the directory that file was in, instead of the
entire library.
Have a go and let me know if this works better for you.

From kadaimx on April 01, 2010 15:07:08

Thank you for your efforts!

Now, I'm going to say what I have found out in my test (some included adding new
links, files and images, othere where just merely updates... usuall onto stressing
Clementine on a relugar usage)

Library information (as for 1/april/2010):
36751 files (ranging from Music files to images and text files in some cases)
2159 sub-folders
135.4 GB total
All of them are properly tagged, and arranged accordly certaing details, as follows:
YEAR/ARTIST/ALBUM/discnumber-tracknumber.extension
So, a file path usually locks alike: /home/kadai/Music/COLLECTION/2010/The Smash
Brothers/Bacon EP. [Digital Edition]/

The first test was to create a empty folder in the library base folder
(~/Music/COLLECTION, a folder called '0'), Clementine detected instantly the folder
and started to update the library, but it took to the software around 10 minutes to
finish, pretty fair considerating the timeit used to spend, but yet a lot.

Console output:
Subdir "/home/kadai/Music/COLLECTION" changed under directory
"/home/kadai/Music/COLLECTION" id 1

Then, I followed to drop in that folder the link of a folder that was outside the
library, and also to copy a folder inside it. Both contained music files (mp3s) and
where named: "GFCA-195 - from CD" (the link) and "GFCA-196 - from CD" (copied
content) and Clementine did not saw the changes.

Moved then them to the folder "Various Artist" inside "1900" (previously existing
when Clementine was started) and the software did not detected the change.

Restarted Clementine then, and console went spammed with mensages like this:
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/2003/ZUN
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/2003/Various
Artists
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/2003/松枝賀子 &
江口貴勅
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/2003/Lix
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1981/Various
Artists/松本零士の新竹取物語１０００年女王　ＴＶシリーズ　音楽編
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1981/Various
Artists
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1988/Nintendo
Sound Team, Koji Kondo/Super Smash Bros. Brawl -Q- NES
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1988/植松伸夫
/ファイナル・ファンタジー Ⅰ・Ⅱ 全曲集
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1988/Various
Artists/スーパーマリオブラザーズ3 -G.S.M. Nintendo 1-
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1988/Various
Artists/[non-album tracks]
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1988/Nintendo
Sound Team, Koji Kondo
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1988/植松伸夫
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1988/Various
Artists
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/2005
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1996

(All those paths existed on my HD)
The main window did not said Clementine was updating the library, but the load on CPU
indicated that it was. the load then finished and Clementine did not detected any
change. Restarted again.

Once restarted, Clementine sucessfully located the newly included folders, but showed
this message:

QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1900/Various
Artists/GFCA-196 - from CD
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1900/Various
Artists/GFCA-195 - from CD
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1900/Various
Artists/GFCA-196 - from CD
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1900/Various
Artists/GFCA-195 - from CD
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1900/Various
Artists/GFCA-196 - from CD
QFileSystemWatcher: failed to add paths: /home/kadai/Music/COLLECTION/1900/Various
Artists/GFCA-195 - from CD

I think that maybe the fault is mine. I'm going to delete the database and re-scan my
entire collection for a more recent version.

From kadaimx on April 01, 2010 19:12:40

Wow, the system you happened to create works wonderfully.

The only issues I see with it, is that it does not puts an eye on folders that were
created on the collection/library root folder while Clementine is running, or at
least that is what happened while I created a folder called '0' (with no quotes) and
yet persisted if I created a folder named "2011" (with no quotes) and dropped there a
folder with files on it. But, the software was somewhat aware of the change:
Subdir "/home/kadai/Music/COLLECTION" changed under directory
"/home/kadai/Music/COLLECTION" id 1

But if I organize a folder within an existing folder (ej:
~/Music/COLLECTION/2010/Various Artists) it is detected correctly and added
instantly. Now, if I then delete the files almost instantly after them being added,
Clementine gets a little confused (it detects the change, but not that the files were
deleted):

"/home/kadai/Music/COLLECTION/2010/Various Artists/GFCA-195 - from CD/15656 -
Unknown.mp3" created
"/home/kadai/Music/COLLECTION/2010/Various Artists/GFCA-195 - from CD/10505 -
Unknown.mp3" created
"/home/kadai/Music/COLLECTION/2010/Various Artists/GFCA-195 - from CD/10909 -
Unknown.mp3" created
Subdir "/home/kadai/Music/COLLECTION/2010/Various Artists" changed under directory
"/home/kadai/Music/COLLECTION" id 1 <-- Clementine detected that I deleted the
files, but had no idea what to do.

It does requires some heavy test and being polished... despite that, it works as a charm.

Kudos!

From kadaimx on April 01, 2010 21:51:30

Just something I re-realized after some close watching.
While Clementine does not say anything in its main window, or either in the console,
I have seen that after having detected the changes (and applied them) the software
takes around ~2 minutes of HD usage.

Hopefully, this is a small drawback.

From kadaimx on April 03, 2010 11:07:34

And, other issue.
Apparently Clementine does not -see- all changes that are made on the collection
structure.

I have been editing the composer tag of various files (from various locations, at
almost the same time) but Clementine does not say anything about detect them being
changed.

From davidsansome on April 04, 2010 08:16:43

Thanks for testing :)

I've fixed your bug from comment 7 (not monitoring new directories that are created
while Clementine is running), and I've fixed a bug that stops directories from being
rescanned if you delete and recreate them.

I can't reproduce the second bug in comment 8 though - it works for me if I delete a
file immediately after creating it.

I'm not going to be able to fix the bug in comment 10 - some other music players
don't "touch" the directory a song is in when they change the tags in it. Clementine
only looks for changes to the directories now, so it won't be able to see these at
all. That's a limitation of the quicker scanning I'm afraid :(

From kadaimx on April 04, 2010 13:32:20

Maybe I just did not wait enough time to make Clementine look for it (I added the
folder, and almost instantly deleted it... I was being evil with the software :p )
and was just a false positive.

And do not worry then for the issue on comment 10, gladly the scanner is pretty quick
as how it is now... and as far I know, if you add files, delete them, or other
certain actions, the folder usually gets touched what is the important thing.

Of course, is important to leave this as reference for if some users uses other
software to edit their files so they can have a idea why this happens, and do not
report bugs against this.

From kadaimx on April 04, 2010 16:05:51

This appear to not to happen always, but sometimes, when adding/moving files on the
library, Clementine does properly detect the changes giving all the proper output,
but then, the system keeps with a high load and HD usage, very similar to the old
behavior.

I was able to reproduce it moving to a album folder and the moving out from it a
"cover.jpg" file, dropped it on my Desktop and saw then Clementine loading heavily
the HD for around 4 to 5 minutes, while I expected it to only watch that small change
and do a small thing.

Aside, Clementine detected changes on all the files of the folder, but not sure if it
is related, or if that makes Clementine look everywhere else:

[Thread 0xb567db70 (LWP 13918) exited]
Clementine(13909)/ KSycocaPrivate::openDatabase: Trying to open ksycoca from
"/var/tmp/kdecache-kadai/ksycoca4"
Subdir "/home/kadai/Music/COLLECTION/2010/Various Artists/ロックマン10 宇宙からの脅
威!! オリジナルサウンドトラック" changed under directory
"/home/kadai/Music/COLLECTION" id 1
"/home/kadai/Music/COLLECTION/2010/Various Artists/ロックマン10 宇宙からの脅威!! オリ
ジナルサウンドトラック/35.mp3" changed
"/home/kadai/Music/COLLECTION/2010/Various Artists/ロックマン10 宇宙からの脅威!! オリ
ジナルサウンドトラック/35.mp3" metadata changed

( ... repeated for other 44 files ... )

"/home/kadai/Music/COLLECTION/2010/Various Artists/ロックマン10 宇宙からの脅威!! オリ
ジナルサウンドトラック/10.mp3" changed
"/home/kadai/Music/COLLECTION/2010/Various Artists/ロックマン10 宇宙からの脅威!! オリ
ジナルサウンドトラック/10.mp3" metadata changed

From kadaimx on April 04, 2010 16:21:37

Well, it took actually around 10 minutes.
And, if I executed "CTRL+C" on the console, I was able to kill this, what links it to
Clementine.

Using the Debug Symbols.

From davidsansome on April 05, 2010 08:23:12

It's fine that all the files in that folder "changed" - that was Clementine clearing
the album artwork for those tracks.
I'm still not sure why disk usage continued for another 10 minutes though. Could you
run it under strace? It will print out loads of output, but maybe you could look at
what it's doing (what files it's opening, etc.) after it's supposedly finished
scanning the library.

Run:
strace -f clementine

From kadaimx on April 05, 2010 09:36:52

Well, executed clementine on console using strace, and the output was too long that
it just got lost beyond the console (and I do not know how to dump it to a text file)

But, one of the things I noticed is that, when I moved the jpg file, the console got
flooded with messages like this one:

[pid 5379] stat64("/home/kadai/Music/COLLECTION/2001/Daft Punk/Discovery/12 - Short
Circuit.mp3", {st_mode=S_IFREG|0644, st_size=5043981, ...}) = 0
[pid 5379] lstat64("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai", {st_mode=S_IFDIR|0700, st_size=24576, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION/2001", {st_mode=S_IFDIR|0755,
st_size=12288, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION/2001/Daft Punk",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION/2001/Daft Punk/Discovery",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION/2001/Daft Punk/Discovery/12 - Short
Circuit.mp3", {st_mode=S_IFREG|0644, st_size=5043981, ...}) = 0
[pid 5379] _llseek(18, 24683520, [24683520], SEEK_SET) = 0
[pid 5379] read(18, "\0\0\0\0Soundtrack/home/kadai/Music/"..., 1024) = 1024
[pid 5379] stat64("/home/kadai/Music/COLLECTION/2008/Various Artists/Dissidia -Final
Fantasy- Original Soundtrack/215 -
\343\200\214\343\203\216\343\203\274\343\203\236\343\203\253\343\203\220\343\203\210\343\203\253
-original-\343\200\215from FINAL FANTASY X.mp3", {st_mode=S_IFREG|0644,
st_size=5062850, ...}) = 0
[pid 5379] lstat64("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai", {st_mode=S_IFDIR|0700, st_size=24576, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION/2008", {st_mode=S_IFDIR|0755,
st_size=40960, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION/2008/Various Artists",
{st_mode=S_IFDIR|0755, st_size=28672, ...}) = 0
[pid 5379] lstat64("/home/kadai/Music/COLLECTION/2008/Various Artists/Dissidia
-Final Fantasy- Original Soundtrack", {st_mode=S_IFDIR|0755, st_size=20480, ...}) = 0

But the problem is that I removed the file from:
"/home/kadai/Music/COLLECTION/2010/Various Artists/ロックマン10 宇宙からの脅威!! オリ
ジナルサウンドトラック/" and Clementine was scanning the files from the 2001 and 2008
folder instead of just the ones on 2010. And I personally think that it was exploring
all the others also, but I was not able to catch that.

Why then was scanning those folders?
Those folders had the next Modified date (according to Dolphin)
2001 - 03/18/10 12:59 am
2008 - 04/05/10 12:41 am
2010 - 04/04/10 3:48 pm
COLLECTION - 04/01/10 8:27 pm
Music 03/22/10 12:44 am
kadai - 04/05/10 - 11:27 am (like 10 minutes ago)
/home - 12/09/09 05:30 pm

But then, no idea why Clementine was stating those folders (specially the last two,
that are not-so-related to the music library), or if having a modified date different
triggered it.

I hope this helps.

From davidsansome on April 15, 2010 09:50:34

I haven't been able to reproduce this since you posted your comment so I'm going to
close the bug. If I discover what was wrong I'll let you know :)

Status: Fixed

From kadaimx on April 24, 2010 16:38:20

I apparently discovered what was wrong with my setup.

In my previous installs, I had activated the -user folder- encryption. I'm not sure
how it exactly works, but it happens to store the encrypted data elsewhere, and then
show the files and folders to the user much like links (but as normal files to the user)

What I'm thinking is that the files where touched -somehow- by the encrypt system,
because once I did not installed my system with such feature, Clementine just updates
everything on the fly, not having that 10 minutes scan.

This is the reason of the problem, so if someone else comes with this, be sure to
understand that it is a drawback from having your personal folder encrypted.

From davidsansome on April 26, 2010 04:19:11

Ah that's useful to know, and interesting that it would cause that behaviour. Glad
you figured it out!