Spotlight returns no results for chat log search in Leopard

[vnoel]

In Leopard, if you search for the contents of a chat log using the Spotlight menu, the correct result is not returned (i.e. the chat log does not appear).

If you use mdfind, however, the chatlog file is found correctly.

Apparently, Apple decided that whatever lies inside the user's library folder is out of bounds for the Spotlight search:  http://discussions.apple.com/thread.jspa?threadID=1201778&tstart=0

Of course, contacts from Addressbook and emails from Mail show up in the results, so there must be a way to work around this limitation (perhaps by specifying a folder as "searchable") except if those data types are hardcoded somehow. The stuff inside ~/Library/Caches/Metadata apparently IS searchable (I'm getting results searching for Bibdesk stuff).

[evands]

If a link is made to the logs directory inside ~/Library/Caches/Metadata, does Spotlight follow it and index appropriately?

Pending a workaround, this should be milestoned 'waiting on apple' and a link to the radar for the issue added to the ticket.

[vnoel]

Replying to evands:

If a link is made to the logs directory inside ~/Library/Caches/Metadata, does Spotlight follow it and index appropriately?

Nope. Even copying a log folder or a chatlog file inside this folder does not work. Copying the log folder or a chatlog file inside the ~/Library folder does not work. But as soon as you move the folder on the desktop, results show up.

Please note that Spotlight actually indexes correctly chatlog files in all of these locations - it's just that the menu chooses not to display them because it does not like their location.

IMHO, there must be some kind of API to "tell" Spotlight that it is ok to return results from a given folder inside ~/Library, but what do I know. I've looked a bit at the ADC web documentation, but I just have a free account so my powers are limited :-)

[evands]

Hm. But you're sure Bibdesk is getting results displayed from within that folder?

[troplin]

What about moving the default location of the logs to the ~/Documents folder? Since these are not program logfiles that are only for debugging/troubleshooting but chat transscripts that are more like documents.

This would also make sense because people may also want to access them from finder. In my opinion you should only have to open ~/Libary to install stuff and delete preference files and so on, but not to access files frequently.

Also many other programs (e.g. Colloquy, Snak) have their transscrpipt folders in ~/Documents.

[cheungpat]

Replying to vnoel:

IMHO, there must be some kind of API to "tell" Spotlight that it is ok to return results from a given folder inside ~/Library, but what do I know. I've looked a bit at the ADC web documentation, but I just have a free account so my powers are limited :-)

You can  write a Spotlight Importer to help Spotlight indexes chatlogs. If you look inside /System/Library/Spotlight you will see there are a bunch of importers provided by the system, one of them being vCard.mdimporter which enable contacts indexing.

[evands]

Replying to cheungpat:

Replying to vnoel:

IMHO, there must be some kind of API to "tell" Spotlight that it is ok to return results from a given folder inside ~/Library, but what do I know. I've looked a bit at the ADC web documentation, but I just have a free account so my powers are limited :-)

You can  write a Spotlight Importer to help Spotlight indexes chatlogs. If you look inside /System/Library/Spotlight you will see there are a bunch of importers provided by the system, one of them being vCard.mdimporter which enable contacts indexing.

Adium already has a Spotlight Importer which ships with it and works great on 10.4. As mentioned above, mdls shows that spotlight is indexing the chatlog files; the spotlight interface is simply filtering out the results from ~/Library/*

[troplin]

imho it is a good decision from apple to prevent searches in ~/Library. Searching the library makes only sense for programmers and not users. Consequence is that chatlogs should be saved into ~/Documents and ~/Library is meant for files used by the program itself and not by the user.

[vnoel]

As a follow-up to vnoel, I should point out that I'm now getting different results - moving the Logs folder in ~/Library/Caches/Metadata, Spotlight searches return results from the chatlog files. So it seems like the right fix. It is unclear to me why it didn't work the last time, but I'd like it if someone else could try it out and confirm.

As an aside, Voodoopad, like Bibdesk, has moved Spotlight-related stuff in ~/Library/Caches/Metadata for 10.5 :  http://flyingmeat.com/download/latest/voodoopadreleasenotes.html (see release notes for 3.2.3). HTH.

[evands]

We definitely should not put logs under Caches.

[chimerical]

Being able to move the default location of the logs to ~/Documents would make sense. iChat places its logs there too.

[willtisdale]

There is a way to show all files in a spotlight search, the only limitation is the search has to be performed from a Finder window.

 http://www.macworld.com/article/60711/2007/10/spotsyssearch.html

[vnoel]

Replying to willtisdale:

There is a way to show all files in a spotlight search, the only limitation is the search has to be performed from a Finder window.

IMHO, a user should not have to search for "system files" in order to search his/her chat logs.

[willtisdale]

Well, I know that. ;-) I was simply saying it is _possible_ to do, that's all. There probably should be an option to move the logs though.

[aragost]

I agree with evands: logs should definitely not go under caches (time machine would ignore them!); Nor they should be put in the library: they're not support files, they're user-created content (and the ichat example confirms this), and as such they should go in the library

[aragost]

Replying to aragost:

I agree with evands: logs should definitely not go under caches (time machine would ignore them!); Nor they should be put in the library: they're not support files, they're user-created content (and the ichat example confirms this), and as such they should go in the library

...in the documents folder, i meant -_-

[aaulick]

Why is this ticket "waiting on apple"? I don't think apple is going to change anything to make it work.

I just spent an hour trying to find a conversation, where I didn't remember which program I used to have the conversation, before finally realizing that spotlight just doesn't show Adium chats. I think that's a common use case, this is a big deal at least for me, and moving the log location seems very easy.

IMO the issue of moving the logs to some other default location where they will be reported by spotlight, is a bug fix that is separate and more important than the feature request of providing a GUI to let the user pick their own arbitrary location.

[evands]

Yeah, I agree... we need to change our logs location.

[kiil]

Please change the milestone from: "Waiting on Apple" to something more suitable such as "Needs feedback from users" or even beter "Adium 1.4" . This ticket has been around now for 10 months and a fix should be quite simple in that chat "protocols" (notice I used the word protocols and not "Logs") could be stored with-in the users document folder as the case is with iChat.

[jokke]

I agree the location should be changed to allow Spotlight to find the logs. In general I dislike applications putting files in my Documents folder, but even Apple does it and it seems to be the best solution to the problem.

If everyone agrees this is the way forward, could someone please fix this soon? I have no commit permissions so cannot do it myself.

[boredzo]

jokke: You don't need commit permission; you can contribute a patch.

[proton]

Mac OS X 10.5.6 will now show files from ~/Library in Spotlight. This may require a reindex of those files (I'm not 100% sure on that part).

If we do require a reindex code to automatically do this on 10.5.6 can be added etc etc.

[evands]

(In [25671]) Reimport all spotlight logs on first launch of Adium 1.3.3 or greater to allow mac OS X 10.5.6 to be sure to have an index of our logs. Fixes #8320

[evands]

(In [25672]) Merged [25671]: Reimport all spotlight logs on first launch of Adium 1.3.3 or greater to allow mac OS X 10.5.6 to be sure to have an index of our logs. Fixes #8320

[aragost]

I'm still not getting results (10.5.6 and 1.3.3b1). Am I missing something?

[dharris]

I'm also not getting results (10.5.6 and 1.3.3). Any other special incantations required? In what category will the results be presented?

FWIW, I never ran Adium under Tiger (on this machine). Could there be a configuration issue relating to older installations having set up some other spotlight configuration option?

[Robby]

Okay, this needs investigating. Thanks for pointing out that it's still broken.

[evands]

I don't want to mess with this on a point release, but I think moving the chat logs to ~/Documents/Adium for 1.4 is the way to be done with this.

[zacw]

I say go for it Evan. Ignore the detractors, it's how every other chat program behaves. It's how it's expected, and we're confusing.

[dharris]

I suppose it's possible that Snow Leopard (OS X 10.6) will provide a new way to handle this. (Disclaimer: I'm just speculating here, I'm not in the developer beta program for this and have no concrete information on this)

However, it will be at least a month until it's released and more like 6-12 months before most Adium users are using it. I say move logs to ~/Documents

[mchmarny]

Sorry, I am coming kind of late into this but wasn't there a way in previous versions to specify the location of your transcripts?

[vnoel]

FWIW, a way to keep the chat logs in the Library and *still* have the logs show up in Spotlight searches would be to create a Spotlight-friendly searchable index of the logs in the ~/Library/Caches/Metadata folder. Searching through spotlight would show results from the cached-indexed-logs, and opening one (from the spotlight menu or wherever) would open Adium which would then focus the search in the "real" chat log.

Apparently this process is required for any app that handles large monolithic files (e.g. Apple Mail or Bibdesk) since it is the only way to present spotlight search results with a level of detail below the file-level. See e.g.  http://lists.apple.com/archives/spotlight-dev/2006/Dec/msg00002.html for a reference concerning Bibdesk.

Of course since Adium saves each chat log in a separate file, this process *should* not be required. But it's an alternate solution, and it might be moderately easy technically speaking, i.e. create duplicates of the chat logs from their usual location in Library to the Caches/Medata location. Both copies would have to stay in sync though, which is a PITA. Dunno, might be easier to just move everything to ~/Documents....

[squirrelsewer]

+1 for storing it in docs - would also help for backups

[David Munch]

Replying to squirrelsewer:

+1 for storing it in docs - would also help for backups

Most people would probably also backup ~/Library/Application Support/Adium/ anyway, so not really..

[simurgh]

Replying to vnoel:

Of course since Adium saves each chat log in a separate file, this process *should* not be required. But it's an alternate solution, and it might be moderately easy technically speaking, i.e. create duplicates of the chat logs from their usual location in Library to the Caches/Medata location. Both copies would have to stay in sync though, which is a PITA. Dunno, might be easier to just move everything to ~/Documents....

I'd rather have the logs moved somewhere else than have duplicates sitting on my system. I also know several people who frequently delete their cache folder to free up space.

[aragost]

Replying to vnoel:

FWIW, a way to keep the chat logs in the Library and *still* have the logs show up in Spotlight searches would be to create a Spotlight-friendly searchable index of the logs in the ~/Library/Caches/Metadata folder. Searching through spotlight would show results from the cached-indexed-logs, and opening one (from the spotlight menu or wherever) would open Adium which would then focus the search in the "real" chat log. Apparently this process is required for any app that handles large monolithic files (e.g. Apple Mail or Bibdesk) since it is the only way to present spotlight search results with a level of detail below the file-level. See e.g.  http://lists.apple.com/archives/spotlight-dev/2006/Dec/msg00002.html for a reference concerning Bibdesk. Of course since Adium saves each chat log in a separate file, this process *should* not be required. But it's an alternate solution, and it might be moderately easy technically speaking, i.e. create duplicates of the chat logs from their usual location in Library to the Caches/Medata location. Both copies would have to stay in sync though, which is a PITA. Dunno, might be easier to just move everything to ~/Documents....

This seems overly complicated to say the least. There is an already proposed solution (moving to ~/Documents/) which has no drawbacks - why is this ticket pushed back to more distant releases every time?

[aragost]

Quick update: no results as well in Snow Leopard

[rmaunder]

Can anyone give an update on this issue? I still see the problem with Adium 1.3.10 & Snow Leopard 10.6.2

mdfind shows the chat logs under ~/Library/Application Support/Adium 2.0/Users/Default/Logs/

Nothing shows up in a Spotlight search.

Is there a fix ever coming, or a recommended work around from the above suggestions?

Thanks