Adium

Opened 12 years ago

Last modified 8 years ago

#8320 new defect

Spotlight returns no results for chat log search in Leopard

Reported by: vnoel Owned by:
Milestone: Adium bugs Component: Logging
Version: 1.1.4 Severity: major
Keywords: spotlight, leopard, search, chatlog Cc:
Patch Status:

Description

In Leopard, if you search for the contents of a chat log using the Spotlight menu, the correct result is not returned (i.e. the chat log does not appear).

If you use mdfind, however, the chatlog file is found correctly.

Apparently, Apple decided that whatever lies inside the user's library folder is out of bounds for the Spotlight search:
http://discussions.apple.com/thread.jspa?threadID=1201778&tstart=0

Of course, contacts from Addressbook and emails from Mail show up in the results, so there must be a way to work around this limitation (perhaps by specifying a folder as "searchable") except if those data types are hardcoded somehow. The stuff inside ~/Library/Caches/Metadata apparently IS searchable (I'm getting results searching for Bibdesk stuff).

Change History (64)

comment:1 Changed 12 years ago by jas8522

  • Milestone set to Adium X 1.1.5

comment:2 follow-up: Changed 12 years ago by evands

  • Milestone changed from Adium X 1.1.5 to Waiting on Apple

If a link is made to the logs directory inside ~/Library/Caches/Metadata, does Spotlight follow it and index appropriately?

Pending a workaround, this should be milestoned 'waiting on apple' and a link to the radar for the issue added to the ticket.

comment:3 in reply to: ↑ 2 ; follow-ups: Changed 12 years ago by vnoel

Replying to evands:

If a link is made to the logs directory inside ~/Library/Caches/Metadata, does Spotlight follow it and index appropriately?

Nope. Even copying a log folder or a chatlog file inside this folder does not work. Copying the log folder or a chatlog file inside the ~/Library folder does not work. But as soon as you move the folder on the desktop, results show up.

Please note that Spotlight actually indexes correctly chatlog files in all of these locations - it's just that the menu chooses not to display them because it does not like their location.

IMHO, there must be some kind of API to "tell" Spotlight that it is ok to return results from a given folder inside ~/Library, but what do I know. I've looked a bit at the ADC web documentation, but I just have a free account so my powers are limited :-)

comment:4 Changed 12 years ago by evands

Hm. But you're sure Bibdesk is getting results displayed from within that folder?

comment:5 Changed 12 years ago by troplin

What about moving the default location of the logs to the ~/Documents folder? Since these are not program logfiles that are only for debugging/troubleshooting but chat transscripts that are more like documents.

This would also make sense because people may also want to access them from finder. In my opinion you should only have to open ~/Libary to install stuff and delete preference files and so on, but not to access files frequently.

Also many other programs (e.g. Colloquy, Snak) have their transscrpipt folders in ~/Documents.

comment:6 in reply to: ↑ 3 ; follow-up: Changed 12 years ago by cheungpat

Replying to vnoel:

IMHO, there must be some kind of API to "tell" Spotlight that it is ok to return results from a given folder inside ~/Library, but what do I know. I've looked a bit at the ADC web documentation, but I just have a free account so my powers are limited :-)

You can write a Spotlight Importer to help Spotlight indexes chatlogs. If you look inside /System/Library/Spotlight you will see there are a bunch of importers provided by the system, one of them being vCard.mdimporter which enable contacts indexing.

comment:7 in reply to: ↑ 6 Changed 12 years ago by evands

Replying to cheungpat:

Replying to vnoel:

IMHO, there must be some kind of API to "tell" Spotlight that it is ok to return results from a given folder inside ~/Library, but what do I know. I've looked a bit at the ADC web documentation, but I just have a free account so my powers are limited :-)

You can write a Spotlight Importer to help Spotlight indexes chatlogs. If you look inside /System/Library/Spotlight you will see there are a bunch of importers provided by the system, one of them being vCard.mdimporter which enable contacts indexing.

Adium already has a Spotlight Importer which ships with it and works great on 10.4. As mentioned above, mdls shows that spotlight is indexing the chatlog files; the spotlight interface is simply filtering out the results from ~/Library/*

comment:8 Changed 12 years ago by troplin

imho it is a good decision from apple to prevent searches in ~/Library. Searching the library makes only sense for programmers and not users.
Consequence is that chatlogs should be saved into ~/Documents and ~/Library is meant for files used by the program itself and not by the user.

comment:9 in reply to: ↑ 3 Changed 12 years ago by vnoel

As a follow-up to vnoel, I should point out that I'm now getting different results - moving the Logs folder in ~/Library/Caches/Metadata, Spotlight searches return results from the chatlog files. So it seems like the right fix. It is unclear to me why it didn't work the last time, but I'd like it if someone else could try it out and confirm.

As an aside, Voodoopad, like Bibdesk, has moved Spotlight-related stuff in ~/Library/Caches/Metadata for 10.5 : http://flyingmeat.com/download/latest/voodoopadreleasenotes.html (see release notes for 3.2.3). HTH.

comment:10 Changed 12 years ago by evands

We definitely should not put logs under Caches.

comment:11 Changed 12 years ago by chimerical

Being able to move the default location of the logs to ~/Documents would make sense. iChat places its logs there too.

comment:12 follow-up: Changed 12 years ago by willtisdale

There is a way to show all files in a spotlight search, the only limitation is the search has to be performed from a Finder window.

http://www.macworld.com/article/60711/2007/10/spotsyssearch.html

comment:13 in reply to: ↑ 12 Changed 12 years ago by vnoel

Replying to willtisdale:

There is a way to show all files in a spotlight search, the only limitation is the search has to be performed from a Finder window.

IMHO, a user should not have to search for "system files" in order to search his/her chat logs.

comment:14 Changed 12 years ago by willtisdale

Well, I know that. ;-) I was simply saying it is _possible_ to do, that's all. There probably should be an option to move the logs though.

comment:15 follow-up: Changed 12 years ago by aragost

I agree with evands: logs should definitely not go under caches (time machine would ignore them!);
Nor they should be put in the library: they're not support files, they're user-created content (and the ichat example confirms this), and as such they should go in the library

comment:16 in reply to: ↑ 15 Changed 12 years ago by aragost

Replying to aragost:

I agree with evands: logs should definitely not go under caches (time machine would ignore them!);
Nor they should be put in the library: they're not support files, they're user-created content (and the ichat example confirms this), and as such they should go in the library

...in the documents folder, i meant -_-

comment:17 Changed 11 years ago by djmori

comment:18 in reply to: ↑ description Changed 11 years ago by aaulick

Why is this ticket "waiting on apple"? I don't think apple is going to change anything to make it work.

I just spent an hour trying to find a conversation, where I didn't remember which program I used to have the conversation, before finally realizing that spotlight just doesn't show Adium chats. I think that's a common use case, this is a big deal at least for me, and moving the log location seems very easy.

IMO the issue of moving the logs to some other default location where they will be reported by spotlight, is a bug fix that is separate and more important than the feature request of providing a GUI to let the user pick their own arbitrary location.

comment:19 Changed 11 years ago by evands

Yeah, I agree... we need to change our logs location.

comment:20 in reply to: ↑ description Changed 11 years ago by kiil

Please change the milestone from: "Waiting on Apple" to something more suitable such as "Needs feedback from users" or even beter "Adium 1.4" . This ticket has been around now for 10 months and a fix should be quite simple in that chat "protocols" (notice I used the word protocols and not "Logs") could be stored with-in the users document folder as the case is with iChat.

comment:21 Changed 11 years ago by jokke

I agree the location should be changed to allow Spotlight to find the logs. In general I dislike applications putting files in my Documents folder, but even Apple does it and it seems to be the best solution to the problem.

If everyone agrees this is the way forward, could someone please fix this soon? I have no commit permissions so cannot do it myself.

comment:22 Changed 11 years ago by boredzo

  • Milestone changed from Waiting on Apple to Good idea for "later"

jokke: You don't need commit permission; you can contribute a patch.

comment:23 Changed 11 years ago by proton

  • Milestone changed from Good idea for "later" to Adium 1.3.x

Mac OS X 10.5.6 will now show files from ~/Library in Spotlight. This may require a reindex of those files (I'm not 100% sure on that part).

If we do require a reindex code to automatically do this on 10.5.6 can be added etc etc.

comment:24 Changed 11 years ago by evands

  • Resolution set to fixed
  • Status changed from new to closed

(In [25671]) Reimport all spotlight logs on first launch of Adium 1.3.3 or greater to allow mac OS X 10.5.6 to be sure to have an index of our logs. Fixes #8320

comment:25 Changed 11 years ago by Robby

  • Milestone changed from Adium 1.3.x to Adium 1.3.3

comment:26 Changed 11 years ago by evands

(In [25672]) Merged [25671]: Reimport all spotlight logs on first launch of Adium 1.3.3 or greater to allow mac OS X 10.5.6 to be sure to have an index of our logs. Fixes #8320

comment:27 Changed 11 years ago by aragost

I'm still not getting results (10.5.6 and 1.3.3b1). Am I missing something?

comment:28 Changed 11 years ago by dharris

I'm also not getting results (10.5.6 and 1.3.3). Any other special incantations required? In what category will the results be presented?

FWIW, I never ran Adium under Tiger (on this machine). Could there be a configuration issue relating to older installations having set up some other spotlight configuration option?

comment:29 Changed 11 years ago by Robby

  • Milestone changed from Adium 1.3.3 to Adium 1.3.4
  • Resolution fixed deleted
  • Status changed from closed to reopened

Okay, this needs investigating. Thanks for pointing out that it's still broken.

comment:30 Changed 10 years ago by evands

  • Milestone changed from Adium 1.3.4 to Adium 1.4

I don't want to mess with this on a point release, but I think moving the chat logs to ~/Documents/Adium for 1.4 is the way to be done with this.

comment:31 Changed 10 years ago by zacw

  • Severity changed from normal to major

comment:32 Changed 10 years ago by zacw

I say go for it Evan. Ignore the detractors, it's how every other chat program behaves. It's how it's expected, and we're confusing.

comment:33 Changed 10 years ago by dharris

I suppose it's possible that Snow Leopard (OS X 10.6) will provide a new way to handle this. (Disclaimer: I'm just speculating here, I'm not in the developer beta program for this and have no concrete information on this)

However, it will be at least a month until it's released and more like 6-12 months before most Adium users are using it. I say move logs to ~/Documents

comment:34 Changed 10 years ago by zacw

  • Milestone changed from Adium 1.4 to Adium 1.4.x

comment:35 Changed 10 years ago by mchmarny

Sorry, I am coming kind of late into this but wasn't there a way in previous versions to specify the location of your transcripts?

comment:36 follow-ups: Changed 10 years ago by vnoel

FWIW, a way to keep the chat logs in the Library and *still* have the logs show up in Spotlight searches would be to create a Spotlight-friendly searchable index of the logs in the ~/Library/Caches/Metadata folder. Searching through spotlight would show results from the cached-indexed-logs, and opening one (from the spotlight menu or wherever) would open Adium which would then focus the search in the "real" chat log.

Apparently this process is required for any app that handles large monolithic files (e.g. Apple Mail or Bibdesk) since it is the only way to present spotlight search results with a level of detail below the file-level. See e.g. http://lists.apple.com/archives/spotlight-dev/2006/Dec/msg00002.html for a reference concerning Bibdesk.

Of course since Adium saves each chat log in a separate file, this process *should* not be required. But it's an alternate solution, and it might be moderately easy technically speaking, i.e. create duplicates of the chat logs from their usual location in Library to the Caches/Medata location. Both copies would have to stay in sync though, which is a PITA. Dunno, might be easier to just move everything to ~/Documents....

comment:37 follow-up: Changed 10 years ago by squirrelsewer

+1 for storing it in docs - would also help for backups

comment:38 in reply to: ↑ 37 Changed 10 years ago by David Munch

Replying to squirrelsewer:

+1 for storing it in docs - would also help for backups

Most people would probably also backup ~/Library/Application Support/Adium/ anyway, so not really..

comment:39 in reply to: ↑ 36 Changed 10 years ago by simurgh

Replying to vnoel:

Of course since Adium saves each chat log in a separate file, this process *should* not be required. But it's an alternate solution, and it might be moderately easy technically speaking, i.e. create duplicates of the chat logs from their usual location in Library to the Caches/Medata location. Both copies would have to stay in sync though, which is a PITA. Dunno, might be easier to just move everything to ~/Documents....

I'd rather have the logs moved somewhere else than have duplicates sitting on my system. I also know several people who frequently delete their cache folder to free up space.

comment:40 in reply to: ↑ 36 Changed 10 years ago by aragost

Replying to vnoel:

FWIW, a way to keep the chat logs in the Library and *still* have the logs show up in Spotlight searches would be to create a Spotlight-friendly searchable index of the logs in the ~/Library/Caches/Metadata folder. Searching through spotlight would show results from the cached-indexed-logs, and opening one (from the spotlight menu or wherever) would open Adium which would then focus the search in the "real" chat log.

Apparently this process is required for any app that handles large monolithic files (e.g. Apple Mail or Bibdesk) since it is the only way to present spotlight search results with a level of detail below the file-level. See e.g. http://lists.apple.com/archives/spotlight-dev/2006/Dec/msg00002.html for a reference concerning Bibdesk.

Of course since Adium saves each chat log in a separate file, this process *should* not be required. But it's an alternate solution, and it might be moderately easy technically speaking, i.e. create duplicates of the chat logs from their usual location in Library to the Caches/Medata location. Both copies would have to stay in sync though, which is a PITA. Dunno, might be easier to just move everything to ~/Documents....

This seems overly complicated to say the least.
There is an already proposed solution (moving to ~/Documents/) which has no drawbacks - why is this ticket pushed back to more distant releases every time?

comment:41 Changed 10 years ago by aragost

Quick update: no results as well in Snow Leopard

comment:42 Changed 10 years ago by rmaunder

Can anyone give an update on this issue? I still see the problem with Adium 1.3.10 & Snow Leopard 10.6.2

mdfind shows the chat logs under ~/Library/Application Support/Adium 2.0/Users/Default/Logs/

Nothing shows up in a Spotlight search.

Is there a fix ever coming, or a recommended work around from the above suggestions?

Thanks

comment:43 Changed 9 years ago by damnnovice

Any update on this one? I'm currently running 1.4b18 on SL 10.6.4 and no results when using regular spotlight search from the upper right corner.

comment:44 Changed 9 years ago by Robby

  • Owner nobody deleted

comment:45 Changed 9 years ago by jeremyw.sherman

Workaround:

  • Find your current logging directory, e.g., ~/Library/Application Support/Adium 2.0/Users/Default/Logs.
  • Move it to ~/Documents/Adium Logs.
  • ln -s ~/Documents/Adium\ Logs ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs

Adium will continue to create its logs in that same directory - only that directory has been rerouted to one under Documents, so your logs will show up in Spotlight searches.

comment:46 Changed 9 years ago by paradoxmo

If we have a Spotlight importer, it seems silly not to use it... we should probably fix this.

I support moving logs to Documents. We could put a symlink in the old location in Library so that Adium 1.3 will continue to work in case someone decides they want to downgrade.

comment:47 Changed 8 years ago by Robby

  • Milestone changed from Adium 1.4.x to Adium bugs

comment:48 Changed 8 years ago by aragost

Quick update: no results as well in Lion. The saga continues.

comment:49 follow-up: Changed 8 years ago by michaeljkrueger

I agree that chat logs are not system files; it appears that Adium is storing them in a directory intended for program log files, not user data. I hope the chat logs will be moved to a location that is properly indexed by default.

comment:50 in reply to: ↑ 49 ; follow-ups: Changed 8 years ago by David Munch

Replying to michaeljkrueger:

I agree that chat logs are not system files; it appears that Adium is storing them in a directory intended for program log files, not user data. I hope the chat logs will be moved to a location that is properly indexed by default.

But the user is not meant to interface with those files through the Finder, the logviewer is there for that resaon.

comment:51 in reply to: ↑ 50 ; follow-up: Changed 8 years ago by dharris

Last edited 8 years ago by David Munch (previous) (diff)

comment:51 in reply to: ↑ 50 ; follow-up: Changed 8 years ago by dharris

Replying to David Munch:

But the user is not meant to interface with those files through the Finder, the logviewer is there for that resaon.

I disagree with this logic. My email messages and calendar entries are not meant to be interfaced through the Finder either but show up in Spotlight results.

The great thing about Spotlight is that it consolidates search results into one list.

If I remember that a friend told about a "fantastic gizmo-widget", but I regularly communicate with that friend by both email and IM, it's easier for me to search for "gizmo-widget" in Spotlight than to execute multiple searches.

comment:52 in reply to: ↑ 51 ; follow-up: Changed 8 years ago by David Munch

Replying to dharris:

Replying to David Munch:

But the user is not meant to interface with those files through the Finder, the logviewer is there for that resaon.

I disagree with this logic. My email messages and calendar entries are not meant to be interfaced through the Finder either but show up in Spotlight results.

The great thing about Spotlight is that it consolidates search results into one list.

That is not what I am saying. Spotlight indexing is something that should work, but you aren't reading your chatlogs or emails in Finder - You read them in Adium and Mail.

comment:53 in reply to: ↑ 52 Changed 8 years ago by dharris

Replying to David Munch:

That is not what I am saying. Spotlight indexing is something that should work, but you aren't reading your chatlogs or emails in Finder - You read them in Adium and Mail.

Thanks for the clarification. I had misread your earlier comment thinking that you thought that searching in the log viewer was sufficient.

comment:54 follow-up: Changed 8 years ago by michaeljkrueger

Just to be clear, I expect Adium to work the way other applications do: Spotlight is able to index and find the content (such as a mail message or an appointment), and when I open one of the Spotlight results, I read it in the relevant application (such as Mail or iCal). In this case, I would expect Spotlight to find results within my chat logs, and when I open one of those results, I would be able to read the log in the Adium chat log viewer. As others have mentioned, the whole point of Spotlight search is to have a single consolidated search instead of having to run a series of different searches from within potentially dozens of different applications.

At any rate, I believe it would be a mistake to mark this "waiting for Apple," because it seems the Adium chat logs are not being stored in a location that is appropriate for user data (which are indexed), as opposed to things like debugging logs (which are not).

comment:55 Changed 8 years ago by Robby

Well said.

comment:56 in reply to: ↑ 54 Changed 8 years ago by David Munch

Replying to michaeljkrueger:

Just to be clear, I expect Adium to work the way other applications do: Spotlight is able to index and find the content (such as a mail message or an appointment), and when I open one of the Spotlight results, I read it in the relevant application (such as Mail or iCal). In this case, I would expect Spotlight to find results within my chat logs, and when I open one of those results, I would be able to read the log in the Adium chat log viewer. As others have mentioned, the whole point of Spotlight search is to have a single consolidated search instead of having to run a series of different searches from within potentially dozens of different applications.

Everybody agrees with this.

At any rate, I believe it would be a mistake to mark this "waiting for Apple," because it seems the Adium chat logs are not being stored in a location that is appropriate for user data (which are indexed), as opposed to things like debugging logs (which are not).

I don't agree with this. Emails, adressbook entries, calenders, system preferences, websites, etc. etc. are all located in the users Library, yet Spotlight is still able to index them. As you said above, you believe yourself that the chatlogs should be viewed in Adiums chat log viewer and not in Finder, and therefore they should not be available to the user in the Documents folder (Or any other place) and is thus not stored in an incorrect place. Therefore Apples restrictions for the Spotlight API is at fault.

comment:57 follow-up: Changed 8 years ago by aaulick

Therefore Apples restrictions for the Spotlight API is at fault.

Regardless of how correct it is to store the chat logs in the library, marking this bug, "waiting for Apple" is only useful if Apple is planning to make a change.

Are they?

So far we have endured four years of (from the user's perspective) Adium failing to work correctly on a really major feature. Are we going to Insist on never fixing this faulty behavior at all, over a filing dispute?

comment:58 in reply to: ↑ 57 ; follow-up: Changed 8 years ago by David Munch

Replying to aaulick:

Regardless of how correct it is to store the chat logs in the library, marking this bug, "waiting for Apple" is only useful if Apple is planning to make a change.
So far we have endured four years of (from the user's perspective) Adium failing to work correctly on a really major feature. Are we going to Insist on never fixing this faulty behavior at all, over a filing dispute?

The discussion is not on waiting, it is on how it should be done properly in theory. No one is holding you back to provide a patch that will fix the problem in the mean time.

comment:59 in reply to: ↑ 58 ; follow-up: Changed 8 years ago by aragost

Replying to David Munch:

No one is holding you back to provide a patch that will fix the problem in the mean time.

Assuming that he is a developer with the skill to fix this bug is quite a leap. This is a major feature which has been shipped as working but has been broken since more than four years; in the meantime valid solution or workarounds have been proposed, but the only official comment ever given was “waiting on apple”. I think that aaulick's point is pretty valid.

comment:60 in reply to: ↑ 59 ; follow-up: Changed 8 years ago by David Munch

Replying to aragost:

Assuming that he is a developer with the skill to fix this bug is quite a leap. This is a major feature which has been shipped as working but has been broken since more than four years; in the meantime valid solution or workarounds have been proposed, but the only official comment ever given was “waiting on apple”. I think that aaulick's point is pretty valid.

I am not assuming anything. I'm just pointing out that if you want a bug fixed, you should do it yourself or pay someone to do it, instead of expecting the developers who work in this in their free time to fix it just because *you* think it should be working.

comment:61 Changed 8 years ago by Robby

I would like to point out that

  1. Adium's lead dev gave his opinion

Replying to evands:

I don't want to mess with this on a point release, but I think moving the chat logs to ~/Documents/Adium for 1.4 is the way to be done with this.

  1. This ticket is on the "Adium bugs" milestone and has not been on the "Waiting on Apple" milestone for years.
  2. David Munch is a user, not a programmer on Adium.
  3. Anyone with interest and knowledge of Objective-C can give this a shot and provide a patch.
Last edited 8 years ago by Robby (previous) (diff)

comment:62 in reply to: ↑ 60 ; follow-up: Changed 8 years ago by aragost

Replying to David Munch:

Replying to aragost:
I am not assuming anything. I'm just pointing out that if you want a bug fixed, you should do it yourself or pay someone to do it, instead of expecting the developers who work in this in their free time to fix it just because *you* think it should be working.

If you're not assuming he's a developer, telling him to provide a fix, (especially on a broken major feature) is quite rude, and one of the main reasons open source projects are often seen as not welcoming. If there are other priorities or not enough resources, it's better to state it clearly; if the ticket is somehow invalid, it's better to close it.

This ticket is not a personal nuisance, it's currently the highest-voted defect (source: http://trac.adium.im/report/23); I can understand how a user can find disheartening that the proposed solution, after not being implemented, is put into discussion (see comment:50:ticket:8320), and some perplexity about the problem being ignored.

This said, I'm going to look into this ASAP and hopefully submit a patch by the end of the week (or the end of the month if things aren't easy as it seems).

comment:63 in reply to: ↑ 62 Changed 8 years ago by David Munch

Replying to aragost:

If you're not assuming he's a developer, telling him to provide a fix, (especially on a broken major feature) is quite rude, and one of the main reasons open source projects are often seen as not welcoming. If there are other priorities or not enough resources, it's better to state it clearly; if the ticket is somehow invalid, it's better to close it.

I didn't say anything about him doing the code himself, I merely told him to provide a fix. How he does it is not my concern. But why is it okay to blame "Adium for failing to work on this", but not ask him to take action himself?
But this discussion has been beaten to death in 1000 other tickets, and I'll let the discussion end here. Feel free to email me at David at Munch dot as if you want to argue further.

This said, I'm going to look into this ASAP and hopefully submit a patch by the end of the week (or the end of the month if things aren't easy as it seems).

Perfect!

Last edited 8 years ago by David Munch (previous) (diff)
Note: See TracTickets for help on using tickets.