Adium

Opened 7 years ago

Closed 5 years ago

Last modified 5 years ago

#13586 closed defect (fixed)

searching transcripts is very unreliable

Reported by: jwz Owned by:
Milestone: Adium 1.5.3 Component: Adium Core
Version: 1.4b17 Severity: major
Keywords: Transcript Viewer Cc:
Patch Status:

Description (last modified by Robby)

Summary

Searching through old Adium transcripts almost never works. The text is there in the log files (as revealed by grep) but I get no (or too few) search results when searching for words in either the Chat Transcript Viewer or with Spotlight. It's not just some words or particular searches. I don't see a pattern as to what works and what doesn't, but it's bad enough that I can never rely on it.

Steps to reproduce

  1. in Chat Transcript Viewer, I select All Contacts, and type "testicles" in the search field. It shows one matching chat transcript. When I click on it, it shows me no results.
  2. In Spotlight, I get no Adium-related results.
  3. In Terminal, I type:

find ~/Library/Application\ Support/Adium\ 2.0/Users/Default/Logs -type f -print0 | xargs -0 grep -i testicles | wc -l

and get hits on 11 .xml, .chatlog and/or .AdiumHTMLLog files.

Expected results

I expected results.

Actual results

I did not get results.

Notes

If you can tell me some things to try to help diagnose this, I'd be happy to.

About a month ago I rebuilt my Spotlight database with "mdutil -E /", so I do know that doing that doesn't fix it.

I do not believe this is a new problem. I believe searching chat transcripts has basically never worked reliably since I started using Adium 4+ years ago.

Change History (51)

comment:1 Changed 7 years ago by proton

  • Status changed from new to pending

Please try deleting ~/Library/Caches/Adium/Default/Logs.index and ~/Library/Caches/Adium/Default/DirtyLogs.plist and try again

comment:2 Changed 7 years ago by jwz

  • Status changed from pending to new

Ok, after doing that, and allowing it to re-index, I get 8 results in the Transcripts window. There are 2 xml files that grep finds but Adium does not.

I still get 0 Adium-related results in Spotlight.

comment:3 Changed 7 years ago by benrobertsabq

I have the same problem with the same symptoms and steps as above. Adium 1.4b17 MacOS 10.5.8.

I closed Adium, deleted ~/Library/Caches/Adium/Default/Logs.index and ~/Library/Caches/Adium/Default/DirtyLogs.plist and reopened Adium. It doesn't look like there was a full reindex, only logs since I deleted the index files are searchable now.

Index file is only 16kb and shows no sign of growing, with 200mb of chat logs so it should be getting a bit bigger if a reindex were occurring.

I then manually edited the XML in the DirtyLogs.plist file to add all the log files from one of my Contacts. On reopening Adium this seems to have correctly forced a reindex of just those files - they now return search results ok. But is a bit of a hack, and I have not yet validated whether the reindex was complete and/or fully accurate.

Is there another way to force a full reindex of all log files? Or other data I can gather to help diagnose?

comment:4 Changed 7 years ago by earthmkii

  • Status changed from new to pending

Try "File -> Import -> Reindex Adium Logs"

comment:5 Changed 7 years ago by earthmkii

  • Milestone set to Adium 1.4.x
  • Status changed from pending to new

Oh, I see, it doesn't highlight the search term in the opened transcript viewer.

We'll look into it, but since it's not a regression, I see no reason to hold up 1.4

comment:6 Changed 7 years ago by iBoo

Searching has also not worked for me since I started using Adium years ago.

I just upgraded to 1.4rc1, it is nice to have the Import Reindex Adium Logs.

When I "Search Content" (transcripts) it still doesn't work, but what is interesting is if my search is up (finding 0 matching) the instant I choose Reindex Adium Logs under the file menu I get results for all of my contacts. With the search word highlighted and everything. The reindex will occur and leave the results intact. But, the second I go and click on another contact the search doesn't work anymore.

I have transcripts from both a Jabber gmail.com account and AOL IM account.

At least I can find something now with one additional step. Let me know if you need any other info.

comment:7 Changed 7 years ago by paradoxmo

Yeah, I'll go ahead and 3rd/4th this as far as search reliability. I almost never get a complete set of search results.

Replying to jwz:

I still get 0 Adium-related results in Spotlight.

jwz, finding 0 spotlight results is #8320, I believe.

comment:8 Changed 6 years ago by Robby

  • Keywords Transcript Viewer added

comment:9 follow-up: Changed 6 years ago by aleph1

I also cannot find some Adium results by searching with Spotlight. I am running Adium 1.4.1 and MacOS 10.6.5.

I've figured that the only transcripts that Spotlight will find, even from the command line using mdfind, are old ones from before 2008-08. Looking at my chat logs folder I notice that after this date Adium stopped storing the chat log in a file with a .chatlog extension. After that it started creating a directory with the .chatlog extension using the same name it previously used for the chatllog file, and storing the actual chat log within the directory using the same name but minus the .chatlog extension and adding an .xml extension. I.e. "user (2010-01-01T00.00.00-0000).chatlog"became "user (2010-01-01T00.00.00-0000).chatlog/user (2010-01-01T00.00.00-0000).chatlog.xml".

This change causes the AdiumSpotlightImporter.mdimporter to fail. If you run "mdimport -d 4 'user (2010-01-01T00.00.00-0000).chatlog'" you will get the error:
(Info) Import: Import 'user (2010-01-01T00.00.00-0000).chatlog' type 'com.adiumx.xmllog' using '/Applications/Adium.app/Contents/Library/Spotlight/AdiumSpotlightImporter.mdimporter'

But if you copy the xml file within the .chatlog directory elsewhere and change its extension to .chatlog and run mdimport on it, the chatlog will be successfully indexed using AdiumSpotlightImporter.mdimporter.

Why are the chat logs now stored within a directory with an xml extensions? Was this done to support group chats, so as to store one individual per xml file?

In any case, it seems AdiumSpotlightImporter.mdimporter was not updated when the change was made. It still asumes that the file with the .chatlog extension is a chat log and not a directory storing the chat log.

comment:10 in reply to: ↑ 9 ; follow-up: Changed 6 years ago by aleph1

Replying to aleph1:

In any case, it seems AdiumSpotlightImporter.mdimporter was not updated when the change was made. It still asumes that the file with the .chatlog extension is a chat log and not a directory storing the chat log.

Hmm. I see in the source that it actually tries to handle this case by appending the last path component minus the chatlog extension and adding the xml extension:

99 	/* If we have a chatLog bundle, we want to get the text content for the xml file inside */
100 	if (isDir) pathToFile = (CFStringRef)[(NSString *)pathToFile stringByAppendingPathComponent:
101 		[[[(NSString *)pathToFile lastPathComponent] stringByDeletingPathExtension] stringByAppendingPathExtension:@"xml"]];

So I am not sure what the error is, but something is causing AdiumSpotlightImporter.mdimporter to fail when the log is within a chatlog directory, but not when it actually has a chatlog extension.

Last edited 6 years ago by wixardy (previous) (diff)

comment:11 in reply to: ↑ 10 Changed 6 years ago by pgurov

Replying to aleph1:

Replying to aleph1:

In any case, it seems AdiumSpotlightImporter.mdimporter was not updated when the change was made. It still asumes that the file with the .chatlog extension is a chat log and not a directory storing the chat log.

Hmm. I see in the source that it actually tries to handle this case by appending the last path component minus the chatlog extension and adding the xml extension:

99 	/* If we have a chatLog bundle, we want to get the text content for the xml file inside */
100 	if (isDir) pathToFile = (CFStringRef)[(NSString *)pathToFile stringByAppendingPathComponent:
101 		[[[(NSString *)pathToFile lastPathComponent] stringByDeletingPathExtension] stringByAppendingPathExtension:@"xml"]];

So I am not sure what the error is, but something is causing AdiumSpotlightImporter.mdimporter to fail when the log is within a chatlog directory, but not when it actually has a chatlog extension.

Hi there. I'm having similar problems so I decided to have a loot into the source code.
I guess the code you are mentioning is correct but it is not on the write place. Currently it is only in CopyTextContentForFile function, but I think it should be also in GetMetadataForXMLLog. Since GetMetadataForXMLLog will be executed when the system is indexing the files.

I copied the code to GetMetadataForXMLLog, reindexed all chat logs and now I can search using the Find function in Finder. The only thing that I don't like is that the chat logs are marked as system files. So every time I need to specify that the "System files" "are included".

Is there a way to tell spotlight that this is not a system file?

comment:12 Changed 6 years ago by Robby

  • Milestone changed from Adium 1.4.x to Adium bugs

comment:13 Changed 5 years ago by dharris

This ticket has two different discussions.

  1. Searching in the Chat Transcript Viewer
  2. Searching via Spotlight

Searching via spotlight is discussed in #8320.

I recommend that any discussion of search results not showing up in spotlight be moved to that ticket.

comment:14 follow-up: Changed 5 years ago by jwz

Wow, I just want to express my continued frustration with this bug. I reported this over two years ago, and searching via the transcripts viewer continues to never, ever work, even after trying all the "clear caches" suggestions above. I've pretty much given up on ever finding an old conversation through a mechanism other than grep, but for laughs, I gave it a try again today. Yup. Still broken.

Is it just me, and this actually works fine for everyone else?

Or do most people who use Adium think it's ok that there's no search function?

comment:15 in reply to: ↑ 14 Changed 5 years ago by earthmkii

  • Milestone changed from Adium bugs to Adium 1.5

Replying to jwz:

Wow, I just want to express my continued frustration with this bug. I reported this over two years ago, and searching via the transcripts viewer continues to never, ever work, even after trying all the "clear caches" suggestions above. I've pretty much given up on ever finding an old conversation through a mechanism other than grep, but for laughs, I gave it a try again today. Yup. Still broken.

Is it just me, and this actually works fine for everyone else?

Or do most people who use Adium think it's ok that there's no search function?

A lot of work went into this bug for the upcoming 1.5 release, which is in beta now. Do you continue to see this bad search behavior after reindexing your logs in the 1.5 betas?

I've updated this ticket's milestone to reflect this.

comment:16 Changed 5 years ago by jwz

Yes, I am running 1.5b9.

comment:17 Changed 5 years ago by Catfish_Man

I can confirm that for me at least, the new log indexing and searching code in 1.5 works *great* on ~70,000 logs. Used it just tonight.

comment:18 Changed 5 years ago by Robby

jwz, please grab a nightly from http://nightly.adium.im and post a debug log for your issue.

Last edited 5 years ago by Robby (previous) (diff)

comment:19 Changed 5 years ago by Thijs Alkemade <thijsalkemade@…>

(In c0dcb2b73496) Don't crash when the log index failed to initialize when trying to delete a log, just bail.

Fixes #15771, refs #13586

comment:20 Changed 5 years ago by Thijs Alkemade <thijsalkemade@…>

(In f272f5b32043) It was possible that two threads were trying to open the same index at the same time, so this check has to be in the dispatch_sync block.

Also, add some more logging when an index is actually closed.

Refs #13586

comment:21 follow-up: Changed 5 years ago by Robby

jwz, please try a current nightly from http://nightly.adium.im/ and check if above change (f272f5b32043) improves things for you. If your issue remains, please post a debug log (CurrentAdiumDebug).

Last edited 5 years ago by Robby (previous) (diff)

comment:22 in reply to: ↑ 21 Changed 5 years ago by jwz

Replying to Robby:

That *may* have fixed it, but I'm not totally sure yet.

I quit; installed nightly build Adium_1.5rc1r4542; "rm -r ~/Library/Caches/Adium/Default/Logs.index ~/Library/Caches/Adium/Default/DirtyLogs.plist"; re-launched; waited for the re-index of 16726 transcripts.

It's finding more than it did before (more than 0), but I think there's still some stuff missing. Though I haven't found a smoking gun yet.

comment:23 Changed 5 years ago by dalahäst

This happens to me as well, even after reindexing logs—reindexing sometimes brings marginal improvements, but things I know are there do not show up, or too many irrelevant results show up (searching for a phrase, for example, often brings up results that contain only one of the words).

comment:24 Changed 5 years ago by robotive

  • Milestone changed from Adium 1.5 to Adium 1.5.1

comment:25 Changed 5 years ago by Robby

jwz, any news on this?

comment:26 Changed 5 years ago by bedge

Still broken in 1.5.1b2
This severely hinders the client functionality.

Can the priority on this be bumped to something above normal?

comment:27 Changed 5 years ago by Robby

  • Description modified (diff)
  • Milestone changed from Adium 1.5.1 to Adium 1.5.x
  • Severity changed from normal to major

Fine with me. :)

comment:28 follow-up: Changed 5 years ago by Robby

Everyone, please try this build: http://nightly.adium.im/adium-adium-1.5.2/Adium_1.5.2b1r4873.dmg and let us know how it works in relation to the issue described here.

comment:29 follow-up: Changed 5 years ago by msmithng

Success!

I was experiencing the same issues, and I just bumped this nightly and finally it's completed the index and searching is back as advertised.

Do you need any details from me for this issue?

comment:30 in reply to: ↑ 29 Changed 5 years ago by Robby

Replying to msmithng:

Success!

I was experiencing the same issues, and I just bumped this nightly and finally it's completed the index and searching is back as advertised.

Thanks for the feedback.

Do you need any details from me for this issue?

Not for now, thanks. :)

comment:31 Changed 5 years ago by ShawnRisk

I just tried this, and this failed for me. What I did was: download the 1.5.2b1 from the beta page and reindexed my chat logs, and try to search for any word and nothing shows up. All it says is Loading... where the text is suppose to show up.

comment:32 follow-up: Changed 5 years ago by msmithng

@ShawnRisk

Did you perform the aforementioned steps of:

rm -r ~/Library/Caches/Adium/Default/Logs.index 

and

rm -r ~/Library/Caches/Adium/Default/DirtyLogs.plist

I did this prior to running File > Import > Reindex Adium Logs

comment:33 in reply to: ↑ 32 Changed 5 years ago by ShawnRisk

Replying to msmithng:

@ShawnRisk

Did you perform the aforementioned steps of:

rm -r ~/Library/Caches/Adium/Default/Logs.index 

and

rm -r ~/Library/Caches/Adium/Default/DirtyLogs.plist

I did this prior to running File > Import > Reindex Adium Logs

I tried this and only found DirtyLogs.plist there, so I deleted that and when I went to reindex nothing happened or it was super fast this time around. Anyways, I was not able to search anything just as before.

comment:34 Changed 5 years ago by djc6

I downloaded the nightly in comment28 and it fixed the problem temporarily (very exciting!). I was suddenly able to search my chat transcipts again.

However, now its back to "Loading..." when searching for terms I know exist in many places. After a handful of successful searches I repeated one of the same searches and it caused adium to crash; ever since then I'm back to getting no results.

comment:35 Changed 5 years ago by djc6

BTW, closing Adium and issuing:

rm -r ~/Library/Caches/Adium/Default/Logs.index 
rm -r ~/Library/Caches/Adium/Default/DirtyLogs.plist

didn't help once I lost the ability to search with this nightly.

Last edited 5 years ago by djc6 (previous) (diff)

comment:36 Changed 5 years ago by mathuaerknedam

Ticket #16052 has been marked as a duplicate of this ticket.

comment:37 follow-up: Changed 5 years ago by meshko

Running 1.5.1 (2012/6/6) here, searching doesn't work. When I try to reindex it just hangs on indexing 406 of 423 transcripts.

Also I need to know 2 things:

  1. where are the actual transcripts so that I can manually look at them?
  2. is jwz who reported this bug originally _the_ jwz or not?
Last edited 5 years ago by Robby (previous) (diff)

comment:38 in reply to: ↑ 37 Changed 5 years ago by mathuaerknedam

Replying to meshko:

Running 1.5.1 (2012/6/6) here, searching doesn't work. When I try to reindex it just hangs on indexing 406 of 423 transcripts.
Wer
Also I need to know 2 things:
1) where are the actual transcripts so that I can manually look at them?

Unless you've setup a different profile name, the logs should be located in ~/Library/Application Support/Adium 2.0/Users/Default/Logs.

If there isn't anything sensitive in your logs, you could zip them up and email them to me and I'll try to reproduce the problem.

comment:39 Changed 5 years ago by meshko

Sorry, sharing logs is out of question. This is work account with potentially sensitive information.
I found a bunch of malformed XML in there and deleted all of them, but it didn't help. Indexing hangs at a random point.

comment:40 in reply to: ↑ 28 ; follow-up: Changed 5 years ago by sphynx

Replying to Robby:

Everyone, please try this build: http://nightly.adium.im/adium-adium-1.5.2/Adium_1.5.2b1r4873.dmg and let us know how it works in relation to the issue described here.

Please do this.

comment:41 in reply to: ↑ 40 Changed 5 years ago by meshko

Works perfectly, thanks!

Replying to sphynx:

Replying to Robby:

Everyone, please try this build: http://nightly.adium.im/adium-adium-1.5.2/Adium_1.5.2b1r4873.dmg and let us know how it works in relation to the issue described here.

Please do this.

comment:42 Changed 5 years ago by CPngN

1.5.2 release from Sunday still does not work. First try is great. Closing the viewer, re-opening it, and searching for the same term results in "Search for 'searchterm' complete" with a blank result box. If I hit Enter after my search term, "Loading..." appears in the results box. Nothing ever appears. With nothing at all in the search box, I can see logs as usual.

Also of note: I had to reindex just to get the initial results to work. After closing and reopening the viewer it seems to get "stuck".

comment:43 Changed 5 years ago by sphynx

Yes, closing the log viewer and reopening will make the search fail. This is a known problem that will be fixed in 1.5.3.

comment:44 Changed 5 years ago by djc6

Downgrading to Adium 1.4.5 is the only workaround that has been successful for me. I've been searching my logs regularly without any issues. At least two other people have posted that this works for them.

http://forums.cocoaforge.com/viewtopic.php?f=13&t=24155#p133465

comment:45 follow-up: Changed 5 years ago by Robby

djc6, CPngN and others,

Please try the 1.5.3 beta and let us know if it works better.

comment:46 in reply to: ↑ 45 Changed 5 years ago by CPngN

Replying to Robby:

djc6, CPngN and others,

Please try the 1.5.3 beta and let us know if it works better.

Seems to be working. Reopened a couple times and found random text globally as well as with 1 contact/channel. Hooray!...?

comment:47 Changed 5 years ago by Robby

Same here... :)

comment:48 Changed 5 years ago by djc6

I've been using the beta for a few days now and I haven't had any issues searching transcripts. Thanks!

comment:49 Changed 5 years ago by Robby

  • Milestone changed from Adium 1.5.x to Adium 1.5.3
  • Resolution set to fixed
  • Status changed from new to closed

Thanks for the feedback. I'll close the ticket for now.

  • If someone finds searching is still fundamentally broken, please comment and we'll reopen.
  • For more specific issues, I would suggest filing a new ticket.

comment:50 Changed 5 years ago by djc6

I broke it again. I'm using 1.5.3rc1.

This is how I broke it.. I was searching my transcripts (4790 transcripts / 307 contacts) for 'esi' (as in edge side includes) and it was taking forever. So I stopped the search thinking I should select a particular contact first and then search for esi (instead of all of my contacts); but now all searches come up empty.

I did Import -> Reindex Adium Logs - it went thru it really fast (saw it climb up to 4790 transcripts in seconds) but still can't search my logs.

I then closed Adium and deleted:

rm -r ~/Library/Caches/Adium/Default/Logs.index 
rm -r ~/Library/Caches/Adium/Default/DirtyLogs.plist

but only DirtyLogs.plist existed.

Tried searching for stuff again - all searches come up empty. I then did Import -> Reindex Adium Logs this time it took a while (minutes).

After this, I was able to search my logs again! I was able to do my 'esi' search on a particular contact.

Last edited 5 years ago by djc6 (previous) (diff)

comment:51 Changed 5 years ago by Robby

djc6, thanks for letting us know.

I think a new ticket would be a good idea though. If you decide to file one, please make sure you include the steps of reproduction. A debug log would probably help as well.

Last edited 5 years ago by Robby (previous) (diff)
Note: See TracTickets for help on using tickets.