Tom's Handy Dandy Space

Windows Search 3.x/4.x has a great story for how to extend it to index a new file type (implement an IFilter), and it has a great story for how to extend it to support a new data store (by implementing an ISearchProtocol handler) but it does not have a very clear mechanism for indexing a compound file format such as a ZIP file so that individual items in the file can can be returned as individual results.

This blog attempts to describe how you would go about achieving that with the current WS3.X/4.x architecture.

I will use .ZIP as an example. 

A zip file starts in the file system namespace (aka, FILE://c:/test/test.zip ) but within it it can have sub-folders with individual items in it.

Test.Zip

    |-folder1

    |    |-folder2

    |           |- FileX.txt

    |- FileY.doc

The FILE protocol handler will discover when FILE://c:/test/test.zip changes by monitoring file system change logs, and it will invoke an IFilter registered for .ZIP files on that file when the file changes, but it has no knowledge of the internal structure of the .ZIP file itself.  The ZIP file is essentially a data store. 

The trick is to figure out a way to tell the indexer that it is a data store so that individual items can be indexed and retrieved as unique entities.

The first step is to create ISearchProtocol handler for ZIP files.  The key is to design into that protocol handler the ability bind to the source file.  An example way of doing that would be to have the root folder name be escaped path to the .ZIP file.  From there, you can use a hierarchy syntax like any other file format.

Here is an example of what I mean:

ZIP:///escapedPathToZipFile/zipfolder/.../zipfile

Using the above sample data for c:\test\test.zip, my unique URLs would be

zip:///FILE:%2f%2f%2fc:%2ftest%2ftest.zip/

zip:///FILE:%2f%2f%2fc:%2ftest%2ftest.zip/FileY.Doc

zip:///FILE:%2f%2f%2fc:%2ftest%2ftest.zip/folder1

zip:///FILE:%2f%2f%2fc:%2ftest%2ftest.zip/folder1/folder2

zip:///FILE:%2f%2f%2fc:%2ftest%2ftest.zip/folder1/folder2/FileX.txt

As you can see, the protocol handler has the information necessary to bind to the .ZIP file and enumerate the child urls including the inner files so that they can be bound to and indexed by the .DOC and .TXT IFilters.

A couple of key details:

• The root url for a given .ZIP file should emit PKEY_Search_IsClosedDirectory (System.Search.IsClosedDirectory) on the urls which are the root .ZIP file urls (Example: zip:///FILE:%2f%2f%2fc:%2ftest%2ftest.zip/ should emit IsClosedDirectory = TRUE)  This tells the indexer that if the date on this url has not changed, it does not need to process any of the child urls.
• Every child url for that url should emit PKEY_Search_IsFullyContained (System.Search.IsFullyContained) on the child urls of the root .ZIP url.  Normally at the end of an incremental crawl the indexer will treat all unvisited urls as items which should be deleted.  But the root .ZIP file we said should not process root Urls because we KNOW nothing changed.  Emitting this property as true tells the indexer that if this url has not been processed at the end of an incremental crawl it should not be deleted.  It will only be deleted if the root item has changed and it is not visited.

Now you have a protocol handler which can process and expose the data of the .Zip file as individual items.  (Of course, for it to work with the shell, you need to implement a shell namespace extension via IShellFolder for the ZIP://* urls, but that's a seperate topic.)

There is only 1 fly in the ointment.  Windows Search requires a start page for a protocol for knowing what urls to do incremental crawls on and also to know which urls should be ignored when they are found.  But we can't start with a url for each .Zip file, because we don't know where each zip file is.  The ZIP protocol handler's start page url needs to be able to enumerate all at the root all of the escaped paths of all of the .ZIP files (which aren't necessarily in the FILE: namespace...aka, their could be a MAPI url which points to a .ZIP file as an attachment).  How do you solve this?

This is where you get a bit tricky.  What you should do is to register a root such as ZIP:/// as a start page.  This basically says, "all zip files start here.".  Then your protocol handler when processing the root ZIP: url should generate the list of child urls to emit by querying Windows Search for all of the urls with System.FileExtension = 'Zip'.  Escape those urls to remove the slashes and return them as child urls and you are good to go. 

An example query to retrieve what you want would be:

SELECT system.itemurl, System.DateModified FROM SystemIndex
WHERE System.FileExtension='ZIP' OR System.MimeType='mimetypeforzip'

Periodically, windows search will do an incremental crawl on your ZIP:/// root url, at which point you simply reflect back the list of urls which windows search already is maintaining which are zip urls.  If a deletion was discovered in the native store where the zip file is stored, then it won't show up in your enumeration and that branch of the tree in the zip will get removed.

NOTE: To bind to the .ZIP data for another protocol handler you should ideally go through the IShellFolder for that url to bind to the storage of the object, and not assume it is always a file...this will allow you to work with attachments in mail stores, etc.

NOTE 2: When emitting child urls for each .ZIP file you should use PKEY_Search_UrlToIndexWithModificationTime to  pass the System.DateModified of the actual ZIP file so that the indexer only crawls it if it has changed. 

This will work just fine, but there is a further optimization which you can do. Ideally, you would like to have your ZIP urls indexed right when they are created or modified and not have to wait for an incremental crawl to discover their new state.

You could in theory monitor the file system yourself for .ZIP file changes, but that's a bit heavy handed, and also won't work for other stores such as MAPI.  he best way to accomplish this is a bit tricky again...create an IFilter for the .ZIP file.  Whenever your IFilter is invoked, it's because that URL has been discovered or changed.  At that point, generate an event for the zip url appropriate for the source url via the IGatherNotifyInline interface. This gives you the ability to immediately tell the indexer there is new data to be indexed without having to wait for the incremental crawl. 

I've been looking at porting an application to work on windows X64.

They did some very strange things with this release, which I guess is to make things easier for people porting their code, but IMHO makes the long term way confusing.

The basic rules around 64 bit is that a 64 bit app can only load 64 bit dlls and a 32 bit app can only load 32 bit dlls.  That's reasonable enough.

Also, Microsoft has gone to great lengths to create a Wow64 layer which allows a 32bit application to run native.  That's cool too.

But you start to look at how they did things and its very strange.

1. Where is System64?

When the transistion from 16 to 32 bits was made there was a problem with where things lived.  The Windows team came up with a sensible solution. C:\windows\system is for 16 bit dlls (because win3.X used that path for 16 bit windows.)  c:\windows\system32 was for 32 bit dlls.  So you would think that 64 bit dll's go into c:\windows\system64, right?  NOPE. Instead...when you are running a 32 bit app on XP x64, the OS lies to the app and tells it is putting it's dll's into c:\windows\system32, but instead it puts them into c:\windows\system32Wow64.  oookaay...So where do the 64 bit DLLs go?  Why it puts them into c:\windows\system32...of course!  So the 64 bit DLLs are in the directory labeled 32 and the 32 bit dlls are in the directory labeled 64. <sigh>

2. Where is InprocServer64?

The original windows with COM servers did the same trick.  If you had an inproc server and it was 16 bit DLL, it was called InProcServer. If you had a 32 bit implementation it was called InProcServer32.  So you would expect that if you had a in proc server which was 64 bit it would be registered as InProcServer64, right?  NOPE.  Again, they decided to have the API lie based on the bitness of the appliaction calling the windows API.  So InProcServer32 puts to a 32 bit DLL if you are a 32 bit app and a 64 bit dll if you are a 64 bit app. What if you want to publish a COM component for both 32 bit and 64 bit? I think you are out of luck.  WTF-99, that's what I say. <sigh, sigh>

3. 32 bit COM servers can be used by using COM marshallers...but they dont.

So if you have a 64 bit app and you call a 32 bit com out of proc server it all works, because COM knows how to marshal across process boundaries.  And COM has the ability of making any inproc COM server run in another process (look up DllSuragate on MSDN)...so you would think that if you had a 64 bit app which tried to use an InprocServer which happened to be the wrong bit-ness they would just move it into an out of proc 32 bit surragate.  They don't.  This would be OK if you could register a COM server for both, but as I said in point #2, I don't think you can.  So how am I supposed to create a COM Server that can be used by anyone?  The only way I can see to do it is to always have it be out of proc.  This causes perf problems of course.  I don't think I can create both a 32 bit and 64 bit COM handler, so I am not certain how on earth OLE will work, when you don't know the bitness of the container you are in.  <sigh, sigh, sigh>

I would love to be wrong about these points.  It seems like they are bending over backwards for this stuff, at the expense of us having to live with badly named directories and confusing and miosleading registry keys for a long long time.

Alas...that's my rant against the wind for today.

-Tom

I am a developer on the MSN Toolbar Suite doing the desktop search component and it has finally released to the world!  Whew, that was a lot of work, but it is really gratifying to see the positive comments out there!

A couple of things that people have made which I would like to comment on are:

A. IE is required (Firefox rulez) etc.

Actually, the only thing that requires IE is the IE toolbar.  The outlook toolbar, deskbar and explorer bar all work just with desktop search without using IE at all. 

B. Microsot if just copying Google/Apple/Copernic (fill in your favorite DS product here.)

Of course we look at our competetors.  If we didn't, these same people would be saying "Microsoft is soo stuuupid cause they don't do what XYZ corp does."  Give me a break.  We've been working on search in different forms for years.  I have been working on search in various forms for years.  Yes, this team started working on MSN Search in June.  OK. Fine. Think what you will.

At any rate, had to get that off my chest.  More will come later. Best regards!

-Tom

I had an urgent change to make to our code-base.  All I had to do was to move some code from one component over to another component. 

The actual coding took only about an hour.  The amount of time figuring out the magic set of .LIBs, flags, include paths, libpaths, #defines etc. which would allow my project to BUILD was 5 freakin hours! Contrast this with .NET and a system which is self-describing and I'm just crying in my beer.

<bah!>

Golly, here I am at work instead of home, and messing with the new spaces.  Kinda cool.