EventID 2567 Unable to resolve Contentdistributor / Renewing self signed certificates used for FAST Search Server for SharePoint 2010

It all started with a lot of events in the application log of the SharePoint app servers. A lot of  events 2567. If you have never seen this event before you might think it is nothing that bad, in the end everything works fine(or at least it looks like).  If you see that event in your Windows logs most probably your crawls for FAST Search server are not working, you will see the crawls working all the time, without ending.

SCOM 2012 SP1 gives no message about this, search still works from SharePoint but the results are not up to date.

EventId 2567

<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Server Search” Guid=”{C8263AFE-83A5-448C-878C-1E5F5D1C4252}” />
<EventID>2567</EventID>
<Version>14</Version>
<Level>2</Level>
<Task>148</Task>
<Opcode>0</Opcode>
<Keywords>0x4000000000000000</Keywords>
<TimeCreated SystemTime=”2013-07-17T11:10:32.008550800Z” />
<EventRecordID>337271</EventRecordID>
<Correlation />
<Execution ProcessID=”4952” ThreadID=”5452” />
<Channel>Application</Channel>
<Computer>SP-S-APP1.domain.com</Computer>
<Security UserID=”S-1-5-21-39937011-1643330254-59529505-77556” />
</System>
<EventData>
<Data Name=”string0“>sp-s-index.domain.com:13391</Data>
<Data Name=”string1“>Failed to initialize session with document engine: Unable to resolve Contentdistributor</Data>
</EventData>
</Event>

You will also see messages in the ULS logs.

This is caused because the certificate used to authenticate SharePoint servers to FAST Search Server has expired.

Renewing the self signed certificate is very easy, I’ll give you an easy step by step below. I am only talking about self signed certificates in this article but  the recommended way to use FAST Search with SharePoint is to use certificates from your CA.

Lets’s start!

You have four big steps:

Step 1

Stop FAST Search services

Step 2

Renew the certificate on the FAST Search primary server  and then on each non-administration FAST Search server.

Step 3

Start FAST Search services on each server after you renewed the certificate.

Step 4

Import and configure the FAST Search connector for SharePoint(on each server used in the FAST Search topology).

Step 1

On your FAST Search boxes stop the FAST Search services. If you have multiple servers in you FAST Search infrastructure start with the administration server.  To find out which one is this server  run  “nctrl status” and see which one has the config role listed, start with this one.

You will need to stop two services: fastsearchservice and fastsearchmonitoring.

In order to stop the services open a powershell window  and type:

net stop fastsearchservice
net stop fastsearchmonitoring

or just use Services and stop: “FAST Search for SharePoint” and “FAST Search for SharePoint Monitoring”.

Step 2

Start with the FAST administration server, and renew the self signed certificate. Open a powershell window and browse to your FAST Search installation folder \FASTSearch\installer\scripts and run:

.\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

Supply a password for this certificate, you will be required to use the same key both on the other FAST servers but also on the SharePoint servers.

If you have multiple FAST Search servers you must first start the services listed at step 3 on the FAST config server before doing step 2 on the second FAST Search server(which is not your FAST config server).

Step 3

Start the two FAST Search services you stopped at step 1 by using start instead of stop.

net start fastsearchservice
net start fastsearchmonitoring

Step 4

Move your attention to the SharePoint servers used in the FAST Search infrastructure.

Create a folder on each of those servers and copy the following two files from you FAST config server:

-The certificate from the FAST administration server located at

FAST_install_folder\data\data_security\cert\FASTSearchCert.pfx

-the script located at the address below, from the same FAST admin server

FAST_install_folder\installer\scripts\securefastsearchconnector.ps1

On each SharePoint server hosting the administration component which is part of the FAST Search topology open an admin powershell window and browse to the location of the folder you created above and run:

.\SecureFASTSearchConnector.ps1 –certPath “folder location\FASTSearchCert.pfx” –ssaName “name of your content SSA” –userName “domain\username”

FAST topology

On each of the remaining SharePoint server configured as a Crawl component you will have to:

  1. Import the FASTSearchCert.pfx certificate in the certificate store under Certificates(Local Computer)\Personal.  Follow this guide to start the Certificates snap-in for the local computer and then this one to import the certificate.
  2. Run the following:
.\SecureFASTSearchConnector.ps1 -ssaName "name of your content SSA" -username "domain\username"

This command will return the thumbprint of the available certificates and a prompt asking whether you want to use the suggested certificate. In my case I had to do a restart of the server running the Admin component and the FAST Search servers before this command being successful.

The ssaName is the name of the content Search Service Application you use and the userName is the  user used on your SharePoint servers to run the “SharePoint Server Search 14” service.

You will receive a confirmation that the FAST Search connector successfully connected to the FAST servers. For example in my case this was not true on each server but everything worked fine even without confirmation.

Make sure you don’t forget any of the servers, if you miss one of them your crawls will start but will not finish, making this very hard to troubleshoot.

Use ULS logs to troubleshoot if you have any problem.

Advertisements