EventID 2567 Unable to resolve Contentdistributor / Renewing self signed certificates used for FAST Search Server for SharePoint 2010

It all started with a lot of events in the application log of the SharePoint app servers. A lot of  events 2567. If you have never seen this event before you might think it is nothing that bad, in the end everything works fine(or at least it looks like).  If you see that event in your Windows logs most probably your crawls for FAST Search server are not working, you will see the crawls working all the time, without ending.

SCOM 2012 SP1 gives no message about this, search still works from SharePoint but the results are not up to date.

EventId 2567

<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Server Search” Guid=”{C8263AFE-83A5-448C-878C-1E5F5D1C4252}” />
<EventID>2567</EventID>
<Version>14</Version>
<Level>2</Level>
<Task>148</Task>
<Opcode>0</Opcode>
<Keywords>0x4000000000000000</Keywords>
<TimeCreated SystemTime=”2013-07-17T11:10:32.008550800Z” />
<EventRecordID>337271</EventRecordID>
<Correlation />
<Execution ProcessID=”4952” ThreadID=”5452” />
<Channel>Application</Channel>
<Computer>SP-S-APP1.domain.com</Computer>
<Security UserID=”S-1-5-21-39937011-1643330254-59529505-77556” />
</System>
<EventData>
<Data Name=”string0“>sp-s-index.domain.com:13391</Data>
<Data Name=”string1“>Failed to initialize session with document engine: Unable to resolve Contentdistributor</Data>
</EventData>
</Event>

You will also see messages in the ULS logs.

This is caused because the certificate used to authenticate SharePoint servers to FAST Search Server has expired.

Renewing the self signed certificate is very easy, I’ll give you an easy step by step below. I am only talking about self signed certificates in this article but  the recommended way to use FAST Search with SharePoint is to use certificates from your CA.

Lets’s start!

You have four big steps:

Step 1

Stop FAST Search services

Step 2

Renew the certificate on the FAST Search primary server  and then on each non-administration FAST Search server.

Step 3

Start FAST Search services on each server after you renewed the certificate.

Step 4

Import and configure the FAST Search connector for SharePoint(on each server used in the FAST Search topology).

Step 1

On your FAST Search boxes stop the FAST Search services. If you have multiple servers in you FAST Search infrastructure start with the administration server.  To find out which one is this server  run  “nctrl status” and see which one has the config role listed, start with this one.

You will need to stop two services: fastsearchservice and fastsearchmonitoring.

In order to stop the services open a powershell window  and type:

net stop fastsearchservice
net stop fastsearchmonitoring

or just use Services and stop: “FAST Search for SharePoint” and “FAST Search for SharePoint Monitoring”.

Step 2

Start with the FAST administration server, and renew the self signed certificate. Open a powershell window and browse to your FAST Search installation folder \FASTSearch\installer\scripts and run:

.\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

Supply a password for this certificate, you will be required to use the same key both on the other FAST servers but also on the SharePoint servers.

If you have multiple FAST Search servers you must first start the services listed at step 3 on the FAST config server before doing step 2 on the second FAST Search server(which is not your FAST config server).

Step 3

Start the two FAST Search services you stopped at step 1 by using start instead of stop.

net start fastsearchservice
net start fastsearchmonitoring

Step 4

Move your attention to the SharePoint servers used in the FAST Search infrastructure.

Create a folder on each of those servers and copy the following two files from you FAST config server:

-The certificate from the FAST administration server located at

FAST_install_folder\data\data_security\cert\FASTSearchCert.pfx

-the script located at the address below, from the same FAST admin server

FAST_install_folder\installer\scripts\securefastsearchconnector.ps1

On each SharePoint server hosting the administration component which is part of the FAST Search topology open an admin powershell window and browse to the location of the folder you created above and run:

.\SecureFASTSearchConnector.ps1 –certPath “folder location\FASTSearchCert.pfx” –ssaName “name of your content SSA” –userName “domain\username”

FAST topology

On each of the remaining SharePoint server configured as a Crawl component you will have to:

  1. Import the FASTSearchCert.pfx certificate in the certificate store under Certificates(Local Computer)\Personal.  Follow this guide to start the Certificates snap-in for the local computer and then this one to import the certificate.
  2. Run the following:
.\SecureFASTSearchConnector.ps1 -ssaName "name of your content SSA" -username "domain\username"

This command will return the thumbprint of the available certificates and a prompt asking whether you want to use the suggested certificate. In my case I had to do a restart of the server running the Admin component and the FAST Search servers before this command being successful.

The ssaName is the name of the content Search Service Application you use and the userName is the  user used on your SharePoint servers to run the “SharePoint Server Search 14” service.

You will receive a confirmation that the FAST Search connector successfully connected to the FAST servers. For example in my case this was not true on each server but everything worked fine even without confirmation.

Make sure you don’t forget any of the servers, if you miss one of them your crawls will start but will not finish, making this very hard to troubleshoot.

Use ULS logs to troubleshoot if you have any problem.

Advertisements

SharePoint 2010 Applying April 2013 Cumulative Update, procedure and problems

This will be a very short post, last time when I applied CU for SharePoint 2010 was in June 2012 and I almost forgot what the procedure was. Of course every time there is a new error to keep you connected, I’ll get into this later.

Step one:

Download SharePoint 2010 CU from here

Step two:

Make sure you have a very good backup, don’t play!

Step three:

Install the CU binaries to your server hosting Central Administration.

Step four:

Install the CU binaries to the remaining servers in your farm.

Step five:

In my case when the CU finished installing on the server hosting the Central Administration required a restart. I did the restart after all the other servers in the farm had the binaries installed.

Step six:

Run the SharePoint Products 2010 Configuration Wizard on the server hosting the CA.

First problem was at this step, for whatever reason running this wizard in GUI did not work, I switched to running it from Power Shell.

Run the following command: PSCONFIG.EXE -cmd upgrade -inplace b2b -wait -force

stsadm upgrade

Step seven:

Run the same command on the remaining servers in the farm but only after you were successful on the server hosting the CA.

This is when I ran into the second problem. The first attempt to run psconfig on the second application server threw an error about

05/09/2013 18:33:52 12 WRN Unable to create a Service Connection Point in the current Active Directory domain. Verify that the SharePoint container exists in the current domain and that you have rights to write to it.
Microsoft.SharePoint.SPException: The object LDAP://CN=Microsoft SharePoint Products,CN=System,DC=yourdomain,DC=com doesn’t exist in the directory.
at Microsoft.SharePoint.Administration.SPServiceConnectionPoint.Ensure(String serviceBindingInformation)
at Microsoft.SharePoint.PostSetupConfiguration.UpgradeTask.Run()

This is when I did a restart of this server, but I’m not sure it helps, I do not think you should do it.

On my second attempt to run PSCONFIG on this server I received this error, and then again:

Exception: Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException: An update conflict has occurred, and you must re-try this action. The object SPUpgradeSession Name=Upgrade-20110604-023550-824 was updated by Topgear\administrator, in the PSCONFIG (4272) process, on machine SharePoint2010. View the tracing log for more information about the conflict

I used the information on this blog to reset the value of the “command-line-upgrade-running” property from Power Shell:

stsadm -o setproperty -pn command-line-upgrade-running -pv Yes

setproperty

You need to reset this property in case an upgrade stopped responding and there is no upgrade running according to TechNet (see the example lower in the TechNet page) and then I went ahead with the PSCONFIG command from above.

Keep in mind it takes a long time to apply this cumulative update, I think I’ve spent about eight  hours doing the upgrade on the staging and production farm, each with two application and two web servers.

If you still did not solve this, read the logs again, the answers are in there!

“Microsoft SharePoint Foundation Web Application” service stuck on starting/ “Microsoft SharePoint Foundation Web Application” service stuck on stopping

I came across this issue today when I had to provision a new web application, I tried to provision the web application from the central administration server where I don’t have the Microsoft SharePoint Foundation Web Application service running, for a good reason in my opinion, this is the first service I will stop on an application server, because it is the primary role for a web server.  In case you did not notice, not running this service on the server used to provision the new application will not give you the option to select an existing application pool, it gives you only the option to select an application pool running in the local IIS.

But if you are like me and decide to provision this web application from the CA server then probably you will start the Microsoft SharePoint Foundation Web Application service on this machine and wonder why it is stuck on starting, you wait 10-15-20 minutes and you decide that there is something wrong with your server, start to panic, search for solutions in places where yo don’t have to search etc…

ADVICE: WAIT

If you started the Microsoft SharePoint Foundation Web Application service on your CA server you have to know that in my case it took almost one hour to start, during this time you can check the progress using he IIS console, you will notice web sites and application pools start to appear one by one. Just keep calm and wait.

In case you do not see any progress in IIS or in the ULS logs then you can start the deployment of this service using this STSADM command:

stsadm -o provisionservice -action start -servicetype spwebservice

stsadm

But keep in mind that you have to wait a lot depending on the number of web applications you have.

Work in progress…..

Phase two

Everything worked fine after the  stsadm command was used. A second issue appeared as soon as I tried to stop the Microsoft SharePoint Foundation Web Application service after I finished my work. Doing so from the CA caused the service to be  stuck on stopping. Again use the same command but this time with the switch -stop instead of start, finished in one minute and then I issued iisreset /noforce.  Everything is back to normal now.

stsadm -o provisionservice -action stop -servicetype spwebservice

Restoring a Windows 8 hard drive image to a RAID 0 failed

Just want to let everyone know that if you plan on restoring an image of Windows 8 taken on a hard drive and you want to restore it to a RAID array (I tested with Stripe, RAID0) you will encounter problems when booting. In my case the image was taken from an SSD Samsung 830 and restored to a RAID0 array made by two Samsung 840 SSDs. Don’t have any explanation and instead of searching for a solution I installed everything from scratch.  Strangely enough the RAID array can be accessed without any problem by Windows 8, the partitions are all there but the problem happens during the boot, a strange Windows error with a smiley, something similar to the attachment. I used Macrium Reflect for imaging and restoring.

Plan it in case you want to do it, you might have the same problem!

Image

SCOM 2012 SP1 error “Failed to run a process or script” during SQL 2008 Discovery

I’ve been searching for a solution to this problem for a few days already. I had a test SCOM 2012 server to monitor SQL, SharePoint, Clusters  and then after I finished my testing I installed a new SCOM 2012 SP1 server in order to monitor my production farm.

I uninstalled the old agents and redeployed new ones from the SCOM 2012 SP1 console. Everything looks good except for one ‘Database Engine’ which is not discovered for one of the clusters SQL instances. Every other SQL role is discovered and monitored correctly on all the instances except for the Database Services on one of the instances. The SQL version used is SQL 2008 R2 and the cluster is running on Server 2008 R2. I run three SQL instances on this cluster.

SCOM will display a warning:

The process started at 11:50:36 failed to create System.Discovery.Data, no errors detected in the output. The process exited with 0

Command executed: “C:\Windows\system32\cscript.exe” /nologo “DiscoverSQL2008DBEngineDiscovery.vbs” {32FBB1E4-C6D1-0517-2F47-3DDA67D46A3B} {B4F2B3BC-D834-17F2-CE0A-2F87B502F7BE} SQL2K8-DB2.domain.com SQL2K8-DB2.domain.com SQL2K8-DB2.domain.com “Exclude:”
Working Directory: C:\Program Files\System Center Operations Manager\Agent\Health Service State\Monitoring Host Temporary Files 11\14968\

One or more workflows were affected by this. 

Workflow name: Microsoft.SQLServer.2008.DBEngineDiscoveryRule.Server

Instance name: SQL2K8-DB2.domain.com

Instance ID: {B4F2B3BC-D834-17F2-CE0A-2F87B502F7BE}

Management group: SP_prod

Warning in SCOM alersts

I tried to run the command manually but I received an error that the script does not exist in this location. I found a blog which is not really giving you everything but it gets you closer to what you should do.

Fist in order to correctly register SQL  browse to “C:\Program Files (x86)\Microsoft SQL Server\100\Shared” with a command prompt.

Then run mofcomp sqlmgmproviderxpsp2up.mof

Perform this step on all of the SQL cluster nodes.

Credits go to Carl.T here http://qa.social.technet.microsoft.com/Forums/en-US/operationsmanagermgmtpacks/thread/01eff618-1087-4b6a-9d3f-9f1402ddf3f4 for this very nice tip.

As soon as I ran this command one of the SQL instances were displayed.

In order to run the command from the SCOM error you will need to:

1) search the installation folder of the agent on the SQL nodes for a file called:

DiscoverSQL2008DBEngineDiscovery.vbs

2) if you don’t find the script, which was my case, you need to:

2.1) Go to your SCOM server and export the Microsoft.SQLServer.2008.Discovery monitoring pack to your local drive as XML. In this file we will find the script which is ran by SCOM in order to discover SQL 2008.

The monitoring pack most probably is sealed and the GUI will not allow you to export so you will need to run in powershell:

Make sure you add the Operations manager snapin to your session before running this command.
PS C:\Windows\system32> get-scommanagementpack -name “Microsoft.SQLServer.2008.Discovery” | export-managementpack -path “f:\backup”
If the name of the pack will change in the future, please use:
PS C:\Windows\system32> get-scommanagementpack | export-managementpack -path “f:\backup”
This will export all your management packs and you can then choose the one for SQL 2008 Discovery.
Open the exported XML and search for the name of the script DiscoverSQL2008DBEngineDiscovery.vbs and copy the long script body to a new text document and save it with .vbs extension.
Copy the resulted script  to c:\temp on the SQL node.
Script in XML
Try to run in an elevated command prompt the same command you found in the SCOM warning, in my case it looked like:
“C:\Windows\system32\cscript.exe” /nologo “c:\temp\DiscoverSQL2008DBEngineDiscovery.vbs” {32FBB1E4-C6D1-0517-2F47-3DDA67D46A3B} {B4F2B3BC-D834-17F2-CE0A-2F87B502F7BE} SQL2K8-DB2.domain.com SQL2K8-DB2.domain.com SQL2K8-DB2.domain.com “Exclude:” Working Directory: C:\Program Files\System Center Operations Manager\Agent\Health Service State\Monitoring Host Temporary Files 11\14968\
Kevin Holman explains this here: http://blogs.technet.com/b/kevinholman/archive/2010/03/09/basic-troubleshooting-of-discovery-scripts.aspx  but he did not specify that you need to give the full path to the .vbs script. He also explains that  SQL2K8-DB2.domain.com  needs to be the name (FQDN) of your virtual cluster SQL server. I did not understand it, it is the cluster FQDN or the SQL instance FQDN or the Cluster node FQDN but I looked for the failed command in SCOM  and used the same Cluster node name and I ran the command on the same node also.
He also explains that the GUIDs used are dummy ones during the discovery.
Now after performing those two steps, registering again SQL on each node  and running the discovery script, two of the missing SQL 2008 instances are already displayed and monitored, probably the third one will also be discovered, I will update this post if it does not.

Error when trying to add a new item into an imported list(SharePoint 2007 to SharePoint 2010)

We have the following case:

A list created in SharePoint 2007 is imported to SharePoint 2010.

The exact method to export a list from SharePoint 2007 to 2010 was the following:

The content database containing the list in 2007 was backed up and then restored and attached to a SharePoint 2010 farm in a new web application. The content database created during the process of creating the web application is removed from the web application in Central Administration.

In order to upgrade and add the old content database to your web app you will use the following power shell command:

Mount-SPContentDatabase “MyDatabase” -DatabaseServer “MyServer” -WebApplication http://sitename

This will not only attach it to your desired web application but it will also upgrade it for SharePoint 2010. In my case it was with a lot of errors but it worked.

The errors are related to solutions which are not present in the 2010 farm but are used in the content database(web app). I was able to browse the sites in this content database using a different name.

Some columns cannot be used in 2010 due to the fact that those are custom types, but before I backup up the content DB in 2007, I created clones of the custom columns and copied the information as text, for example for the “block” column I created “block_new” to keep the information as text. I used the datasheet view to copy the information, for a small list it might work fine but for very large lists I am not sure if this is a solution.  After the attach I deleted the columns which were not usable and edited the name for the clones to match the old column names.

I used Central administration to export the list to a .cmp file which was imported in a production farm.

Everything worked fine except that when I tried to add a new item to the list I got the following error:

“Web Part Error: A Web Part or Web Form Control on this Page cannot be displayed or imported. The type could not be found or it is not registered as safe.”

I started to believe that I still have problems because of the import and the customization from 2007, because I don’t really know the old farm, have no idea what was used there and started to look on Google.

The problem is a lot of people suggest a different kind of solution. In my case it was very simple, I used SharePoint Designer to connect to the site, then opened the list and created a new aspx form for new items and selected this as default. I deleted the old one after that. In my case this solved the problem and I think it is related to the fact that we used a custom form used for new items.