Cannot start Sitecore xConnect Search Indexer Windows Service

From certain point in time I couldn’t start “Sitecore XConnect Search Indexer” Windows Service anymore. I could see just below exceptions in logs whenever it tried to start itself automatically during Windows start or when I tried to start it manually from Services.

I was running Sitecore 9 Update 2 with xConnect up and running.

Problem definition

I could see these kind of exceptions in AppDomain log:

2018-07-27 15:16:23.986 +02:00 [Information] Starting Xdb Indexer App Domain, Machine: WIN-G4E0D3RI00V, InstanceName: XConnectIndexer_AppDomain

2018-07-27 15:16:26.773 +02:00 [Information] SystemPerformanceCounters Constructor, Instance:XConnectIndexer, Path: App_Data\Diagnostics, CounterFilePattern: *.json

2018-07-27 15:16:27.976 +02:00 [Error] Exception occurred during instantiating of IndexerHost.
System.InvalidOperationException: Unable to resolve service for type 'Sitecore.Xdb.Collection.Indexing.IIndexInitializer' while attempting to activate 'Sitecore.Xdb.Collection.Indexing.SingleThreadedIndexer'.
   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
   at Microsoft.Extensions.Configuration.ConfigurationServiceType.c__DisplayClass24_0.b__0(IServiceProvider sp)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.ScopedCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
   at Sitecore.XConnect.Search.Indexer.IndexerHost..ctor(IConfigurationRoot config)
   at Sitecore.XConnect.Search.Indexer.IndexerAppDomainHost.IndexerHostAppDomainBridge..ctor()

And I could see these kind of exceptions in it’s AppHost log:

2018-07-27 15:16:22.027 +02:00 [Information] Xdb Indexer Application Start, Machine: WIN-G4E0D3RI00V, InstanceName: XConnectIndexer_AppHost

2018-07-27 15:16:28.068 +02:00 [Error] Failed to start the indexer host
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Unable to resolve service for type 'Sitecore.Xdb.Collection.Indexing.IIndexInitializer' while attempting to activate 'Sitecore.Xdb.Collection.Indexing.SingleThreadedIndexer'.
   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
   at Microsoft.Extensions.Configuration.ConfigurationServiceType.c__DisplayClass24_0.b__0(IServiceProvider sp)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.ScopedCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
   at Sitecore.XConnect.Search.Indexer.IndexerHost..ctor(IConfigurationRoot config)
   at Sitecore.XConnect.Search.Indexer.IndexerAppDomainHost.IndexerHostAppDomainBridge..ctor()
   --- End of inner exception stack trace ---
   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName)
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName)
   at Sitecore.XConnect.Search.Indexer.IndexerAppDomainHost.CreateAppDomainAndIndexer()
   at Sitecore.XConnect.Search.Indexer.IndexerAppDomainHost.Start()
   at Sitecore.XConnect.Search.Indexer.Service.OnStart(String[] args)

2018-07-27 15:16:28.083 +02:00 [Information] Xdb Indexer Application End, Machine: WIN-G4E0D3RI00V, InstanceName: XConnectIndexer_AppHost

“Sitecore Marketing Automation Engine” Windows Service was working as expected. But “Sitecore XConnect Search Indexer” service always failed to start and status was always blank:

Solution

I have remembered at the end that I was setting indexing anonymous contacts to true in C:\inetpub\wwwroot\sc9u2.xconnect\App_data\jobs\continuous\IndexWorker\App_data\Config\Sitecore\SearchIndexer\sc.XConnect.IndexerSettings.xml file based on this article: https://doc.sitecore.net/developers/xp/xconnect/xconnect-search-indexer/enable-anonymous-contact-indexing.html. Path can vary on your end based on your installation folder.

Config taken from current version of article:

<Settings>
<Sitecore>
    <XConnect>
    <!-- SearchIndexer role requires Collection and CollectionSearch role services -->
    <SearchIndexer>
        <Services>
        <IndexerSettings>
            <Type>Sitecore.Xdb.Collection.Indexing.IndexerSettings, Sitecore.Xdb.Collection</Type>
            <LifeTime>Singleton</LifeTime>
            <Options>
            <IndexPIISensitiveData>false</IndexPIISensitiveData>
            <IndexAnonymousContactData>false</IndexAnonymousContactData>
            </Options>
        </IndexerSettings>
        <IIndexer>
            <Type>Sitecore.Xdb.Collection.Indexing.DecoratedIndexer, Sitecore.Xdb.Collection</Type>
            <As>Sitecore.Xdb.Collection.Indexing.IIndexer, Sitecore.Xdb.Collection</As>
            <LifeTime>Singleton</LifeTime>
        </IIndexer>
        <IIndexRebuildFlow>
            <Type>Sitecore.Xdb.Collection.Indexing.IndexRebuildFlow, Sitecore.Xdb.Collection</Type>
            <As>Sitecore.Xdb.Collection.Indexing.IIndexRebuildFlow, Sitecore.Xdb.Collection</As>
            <LifeTime>Singleton</LifeTime>
            <Options>
            <IncomingDataLagOnCompletion>0.00:00:05</IncomingDataLagOnCompletion>
            <!--Enable ParallelizationDegree setting to override default value which is (processorCount*4)-->
            <!--<ParallelizationDegree>16</ParallelizationDegree>-->
            <BatchSize>1000</BatchSize>
            </Options>
        </IIndexRebuildFlow>
        </Services>
    </SearchIndexer>
    </XConnect>
</Sitecore>
</Settings>

For God knows which reason I have copied the whole config from that article. Problem was that I was already using Sitecore 9 Update 2 and the article was made for Initial Release probably. Since then config was slightly changed and in Update 2 you need to use these settings instead:

<Settings>
  <Sitecore>
    <XConnect>
      <!-- SearchIndexer role requires Collection and CollectionSearch role services -->
      <SearchIndexer>
        <Services>
          <IndexerSettings>
            <Type>Sitecore.Xdb.Collection.Indexing.IndexerSettings, Sitecore.Xdb.Collection</Type>
            <LifeTime>Singleton</LifeTime>
            <Options>
                <!-- Indexer will split change set on chunks to improve memory consumption. Setting this option to 0, a negative value or removing the element completely, results in no splitting.-->
              <SplitRecordsThreshold>25000</SplitRecordsThreshold>
              <IndexPIISensitiveData>false</IndexPIISensitiveData>
              <IndexAnonymousContactData>true</IndexAnonymousContactData>
            </Options>
          </IndexerSettings>
          <IIndexer>
            <Type>Sitecore.Xdb.Collection.Indexing.DecoratedIndexer, Sitecore.Xdb.Collection</Type>
            <As>Sitecore.Xdb.Collection.Indexing.IIndexer, Sitecore.Xdb.Collection</As>
            <LifeTime>Singleton</LifeTime>
          </IIndexer>
          <IIndexerInitializer>
            <Type>Sitecore.Xdb.Collection.Indexing.DefaultIndexInitializer, Sitecore.Xdb.Collection</Type>
            <As>Sitecore.Xdb.Collection.Indexing.IIndexInitializer, Sitecore.Xdb.Collection</As>
            <LifeTime>Singleton</LifeTime>
          </IIndexerInitializer>
          <IIndexRebuildFlow>
            <Type>Sitecore.Xdb.Collection.Indexing.IndexRebuildFlow, Sitecore.Xdb.Collection</Type>
            <As>Sitecore.Xdb.Collection.Indexing.IIndexRebuildFlow, Sitecore.Xdb.Collection</As>
            <LifeTime>Singleton</LifeTime>
            <Options>
              <IncomingDataLagOnCompletion>0.00:00:05</IncomingDataLagOnCompletion>
              <!--Enable ParallelizationDegree setting to override default value which is (processorCount)-->
              <!--<ParallelizationDegree>16</ParallelizationDegree>-->
              <BatchSize>5000</BatchSize>
            </Options>
          </IIndexRebuildFlow>
          <IIndexRebuilderCountersDecorator>
              <Type>Sitecore.XConnect.Configuration.ServiceDecorator`2[[Sitecore.Xdb.Collection.Indexing.IIndexRebuilder, Sitecore.Xdb.Collection], [Sitecore.Xdb.Collection.Indexing.IndexRebuilderCountersDecorator, Sitecore.Xdb.Collection]], Sitecore.XConnect.Configuration</Type>
              <LifeTime>Singleton</LifeTime>
          </IIndexRebuilderCountersDecorator>
        </Services>
      </SearchIndexer>
    </XConnect>
  </Sitecore>
</Settings>

Mind the two new services definitions for IIndexerInitializer and IIndexRebuilderCountersDecorator which are only present in Update 2 version of that xml. Especially IIndexerInitializer was crutial for solving this problem.

I have also changed BatchSize to 5000 as also this setting / value has been changed.

When I compared correct one from another instance with faulty one In WinMerge it looked like this:

On left settings from faulty instance, on right correct version for Update 2.

After applying these settings and saving the xml file, I just starting the “Sitecore XConnect Search Indexer” Windows service for that instance and everything started to work once again. Without any restart or what so ever.

Documentation is not wrong, it was simply my mistake that I have copied the whole file not only the part I needed.

Lessons learnt – Always copy only settings that you need, not the whole config file cause it might change between versions!

I was always doing this but for God knows which reason I have copied it all this time 🙁

Happy coding!