We have set up automated way of generation and installation of Sitecore packages via Azure Devops. This is happening during execution of Build Pipelines.
We are using Sitecore CLI – Sitecore Content Serialization (SCS) – to generate and install these packages.

This command is the one that we are using for package installation:

dotnet sitecore ser pkg install

Problem Definition

For some reason, on one of our environments it started throwing these exception on one of the generated packages:

One or more errors occurred. (One or more errors occurred. (A task was canceled.)) (One or more errors occurred. (A task was canceled.)) (One or more errors occurred. (A task was canceled.)) (One or more errors occurred. (A task was canceled.)) (One or more errors occurred. (A task was canceled.)) (One or more errors occurred. (A task was canceled.))
   at Sitecore.DevEx.Serialization.Client.Datasources.Sc.Query.SitecoreTreeDataStore.CreateParallelQuery[T](String parallelQueryFragment, TreeScope scope, ItemSpec[] specs, CancellationToken cancellationToken)
   at Sitecore.DevEx.Serialization.Client.Datasources.Sc.Query.SitecoreTreeDataStore.GetTreeNodesInternal(IEnumerable`1 itemSpecs, TreeScope scope, CancellationToken cancellationToken)
   at Sitecore.DevEx.Serialization.Client.Datasources.Sc.Query.SitecoreTreeDataStore.Initialize(CancellationToken cancellationToken)
   at Sitecore.DevEx.Serialization.Client.TreeSyncOperation.Initialize(CancellationToken cancellationToken)
   at Sitecore.DevEx.Serialization.Client.TreeSyncOperationExecutor.ExecuteTreeSyncOperation(TreeSyncOperation operation, Boolean whatIf, Boolean performValidation, Boolean force, SyncDirection direction, CancellationToken cancellationToken)
   at Sitecore.DevEx.Extensibility.Serialization.Tasks.PackageInstallTask.Execute(PackageInstallTaskOptions options)
   at Sitecore.DevEx.Extensibility.Serialization.Commands.PackageInstallCommand.Handle(PackageInstallTask task, PackageInstallArgs args)
   at Sitecore.DevEx.Extensibility.Serialization.Commands.PackageInstallCommand.Handle(PackageInstallTask task, PackageInstallArgs args)
   at Sitecore.Devex.Client.Cli.Extensibility.Subcommands.SubcommandBase`2.HandleInternal(TArgs args) in C:\BA\ca7111d945a16af4\src\Sitecore.Devex.Client.Cli.Extensibility\Subcommands\SubcommandBase.cs:line 84
Execution was cancelled. This occurs when the task is killed or requests timeout.

We were using Sitecore CLI 5.1.25 with these plugins:

  • Sitecore.DevEx.Extensibility.Serialization v.5.1.25
  • Sitecore.DevEx.Extensibility.Publishing v.5.1.25
  • Sitecore.DevEx.Extensibility.Indexing v.5.1.25
  • Sitecore.DevEx.Extensibility.ResourcePackage v.5.1.25
  • Sitecore.DevEx.Extensibility.Database v.5.1.25

When I have installed the package from local machine, it installs it without this error. Very same package on same higher environment. I was wondering why’s that…

I have reached out to Sitecore Community via Slack in #sitecore-dev-collection channel and I have also raised a question on Sitecore Stack Exchange. At the end I have received only one suggestion by Maarten Willebrands.

He was suggesting to increase apiClientTimeoutInMinutes parameter in sitecore.json definition:

{
  "$schema": "./.sitecore/schemas/RootConfigurationFile.schema.json",
  "modules": [
    ...
  ],
  "plugins": [
    "Sitecore.DevEx.Extensibility.Serialization@5.1.25",
    "Sitecore.DevEx.Extensibility.Publishing@5.1.25",
    "Sitecore.DevEx.Extensibility.Indexing@5.1.25",
    "Sitecore.DevEx.Extensibility.ResourcePackage@5.1.25",
    "Sitecore.DevEx.Extensibility.Database@5.1.25"
  ],
  "serialization": {
    ...
  },
  "settings": {
    "apiClientTimeoutInMinutes": 30,
    "telemetryEnabled": false,
    "versionComparingEnabled": true
  }
}

Unfortunatelly, It did not help. See the solution below.

Solution

I have then attempted to install the same package from my local instance connected to that higher environment and it was successful.

I have then used Fiddler Everywhere to sniff all traffic and requests done during package installation and found the root cause of time outs.

Sitecore CLI is doing requests to GraphQL /api/management endpoint based on your json definitions:

{
   "namespace": "Chorpo.Feature.Blog.Content",
   "references": [ "Chorpo.Feature.Blog" ],
   "items": {
      "includes": [
           {
               "name": "points",
               "path": "/sitecore/content/Internet/Global Data/Modules/Points",
               "scope": "SingleItem",
               "allowedPushOperations": "CreateOnly"
           },
           {
               "name": "globalData",
               "path": "/sitecore/content/Internet/Settings/Get Started",
               "allowedPushOperations": "CreateOnly"
           },
           {
               "name": "Secondary Folder",
               "path": "/sitecore/content/Internet/Global Data/Modules/Secondary",
               "scope": "SingleItem",
               "allowedPushOperations": "CreateOnly"
           },
           {
               "name": "Primary Folder",
               "path": "/sitecore/content/Internet/Global Data/Modules/Primary",
               "scope": "SingleItem",
               "allowedPushOperations": "CreateOnly"
           }
       ]
   }
}

Based on the paths defined it will generate requests and then based on scope and allowedPushedOperations it will generate delta of operations which need to happen consequently based on these settings…

One of these sniffed requests was particularly interesting as it took more than 80 seconds to respond and the response was 8 MB huge json…

I have determined that this request was coming from this definition:

{
   "name": "site",
   "path": "/sitecore/content/Internet/MarketingSite",
   "rules": [
        {
            "path": "/Data",
            "scope": "SingleItem",
            "allowedPushOperations": "CreateOnly"
        },
        {
            "path": "/Settings",
            "scope": "ItemAndDescendants",
            "allowedPushOperations": "CreateOnly"
        },
        {
            "path": "*",
            "scope": "ignored"
        }
    ]
}

Problem was the path and amount of items we have underneath. So first Sitecore CLI created request to get all descendants and then it was applying the rule (or it appeared to me, that this is happening). I have already raised this potential bug / feature improvement on how rules are applied to Sitecore Support. These rules should be part of graphql request IMO so amount of data which is coming back is reduced to only items that conform rules defined. Waiting for reply from Sitecore Support.

Meanwhile, I have rewritten the definition, not to use rules at all.

So this definition:

{
   "name": "site",
   "path": "/sitecore/content/Internet/MarketingSite",
   "rules": [
        {
            "path": "/Data",
            "scope": "SingleItem",
            "allowedPushOperations": "CreateOnly"
        },
        {
            "path": "/Settings",
            "scope": "ItemAndDescendants",
            "allowedPushOperations": "CreateOnly"
        },
        {
            "path": "*",
            "scope": "ignored"
        }
    ]
}

was rewritten to these three definitions at the end:

    {
      "name": "site",
      "path": "/sitecore/content/Internet/MarketingSite",
      "scope": "SingleItem",
      "allowedPushOperations": "CreateOnly"
    },
    {
      "name": "siteData",
      "path": "/sitecore/content/Internet/MarketingSite/Data",
      "scope": "SingleItem",
      "allowedPushOperations": "CreateOnly"
    },
    {
      "name": "siteSettings",
      "path": "/sitecore/content/Internet/MarketingSite/Settings",
      "scope": "ItemAndDescendants",
      "allowedPushOperations": "CreateOnly"
    }

These three new definitions are not only solving problem with timeout but they are also speeding package installation significantly.

These three new definitions are valid as we had ignored rule in original one. Mind this when rewriting your definitions!

This elegant solution can be used everywhere to solve current performance issue with rules applied on Sitecore CLI / SCS module definitions.