Share This Post

Troubleshooting On-premises Deployments – Part 1

This article provides Troubleshooting On-premises Deployments information of Microsoft Dynamics 365 for Finance and Operations. When you choose an on-premises deployment type, the system requirements, hardware sizing and functionality differ from a cloud deployment.

With this deployment option, application servers and the Microsoft SQL Server database will run in the customers data center. Customers and partners will utilize Microsoft Dynamics Life Cycle Services (LCS) to manage their on-premises deployments.

Troubleshooting On-premises Deployments Steps

1. Access Service Fabric Explorer

You can access Service Fabric Explorer in a web browser by using the default address, https://sf.d365ffo.onprem.contoso.com:19080.

To verify the address, note the value that was used in the “Create DNS zones and add A records” section of the appropriate setup and deployment for your environment.

You can access the site only if the client certificate is in cert:\CurrentUser\My on the machine that you are accessing the site on. (In Certificate Manger go to Certificates – Current User > Personal > Certificates.) When you access the site, select the client certificate when you are prompted.

2. Monitoring the Deployment

Identifying the Primary Orchestrator

To regulate the machine that is the primary instance for stateful services such as a local agent, in Service Fabric Explorer, expand Cluster > Applications > <intended application example> LocalAgentType > fabric:/LocalAgent/OrchestrationService > (GUID).

The primary node is shown for Troubleshooting On-premises Deployments. For stateless services or the existing applications, you must go through all the nodes.

Note the following points:

  • OrchestrationService orchestrates the deployment and servicing actions for D365 F&O.
  • ArtifactsManager downloads files from the MS Azure cloud storage to the local agent file share. It also unzips the files into the required format.
Review the Orchestrator Event Logs

From the primary OrchestrationService orchestrator machine, in Event Viewer, go to Applications and Services Logs > Microsoft > Dynamics > AX-LocalAgent.

The following modules must be installed:

  • Common
  • Reporting Services
  • AOS
  • Financial Reporting

The following commands must be run:

  • Setup
  • Dvt – This command runs a deployment verification test.
  • Cleanup – This command is used to service and delete an environment.

The following folders contain additional information:

  • AX-SetupModuleEvents
  • AX-SetupInfrastructureEvents
  • AX-BridgeService

To review Microsoft Dynamics entries in Event Viewer, follow these steps.

  • In Event Viewer, right-click Custom Views and then select Create Custom View.
  • In the Event logs field, select Dynamics.

Service Fabric Explorer

Note the state of the cluster, application and nodes.

Error: “Partition is below target replica or instance count”

You might receive the following error during Troubleshooting On-premises Deployments:

Partition is below target replica or instance count

This error isn’t a root error. It indicates that the status of each node isn’t ready. For AXSFType (AOS), the status might still be InBuild.

On the machines that are related to the error message, use Event Viewer to view the latest activity.

AXSFType

If a status of InBuild is shown for AXSFType (AOS), review the DB Sync status and other events from Application Object Server (AOS) machines.

To diagnose errors, use Event Viewer to review the following event logs:

  • Applications and Services Logs > Microsoft > Dynamics > AX-DatabaseSynchronize
  • Custom Views > Administrative Events

Error: “‘ExtractInstallerService failed to extract’ C:\Users\dynuser.CONTOSO\AppData\Local\Temp\1blssblh.w0n\FabricInstallerService.Code\FabricClient.dll”

You might receive the following error during Troubleshooting On-premises Deployments:

ExtractInstallerService failed to extract C:\Users\dynuser.CONTOSO\AppData\Local\Temp\1blssblh.w0n\FabricInstallerService.Code\FabricClient.dll.

If you receive this error, download the latest version of Azure Service Fabric. Note that the user name and path in the error message vary, depending on your environment.

Service Fabric logs

You can find more details about Service Fabric applications in the log files at C:\ProgramData\SF\<OrchestratorMachineName>\Fabric\work\Applications\LocalAgentType_App<N>\log.

Life Cycle Services

Note the current deployment status for the environment in Microsoft Dynamics Life Cycle Services (LCS).

3. A Time-out Error Occurs when a Service Fabric Cluster is Created

Run Test-D365FOConfiguration.ps1 as noted in the “Set up a standalone Service Fabric cluster” section of the appropriate setup and deployment for your environment. Note any errors.

Be sure to complete these steps:

  • Verify that the Service Fabric Server client certificate exists in the LocalMachine store on all Service Fabric nodes.
  • Verify that the Service Fabric Server certificate has the access control list (ACL) for Network Service on all Service Fabric nodes.
  • Review the antivirus exclusions that are noted in Environment setup.

4. A Time-out Error Occurs while you are Waiting for Installer Service to be Completed for Machine x.x.x.x

Only one node type is supported for each Internet Protocol (IP) address (that is, for each machine). Check whether the nodes are being reused on the same machine. For example, AOS and ORCH must not be on the same machine, and ConfigTemplate.xml must be correctly defined.

5. Remove a Specific Application

We recommend that you use LCS to remove or clean up deployments. Although, you can also utilize Service Fabric Explorer to remove an application as you require.

In Service Fabric Explorer, go to Application node > Applications > MonitoringAgentAppType-Agent. Choose the ellipsis button () next to fabric:/Agent-Monitoring and delete the application. Enter the full name of the application to verify the deletion of the application.

You can also remove MonitoringAgentAppType-Agent by selecting the ellipsis button and then selecting Unprovision Type. Enter the full name to confirm the removal of the application.

6. Remove all Applications from Service Fabric

The following script removes and unprovisions all Service Fabric applications except LocalAgent and the monitoring agent for LocalAgent. Execute this script on an orchestrator virtual machine.

$applicationNamesToIgnore = @(‘fabric:/LocalAgent’, ‘fabric:/Agent-Monitoring’)
$applicationTypeNamesToIgnore = @(‘MonitoringAgentAppType-Agent’, ‘LocalAgentType’)

Get-ServiceFabricApplication | `
Where-Object { $_.ApplicationName -notin $applicationNamesToIgnore } | `
Remove-ServiceFabricApplication -Force

Get-ServiceFabricApplicationType | `
Where-Object { $_.ApplicationTypeName -notin $applicationTypeNamesToIgnore } | `
Unregister-ServiceFabricApplicationType -Force

7. Remove Service Fabric

To completely remove the Service Fabric cluster, follow these steps.

  • Run the following command.

.\RemoveServiceFabricCluster.ps1 -ClusterConfigFilePath .\ClusterConfig.json

  • If an error occurs, remove a specific node on the cluster by using the CleanFabric.ps1 command. You can see this command in C:\Program Files\Microsoft Service Fabric\bin\fabric\fabric.code.
  • Remove the C:\ProgramData\SF folder, if you are using the default location. Or, remove the specified folder.

If you receive an “Access denied” error, restart Microsoft Windows PowerShell or the machine.

8. Clean up an Existing Environment and Redeploy

To clean up an existing environment and redeploy, follow these steps.

  • In LCS, open the project and then, in the Environments section, remove the deployment.

The applications should start to fade from Service Fabric Explorer in the environment. This process will take one to two minutes.

  • Access the orchestrator machine that contains LocalAgentCLI.exe, and follow these steps:
      • Run the local agent cleanup.

.\LocalAgentCLI.exe Cleanup ‘<path of localagent-config.json>’

      • Remove Service Fabric.

.\RemoveServiceFabricCluster.ps1 -ClusterConfigFilePath ‘<path of ClusterConfig.json>’

      • If any nodes fail, run the CleanFabric.ps1 command. You can find this command in C:\Program Files\Microsoft Service Fabric\bin\fabric\fabric.code.
      • Remove the C:\ProgramData\SF\ folder on all Service Fabric nodes.

If you receive an “Access denied” error, restart the machine, and try again.

  • Remove or update certificates as required for Troubleshooting On-premises Deployments.

Remove old certificates from all AOS, BI, ORCH, and DC nodes.

      • The certificates exist in the following certificate stores: Cert:\CurrentUser\My\, Cert:\LocalMachine\My, and Cert:\LocalMachine\Root.
      • If the setup of MS SQL Server will be altered, delete the SQL Server certificates.
      • If the settings for Active Directory Federation Services (AD FS) will be modified, remove the AD FS certificate.
  • Update the following configuration files as required:
      • ConfigTemplate.xml
      • ClusterConfig.json
  • In LCS, open the project, and update the LCS on-premises connector as required.
      • Re-create the LCS on-premises connector for the environment, or edit the settings of an existing connector. To obtain easy-to-copy values for LCS, use the .\Get-AgentConfiguration.ps1 script.
      • Download the latest local agent configuration, localagent-config.json.
  • Deploy again by following the instructions in the appropriate setup and deployment for the environment.

9. Find the Local Agent Values that are Used

You can find local agent values in Service Fabric Explorer. Go to Cluster > Applications > LocalAgentType > fabric:/LocalAgent and then select Details.

Alternatively, run the following Windows PowerShell command.

.\Get-AgentConfiguration.ps1 -ConfigurationFilePath .\ConfigTemplate.xml

10. Install, Upgrade or Uninstall a Local Agent

You can also use the following upgrade and un-installation commands:

LocalAgentCLI.exe Install <path of localagent-config.json>
LocalAgentCLI.exe Cleanup <path of localagent-config.json>

The Cleanup command doesn’t remove any files that were put in the file share. The file share can be reused. For more information on Troubleshooting On-premises Deployments, please contact us. Please refer to this article for the continuation of Microsoft Dynamics 365 On-premises Deployments Troubleshooting: https://msdynamics.net/dynamics-deployments/troubleshooting-on-premises-deployments-part-2/

Share This Post

Leave a Reply

avatar
  Subscribe  
Notify of
Skip to toolbar