In SQL Server 2012 and 2014, AlwaysOn health diagnostics detect the health of the SQL Server process in several ways. However, no health detection is performed on the accessibility or viability of the databases defined in AlwaysOn availability groups. If the disk hosting the availability group database or log files is lost, AlwaysOn health does not detect this event, and application runtime errors accessing the database will ensue. Loss of the drive or errors accessing the drive that host your availability database data and log files will affect access to your production data.

As per the Flexible Failover Policy for Automatic Failover of an Availability Group (SQL Server)

Damaged databases and suspect databases are not detected by any failure-condition level. Therefore, a database that is damaged or suspect (whether due to a hardware failure, data corruption, or other issue) never triggers an automatic failover.

NOTE SQL Server 2016 enhances the AlwaysOn health diagnostics with database health detection. If AlwaysOn availability group database health detection has been selected for your availability group and an availability database transitions out of the ONLINE state (sys.databases.state_desc), the entire availability group will failover automatically if configured to do so. For more information see MSDN topic ‘CREATE AVAILABILITY GROUP’ and section on ‘DB_FAILOVER’

CREATE AVAILABILITY GROUP (Transact-SQL)

You can enhance the availability of your databases by detecting disk health. Add a generic script resource to your availability group resource group that does basic read or write tests against the drives hosting your availability group database data and log files. The following describes how to add a generic script resource as a dependency to your availability group resource to enhance AlwaysOn health detection with a basic disk health check.

Use a generic script resource to do basic health check on SQL Server data and log drives

Here is the high level description of the implementation of the generic script resource to detect availability group database drive health.

Add a generic script clustered resource to the availability group resource group. Make the availability group resource dependent on the generic resource script. That way, if the script resource reports IsAlive failure, Windows Cluster will attempt to restart or failover the availability group resource if configured to do so.

The generic script clustered resource IsAlive test creates a text file in the specified Data drive location and a text file in the specified Log drive location. If the file exists, the script will overwrite it.

Attached to this blog is a zipped file, GenericScript_SQLIsAlive.zip, containing

sqlisalive.vbs <- The generic script written in Visual Basic Script and implements Windows Cluster IsAlive

Add_SQLIsAliveScript.ps1 <- The PowerShell script which adds the generic script resource to your availability group resource group and sets the availability group dependency on your generic script resource

readme.txt <- Step by step instructions for implementing the generic script resource and additional instructions on how to test the script.

Implement the generic script resource

I Configure the generic script, sqlisalive.vbs

Data and Log Drive Paths Currently, the generic script is configured to test the following drive and paths: c:\temp\data and c:\temp\log. For testing purposes, create these paths on the local drive of each replica (primary and automatic failover partner secondary). Later, you can change them to the appropriate drive and paths where your database data and log files live.

    DataDriveFile=”c:\temp\data\ScriptFileData.txt”
    LogDriveFile=”c:\temp\log\ScriptFileLog.txt”

II Configure and execute the Powershell script to deploy the generic script to your availability group

NOTE This generic script only implements IsAlive which runs every 60 seconds.

1 Ensure your availability group has two replicas configured for automatic failover.

2 Copy the generic script file to an identical local storage location like ‘C:\temp\sqlisalive.vbs’ on both servers whose replicas are configured for automatic failover.

3 Create the paths for the health check, c:\temp\data and c:\temp\log.

4 The Add_SQLIsAliveScript.ps1 PowerShell script adds the generic script resource to your availability group and adds a dependency to your availability group resource, on the generic script resource. In the Add_SQLIsAliveScript.ps1, change the following variables.

     Set $ag to your availability group name
     Set $listener to your availability group listener name. If your availabiltiy group does not have a listener, set $listener to “”
     Set $scriptfilepath to the path and file name of your sqlisalive.vbs script

5 On the server hosting the primary replica, run the PowerShell script Add_SQLIsAliveScript.ps1 to add the generic script resource to your availability group resource group.

6 Launch Failover Cluster Manager and review the availability group resource group to confirm addition of the generic script resource to the availability group resource group. The generic script should appear and come online in the availability group resource group under the Resources tab.

7 Confirm that the dependency has been created in the availability group resource, on the generic script resource.

NOTES

The attached readme.txt file has instructions on how to test the script resource to ensure that it can failover your availability group resource.

Diagnose failure detection by generic script resource

Generate the cluster log for the node hosting the primary replica and search for ‘Data Drive Create File’ or ‘Log Drive Create File’ – to locate success or failure report of the generic script resource IsAlive:

00001b04.00002924::2015/12/07-17:16:41.798 INFO  [RES] Generic Script <sqlisalive>: Entering IsAlive
00001b04.00002924::2015/12/07-17:16:41.801 INFO  [RES] Generic Script <sqlisalive>: Data Drive Create File Succeeded
00001b04.00002924::2015/12/07-17:16:41.801 INFO  [RES] Generic Script <sqlisalive>: Log Drive Create File Succeeded

Or for example, you set the \Data folder to read only:

00001b04.00002924::2015/12/07-17:17:41.801 INFO  [RES] Generic Script <sqlisalive>: Entering IsAlive
00001b04.00002924::2015/12/07-17:17:41.804 INFO  [RES] Generic Script <sqlisalive>: Data Drive Create File Succeeded
00001b04.00002924::2015/12/07-17:17:41.804 INFO  [RES] Generic Script <sqlisalive>: Data Drive Create File Failed
00001b04.00002924::2015/12/07-17:17:41.804 INFO  [RES] Generic Script <sqlisalive>: Permission denied
00001b04.00002924::2015/12/07-17:17:41.804 ERR   [RES] Generic Script <sqlisalive>: ‘IsAlive’ script entry point returned FALSE.’
00001b04.00002924::2015/12/07-17:17:41.804 INFO  [RES] Generic Script <sqlisalive>: Return value of ‘IsAlive’ script entry point caused HRESULT to be set to 0x00000001.
00001b04.00002298::2015/12/07-17:17:41.804 WARN  [RHS] Resource sqlisalive IsAlive has indicated failure.

GenericScript_SQLIsAlive.zip

Configuring an availability group listener in Azure is much more complex than doing it on-premises, due to the some limitations of Azure networking. This topic helps you troubleshoot your availability group listener, whether your AlwaysOn Availability Groups deployment is in Azure only or in a hybrid IT environment using a site-to-site VPN.

Some steps in the listener configuration involve configuration of Azure itself, such as the load-balanced virtual machine (VM) endpoint and direct server return. However, Azure currently does not provide any tools to help you verify your configuration is working as expected. Therefore, you need a network analyzer to help you verify your configuration as well as troubleshoot any problem. This topic shows you how to use Microsoft Network Monitor to troubleshoot your availability group listener.

Availability Group Listener Configuration Summary

This section provides a list of configuration options to check while troubleshooting your availability group listener.

Verify Probe Pings in Availability Group Listener Configuration

Note: Azure Load Balancers don’t support the ping (ICMP) protocol.

To determine whether the probe port on the load-balanced endpoint is working properly on the Azure VMs, you use Network Monitor to flter your packet capture on the probe port.

When the load-balanced endpoint is configured properly, the Azure load balancer continuously pings each of the VMs to determine if it has the primary replica so that it can route client connections to the correct VM. If the VM is the primary replica, the cluster service is configured with the probe port and responds to the probe pings. This traffic can be seen in Network Monitor by performing packet capture with the following flter applied in the Display Filter pane:

TCP.DstPort == 59999 OR TCP.SrcPort == 59999

The first clause captures the incoming pings from the Azure load balancer and the second clause captures the reply by the primary replica. The screenshot below shows what it looks like when you do a packet capture on the primary replica in Azure.

The Con Id column shows you packets related to the same probe ping (shown in yellow). The response message from your VM acknowledges each ping message by incrementing the Seq value in the Ack value. You can also see the load balancer’s source IP address (shown in orange).

If you do not see the response messages from the VM, but SynReTransmit messages instead from the load balancer, the VM is not responding to the load balancer, which can mean that it is not the primary replica or that the probe ping are not working as expected.

Verify Listener Connectivity in Availability Group Listener Configuration

To determine whether the availability group listener is working properly on the Azure VMs, you use Network Monitor to filter your packet capture on the listener port.

When the client tries to connect to the availability group listener using the cloud service’s IP address and the listener port, Azure verifies that the connection port is the same as the one configured in the load-balanced endpoint and lets the TCP connection through to the primary replica, which has been responding to the probe pings. If the VM’s firewall has a corresponding rule, the client access point is configured properly, and the listener port is configured in your availability group, the availability group listener accepts the connection and the client can perform updates and queries. This traffic can be seen in Network Monitor by performing packet capture with the following filter applied in the Display Filter pane (assuming that the listener port is 10000):

TCP.DstPort == 10000 OR TCP.SrcPort == 10000

The screenshot below shows what it looks like when you successfully connect to the availability group listener in Azure and perform a simple query.

The Con Id column shows you packets related to the same client connection (shown in yellow). In the packets sent from the VM, you can see information regarding the client’s hostname, domain, and username (shown in red). You can also see the client’s IP address on the internet (shown in orange). In this case, it is a client VM in a different cloud service, so the IP address shown is the client VM’s cloud service IP address.

Troubleshoot Availability Group Listener Configuration in Azure

Tip: The Windows Ping.exe command does not work on the availability group listener in Azure. The load-balanced endpoint only accepts TCP connections, while Ping.exe uses ICMP.

When your AlwaysOn availability group is configured for automatic failover, you may find your availability group failed over, or if configured for manual failover manual, you may observe your availability group transition from PRIMARY role to the RESOLVING role, during which users cannot access the availability group databases.

Checking the SQL Server error log, you find that a lease timeout occurred, and SQL Server reports the error 19407 in the SQL Server error log:

2016-01-24 14:56:37.51 Server      Error: 19407, Severity: 16, State: 1.
2016-01-24 14:56:37.51 Server      The lease between availability group <ag> and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is failing over correctly, check the corresponding availability group resource in the Windows Server Failover Cluster.

Lease timeout is a very common health issue detected by AlwaysOn health detection. Additional diagnostics have been introduced to help diagnose the root cause of lease timeout events.

Lease Health Check is One Element of AlwaysOn Health

AlwaysOn availability groups implement a failover policy that monitors the health of SQL Server, and is scalable using the availability group’s FAILURE_CONDITION_LEVEL property. The lease is one of several health checks AlwaysOn health detection uses to monitor SQL Server.

The FAILURE_CONDITION_LEVEL can be set from 1 to 5 and as you increase the property value, additional checks on health are cumulative. The lease health is checked at FAILURE_CONDITION_LEVEL=1, therefore, because of the cumulative behavior, the lease is monitored at all FAILURE_CONDITION_LEVELs of an availability group, one through five, and cannot be disabled.

What kind of Health Issues Cause Lease Timeout?

In SQL Server, the dedicated thread for the lease mechanism is preemptive and outside normal SQLOS control running at higher priority. Therefore, outside of an event in which SQL Server’s dump diagnostics takes place, where the SQL Server process is frozen to produce a user dump of the process, a lease timeout indicates that there is a system wide issue that caused the lease timeout. Examples of this kind of system wide event which could result in lease timeout are sustained 100% cpu utilization or an unresponsive disk subsystem.

For more information on how the AlwaysOn lease monitors health check out:

How It Works: SQL Server AlwaysOn Lease Timeout

For more information on diagnosing AlwaysOn health detection including lease timeouts, check out:

Diagnose Unexpected Failover or Availability Group in RESOLVING State

Introducing New Lease Diagnostics

Lease timeout is the single most common issue detected by AlwaysOn health detection. Therefore, additional diagnostics have been introduced to help diagnose the root cause.

SQL Server Builds Introducing the New Lease Diagnostics

The lease timeout diagnostic improvements have been added to the following builds of SQL Server:

SQL Server 2012 Service Pack 3
SQL Server 2014 Service Pack 2 (upcoming)
SQL Server 2016 (upcoming)

 

IMPROVEMENT New SQL Server Messages Help Diagnose Lease Timeout

To provide additional insight, new messages have been added to SQL Server. Below is a list and explanation for what each provides.

Example 19419 As an example, I use a debugger to attach to SQL Server, which interrupts any servicing of threads in the SQL Server process until I resume the SQL Server process. When I resume SQL Server, the following is reported in the SQL Server error log:

2016-02-16 11:37:03.05 Server      Error: 19419, Severity: 16, State: 1.
2016-02-16 11:37:03.05 Server      Windows Server Failover Cluster did not receive a process event signal from SQL Server hosting availability group ‘ag’ within the lease timeout period.
2016-02-16 11:37:03.07 Server      Error: 19407, Severity: 16, State: 1.
2016-02-16 11:37:03.07 Server      The lease between availability group ‘ag’ and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is failing over correctly, check the corresponding availability group resource in the Windows Server Failover Cluster.
2016-02-16 11:37:03.07 Server      AlwaysOn: The local replica of availability group ‘ag’ is going offline because either the lease expired or lease renewal failed. This is an informational message only. No user action is required.
2016-02-16 11:37:03.07 Server      The state of the local availability replica in availability group ‘ag’ has changed from ‘PRIMARY_NORMAL’ to ‘RESOLVING_NORMAL’. The replica state changed because of either a startup, a failover, a communication issue, or a cluster error. For more information, see the availability group dashboard, SQL Server error log, Windows Server Failover Cluster management console or Windows Server Failover Cluster log.

The 19419 is raised because SQL Server was not responsive to Cluster. We also get the 19407 error which reports the lease timeout.

Example 19424 Here is the excess lease time message reported just prior to the availability group transitioning to PRIMARY role.

2016-02-16 11:31:54.76 Server      The lease worker of availability group ‘ag’ is now sleeping the excess lease time (164766 ms) supplied during online. This is an informational message only. No user action is required.
2016-02-16 11:31:54.80 Server      The state of the local availability replica in availability group ‘ag’ has changed from ‘PRIMARY_PENDING’ to ‘PRIMARY_NORMAL’. The replica state changed because of either a startup, a failover, a communication issue, or a cluster error. For more information, see the availability group dashboard, SQL Server error log, Windows Server Failover Cluster management console or Windows Server Failover Cluster log.

IMPROVEMENT Extended Event Reports Lease Timeout With Lease Stages

The availability_group_lease_expired and hadr_ag_lease_renewal and XEvents have been improved, with the addition of data points that provide more information on the condition of the lease.

availability_group_lease_expired	hadr_ag_lease_renewal

When availability_group_lease_expired XEvent is raised, expect that the current_time to be greater than the new_timeout, which is the current deadline that the lease must renew by. In the above example, calculating current_time – new_timeout, indicates that the lease expiration was reported 8,203 ms after the timeout deadline.

The hadr_ag_lease_renewal XEvent ill generally report RenewSucceeded state every five seconds.

Here is more information on the new data points for each of these XEvents:

Lease Stages and definitions

The following table lists the possible lease stages and explains their function.

IMPROVEMENT System Performance Data Reported On Lease Timeout

IMPORTANT This improvement was not ported to SQL Server 2012, it will only be available in SQL Server 2014 Service Pack 2 and SQL Server 2016.

Lease timeouts are often a symptom of a system wide event that prevents lease renewal for a lease timeout period. Unresponsive disk subsystem, 100% CPU utilization, or in the case of virtualization, contention on the host server, are examples of issues that could trigger the lease timeout.

In the event of a lease timeout, the following system performance objects are reported in the Cluster log: Processor time (%), Available memory (bytes), Avg disk sec/ read and Avg disk sec/write are now reported. The last five retained sets of performance data are reported, whose collection interval is dictated by the frequency at which sp_server_diagnostics results are processed by the SQL Server resource DLL (hosted in RHS.EXE), which by default, is every 10 seconds.

Below is an example of the data set. Assuming 10 second interval, the performance data reports on these objects for the last 50 seconds. The first message below reports the lease timeout. The next message is a heading of the forthcoming performance data in the next five messages, so you can read the output like a table:

00002968.000019c8::2015/09/03-17:20:40.052 WARN  [RES] SQL Server Availability Group: [hadrag] Lease timeout detected, logging perf counter data collected so far

00002968.000019c8::2015/09/03-17:20:40.052 WARN  [RES] SQL Server Availability Group: [hadrag] Date/Time, Processor time(%), Available memory(bytes), Avg disk read(secs), Avg disk write(secs)

00002968.000019c8::2015/09/03-17:20:40.052 WARN  [RES] SQL Server Availability Group: [hadrag] 9/3/2015 17:19:56.0, 3.216014, 2219802624.000000, 0.001333, 0.000355

00002968.000019c8::2015/09/03-17:20:40.052 WARN  [RES] SQL Server Availability Group: [hadrag] 9/3/2015 17:20:6.0, 3.010540, 2219888640.000000, 0.001150, 0.000963

00002968.000019c8::2015/09/03-17:20:40.052 WARN  [RES] SQL Server Availability Group: [hadrag] 9/3/2015 17:20:16.0, 5.524332, 2224078848.000000, 0.000000, 0.000697

00002968.000019c8::2015/09/03-17:20:40.052 WARN  [RES] SQL Server Availability Group: [hadrag] 9/3/2015 17:20:26.0, 3.334432, 2224504832.000000, 0.001367, 0.000438

00002968.000019c8::2015/09/03-17:20:40.052 WARN  [RES] SQL Server Availability Group: [hadrag] 9/3/2015 17:20:36.0, 9.189380, 2225078272.000000, 0.001367, 0.000834 

 

IMPROVEMENT Extended Event Reports Lease Timeout With Lease Stages

The cluster log has a new message that reports the timeout and the expected renewal deadline that was violated. Here is the message.

The lease is expired. The lease should have been renewed by (year)/(month)/(day)-(hour):(min)(sec).(msec)

Example When a 19407 is reported in the SQL Server error log, a corresponding event is now recorded in the Cluster log.

Here is an example of the 19407 reported in the SQL Server error log. Note the timestamp of the 19407 in the SQL Server error log and the timestamp of the cluster log message. SQL Server is reporting local time and the Cluster log is reporting in UTC, which explains the seven hour difference. However, importantly, the cluster log reports the lease expired 10 seconds after its expected renewal (21:04:18.710 versus 21:04:08.711) which makes sense since that would be the default timeout period (20 seconds divided by 2).

2016-02-16 13:09:17.63 Server      Error: 19407, Severity: 16, State: 1.
2016-02-16 13:09:17.63 Server      The lease between availability group ‘ag’ and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is failing over correctly, check the corresponding availability group resource in the Windows Server Failover Cluster.

Here is the corresponding error reported in the Cluster log (note the time is in UTC):

00000690.00001330::2016/02/16-21:04:18.710 ERR   [RES] SQL Server Availability Group <ag>: [hadrag] The lease is expired. The lease should have been renewed by 2016/02/16-21:04:08.711

Basic Availability Group on Standard Edition

Historically, SQL Server database mirroring offered high availability of a SQL Server database. AlwaysOn availability groups has supplanted database mirroring as the future of high availability in SQL Server, meanwhile, database mirroring has been marked for deprecation. As database mirroring is deprecated, SQL Server 2016 introduces AlwaysOn Basic Availability Groups, offered as a database mirroring replacement, offering a similar feature profile.

Basic availability groups provide the capability to use AlwaysOn availability groups with SQL Server Standard Edition servers, with the same limitations that database mirroring imposes today.

Basic Availability Group Limitations

Basic availability group ‘limitations’ are those that reflect legacy SQL Server mirroring functionality. In other words, these limits on a basic availability group result in behavior that ‘mirrors’ SQL Server database mirroring.

• A basic availability group may not contain replicas on instances older than SQL Server 2016.
• Only one database may exist in a basic availability group.
• The AUTOMATED_BACKUP_PREFERENCE setting of the availability group will be restricted to ‘PRIMARY’.
• The secondary replica is not readable:
  • Any backup operations on the secondary will fail.
  • CHECKDB will not be allowed on a secondary.
• The replicas in a basic availability group may be configured for synchronous or asynchronous commit.
• Basic availability groups may form a hybrid availability group (one replica on premise and one in an Azure virtual machine).

Create a Basic Availability Group

A basic availability group can be created two ways: using the CREATE AVAILABILITY GROUP syntax with the new BASIC keyword, or using the availability group wizard.

CREATE AVAILABILITY GROUP (Transact-SQL)

You can create a basic availability group using the Transact-SQL CREATE AVAILABILITY GROUP. Here is the syntax:

CREATE AVAILABILITY GROUP <My_AG> WITH [BASIC | ADVANCED]…

The following is the syntax to create a basic availability group named agBasic containing database agdb and hosted on replicas SQL16N1 and SQL16N2:

CREATE AVAILABILITY GROUP [agBasic]
WITH (BASIC)
FOR DATABASE [agdb]
REPLICA ON
N’SQL16N1′ WITH (ENDPOINT_URL = N’TCP://SQL16N1.AGDC.COM:5022′, FAILOVER_MODE = MANUAL, AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT),
N’SQL16N2′ WITH (ENDPOINT_URL = N’TCP://SQL16N2.AGDC.COM:5022′, FAILOVER_MODE = MANUAL, AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT);

Create an Availability Group using the New Availability Group Wizard

When creating an availability group on a standard edition of SQL Server using the new availability group wizard, a basic availability group is created. When creating an availability group on an enterprise edition of SQL Server an advanced availability group is created.

When creating a new basic availability group using the new availability group wizard certain features will be disabled that are not compatible with this restricted version of the availability group. For example, in the Add Replica dialogue once two replicas have been added, the Add Replica button grays out. Also, under Backup Preferences, those settings are grayed out as well.

DMVs for Basic Availability Groups

How can I tell if the availability group is Basic or Advanced? Query sys.availability_groups for the basic_features column, if it is set to 1, the availability group is a basic availability group.

select name, basic_features from sys.availability_groups

FAQ for Basic Availability Groups

When I attempt to create an availability group or alter an availability group, message 41199 is returned Message 41199 is reported when you attempt to create an availability group or modify an availability group in such a way that is not compatible with basic availability groups. For example, since basic availability groups do not support readable secondary, attempting to create an availability group and configure the SECONDARY_ROLE for a replica fails:

Msg 41199, Level 16, State 8, Line 11
The specified command is invalid because the Always On Availability Groups allow_connections feature is not supported by this edition of SQL Server. For information about features supported by the editions of SQL Server, see SQL Server Books Online.

Here is another example, attempting to join a replica running on an Enterprise Edition of SQL Server to an existing basic availability group fails:

Msg 41199, Level 16, State 2, Line 1
The specified command is invalid because the Always On Availability Groups join availability group (basic) feature is not supported by this edition of SQL Server.
For information about features supported by the editions of SQL Server, see SQL Server Books Online.
Msg 41158, Level 16, State 3, Line 1
Failed to join local availability replica to availability group ‘agbasic’.  The operation encountered SQL Server error 41199 and has been rolled back.
Check the SQL Server error log for more details.  When the cause of the error has been resolved, retry the ALTER AVAILABILITY GROUP JOIN command.

When I try to create a basic availability group it fails Basic availability groups can only be created on a standard edition of SQL Server. Attempting to create a basic availability group in SQL Server Enterprise Edition will fail:

Msg 534, Level 15, State 1, Line 3
‘BASIC’ failed because it is not supported in the edition of this SQL Server instance ‘SQL16N1’. See books online for more details on feature support in different SQL Server editions.

I cannot set Readable Secondary to anything other than No This is expected. SQL Server Management Studio restricts settings like readable secondary to those values that are supported by basic availability groups:

How do I upgrade from a basic availability group to an advanced availability group? There is no upgrade path to change a basic availability group to an advanced availability group. This comes into play if you edition upgrade SQL Server from standard edition to enterprise edition.

The availability group will still be a basic availability group and restricted in the ways a basic availability group is restricted. In order to remove these restrictions, the availability group must be dropped and recreated. Attempting to issue ALTER AVAILABILITY GROUP SET (ADVANCED) is not recognized by SQL Server.

For more information see the Basic Availability Groups page:

Basic Availability Groups (AlwaysOn Availability Groups)

Overview

In SQL Server 2012 and 2014, initializing a secondary replica is cumbersome to do manually, and requires backup and restore of your database and transaction log from the primary to a network share, and then a restore of the database and log backup files to the secondary replica. The availability group wizard can automate this initialization, but still requires additional configuration of a network share and permissions to access that network share by the SQL Server service startup accounts that SQL Server is running with.

SQL Server 2016 introduces a simpler way to initialize databases on secondary replicas, by using direct seeding. With direct seeding, when a database is added to the availability group at the primary, it is automatically initialized at the secondary with no further user intervention.

Behind the scenes, a VDI backup is performed over the network to the secondary replica where it is restored.

Benefits

The benefits are simpler, more automated initialization of your availability group databases to the secondary replica. When using direct seeding there is no need to configure a file share for backup and restore which is one less configuration headache.

When Would You not use Direct Seeding

Some scenarios may not be optimal for using direct seeding when initializing a secondary replica. Initialization of the availability database using direct seeding does not use compression, while the availability group wizard, performs backup and restore using compression.

For that reason, initialization can be slow if some databases are very large, or the secondary replica is remote.

Compression is disabled by default for direct seeding. You can enable compression by enabling trace flag 9567 as a startup trace flag or using DBCC TRACEON. When adding a large database or several databases using direct seeding you may observe high cpu utilization on the SQL Server during direct seeding with compression enabled. It is recommended that you test the trace flag prior to implementing in a production environment.

IMPORTANT The transaction log for these databases cannot be truncated during direct seeding, so a prolonged initialization process on a busy database can result in significant transaction log growth. For more information on operations that can delay log truncation check out Factors That Can Delay Log Truncation.

It is recommended that database size, replication distance and primary database load be considered before using direct seeding.

Demo: Initialize a secondary replica with direct seeding

The new CREATE and ALTER AVAILABILITY GROUP syntax includes a new parameter, SEEDING_MODE and is configured on a per-replica basis. SEEDING_MODE has two settings:

• Manual which indicates legacy backup and restore to initialize the databases onto the secondary. This is the default.
• Automatic which defines the new automated initialization of the availability group databases on the secondary replica

This feature is available when creating and configuring an availability group using transact-sql. The new availability group wizard does not provide knobs for enabling SEEDING_MODE.

The following demo shows how to automatically seed availability group databases to the secondary replica.

Create an empty availability group

Create an empty availability group on the primary replica (no databases) and configure each replica for SEEDING_MODE=AUTOMATIC:

:Connect SQL16N1
CREATE AVAILABILITY GROUP [SEEDag]
FOR
REPLICA ON N’SQL16N1′ WITH (ENDPOINT_URL = N’TCP://SQL16N1.AGDC.COM:5022′, FAILOVER_MODE = AUTOMATIC, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT, SEEDING_MODE=AUTOMATIC),
N’SQL16N2′ WITH (ENDPOINT_URL = N’TCP://SQL16N2.AGDC.COM:5022′, FAILOVER_MODE = AUTOMATIC, AVAILABILITY_MODE = SYNCHRONOUS_COMMIT, SEEDING_MODE=AUTOMATIC);
GO

With the empty availability group created, connect to SQL Server that is your secondary replica and join the availability group. Then, configure the newly joined replica permissions to create database:

:Connect SQL16N2
ALTER AVAILABILITY GROUP [SEEDag] JOIN
GO

:Connect SQL16N2
ALTER AVAILABILITY GROUP SEEDag
GRANT CREATE ANY DATABASE
GO

On the primary replica, add databases to your availability group. When you add the database to the availability group, seeding of the database to the secondary replica(s) commences. When complete, these databases should appear on the primary and secondary as part of the availability group.

:Connect SQL16N1
alter AVAILABILITY group SEEDag add database db1
go
alter AVAILABILITY group SEEDag add database db2
go

Troubleshooting Direct Seeding

Monitor and investigate automatic availability replica seeding using new DMVs, XEvents and the SQL Server error log.

Direct Seeding DMVs

Two new DMVs report information on direct seeding and each report unique information. Query them at the primary and the secondary when trouble-shooting seeding failures, since the errors may be reported more accurately at the replica where the failure occurred.

Query sys.dm_hadr_automatic_seeding for information on direct seeding Here you can find information on successful or failed database seedings and error messages explaining why the seeding may have failed.

select * from sys.dm_hadr_automatic_seeding

Query sys.dm_hadr_physical_seeding_stats for statistical information on completed and ongoing availability databases seeding You can get information on currently running seedings like estimated completion time, and also statistics on completed seedings.

select * from sys.dm_hadr_physical_seeding_stats

Here is a sample of the output when this is queried during an automatic seed.

Diagnose Database Initialization Using direct seeding in the SQL Server error log

When you add a database to your availability group configured for direct seeding, a VDI backup is performed over the availability group endpoint connection that is used for synchronization. The SQL Server error log can be reviewed to find information on when the backup completed and the secondary was synchronized:

Primary replica SQL Server error log

2016-04-27 17:32:02.61 spid38s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:0
2016-04-27 17:32:02.63 spid38s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:0
2016-04-27 17:32:02.89 spid99s     Processed 296 pages for database ‘db1’, file ‘db1’ on file 1.
2016-04-27 17:32:02.96 spid99s     Processed 13 pages for database ‘db1’, file ‘db1_log’ on file 1.
2016-04-27 17:32:03.10 Backup      Database backed up. Database: db1, creation date(time): 2016/04/27(15:40:44), pages dumped: 530, first LSN: 34:112:65, last LSN: 34:320:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {‘{840AB572-75B6-4577-894C-85505ABAC89A}’}). This is an informational message only. No user action is required.
2016-04-27 17:32:03.16 spid99s     BACKUP DATABASE successfully processed 309 pages in 0.347 seconds (6.956 MB/sec).
2016-04-27 17:32:03.16 Backup      BACKUP DATABASE successfully processed 309 pages in 0.347 seconds (6.956 MB/sec).
2016-04-27 17:32:03.98 spid96s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:0
2016-04-27 17:32:03.98 spid96s     Always On Availability Groups connection with secondary database established for primary database ‘db1’ on the availability replica ‘SQL16N2’ with Replica ID: {0883a09e-7587-42d7-a6e6-d7ebf96079f4}. This is an informational message only. No user action is required.
2016-04-27 17:32:04.00 spid96s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:1
2016-04-27 17:32:04.01 spid96s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:1
2016-04-27 17:32:04.31 spid96s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:1
2016-04-27 17:32:04.31 spid96s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:1
2016-04-27 17:32:04.32 spid96s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:1
2016-04-27 17:32:04.32 spid96s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 0883A09E-7587-42D7-A6E6-D7EBF96079F4:1
2016-04-27 17:32:04.32 spid96s     Always On Availability Groups connection with secondary database established for primary database ‘db1’ on the availability replica ‘SQL16N2’ with Replica ID: {0883a09e-7587-42d7-a6e6-d7ebf96079f4}. This is an informational message only. No user action is required.

On the secondary replica we see the database restored and the hardened LSN for the database on the secondary is reported.

Secondary replica SQL Server error log

2016-04-27 17:32:02.65 spid69s     Error: 911, Severity: 16, State: 1.
2016-04-27 17:32:02.65 spid69s     Database ‘db1’ does not exist. Make sure that the name is entered correctly.
2016-04-27 17:32:03.41 spid69s     Processed 296 pages for database ‘db1’, file ‘db1’ on file 1.
2016-04-27 17:32:03.41 spid69s     Processed 13 pages for database ‘db1’, file ‘db1_log’ on file 1.
2016-04-27 17:32:03.53 spid69s     [INFO] HkHostDbCtxt::Initialize(): Database ID: [5] ‘db1’. XTP Engine version is 0.0.
2016-04-27 17:32:03.53 spid69s     Starting up database ‘db1’.
2016-04-27 17:32:03.57 spid69s     [INFO] HkHostDbCtxt::Initialize(): Database ID: [5] ‘db1’. XTP Engine version is 0.0.
2016-04-27 17:32:03.60 spid69s     The database ‘db1’ is marked RESTORING and is in a state that does not allow recovery to be run.
2016-04-27 17:32:03.91 Backup      Database was restored: Database: db1, creation date(time): 2016/04/27(15:40:44), first LSN: 34:112:65, last LSN: 34:320:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {‘{0657A8FD-46D1-44FF-9A74-9C12D9D5E152}’}). Informational message. No user action required.
2016-04-27 17:32:03.95 spid69s     RESTORE DATABASE successfully processed 309 pages in 0.103 seconds (23.437 MB/sec).
2016-04-27 17:32:03.95 Backup      RESTORE DATABASE successfully processed 309 pages in 0.103 seconds (23.437 MB/sec).
2016-04-27 17:32:04.02 spid69s     Always On Availability Groups connection with primary database established for secondary database ‘db1’ on the availability replica ‘SQL16N1’ with Replica ID: {12dc5154-baa8-475d-b2ff-bf093c6f97eb}. This is an informational message only. No user action is required.
2016-04-27 17:32:04.04 spid47s     State information for database ‘db1’ – Hardened Lsn: ‘(34:320:1)’    Commit LSN: ‘(0:0:0)’    Commit Time: ‘Jan  1 1900 12:00AM’

Here is the error log from the secondary for a failed direct seeding The failure occurred because the path to the database and log files on the primary did not exist on the secondary, resulting in failed seeding:

2016-04-27 18:18:29.57 spid58s     Starting up database ‘db1’.
2016-04-27 18:18:29.62 spid58s     [INFO] HkHostDbCtxt::Initialize(): Database ID: [6] ‘db1’. XTP Engine version is 0.0.
2016-04-27 18:18:29.65 spid58s     The database ‘db1’ is marked RESTORING and is in a state that does not allow recovery to be run.
2016-04-27 18:18:29.66 spid40s     The state of the local availability replica in availability group ‘SEEDag’ has changed from ‘RESOLVING_NORMAL’ to ‘NOT_AVAILABLE’.  The state changed because either the associated availability group has been deleted, or the local availability replica has been removed from another SQL Server instance.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.
2016-04-27 18:19:34.22 spid57      The state of the local availability replica in availability group ‘SEEDag’ has changed from ‘NOT_AVAILABLE’ to ‘RESOLVING_NORMAL’.  The state changed because the local availability replica is joining the availability group.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.
2016-04-27 18:19:34.26 spid37s     The state of the local availability replica in availability group ‘SEEDag’ has changed from ‘RESOLVING_NORMAL’ to ‘SECONDARY_NORMAL’.  The state changed because the availability group state has changed in Windows Server Failover Clustering (WSFC).  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.
2016-04-27 18:19:34.37 spid36s     A connection for availability group ‘SEEDag’ from availability replica ‘SQL16N2\STAND’ with id  [B4B833CB-032E-45D6-9CEB-CB8C34C61224] to ‘SQL16N1’ with id [DC980B48-B7CE-41AC-A54B-EB67E6F1B98F] has been successfully established.  This is an informational message only. No user action is required.
2016-04-27 18:19:44.68 spid47s     Error: 911, Severity: 16, State: 1.
2016-04-27 18:19:44.68 spid47s     Database ‘db1’ does not exist. Make sure that the name is entered correctly.
2016-04-27 18:19:44.91 spid47s     Error: 5133, Severity: 16, State: 1.
2016-04-27 18:19:44.91 spid47s     Directory lookup for the file “C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\DATA\db1.mdf” failed with the operating system error 2(The system cannot find the file specified.).
2016-04-27 18:19:44.91 spid47s     Error: 3156, Severity: 16, State: 3.
2016-04-27 18:19:44.91 spid47s     File ‘db1’ cannot be restored to ‘C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\DATA\db1.mdf’. Use WITH MOVE to identify a valid location for the file.

Diagnose direct seeding with extended events

Direct seeding has new XEvents for tracking state change, failures, and performance statistics as an initialization is progressing.

Here is a script to create an XEvent session that captures this event.

CREATE EVENT SESSION [AlwaysOn_autoseed] ON SERVER
ADD EVENT sqlserver.hadr_automatic_seeding_state_transition,
ADD EVENT sqlserver.hadr_automatic_seeding_timeout,
ADD EVENT sqlserver.hadr_db_manager_seeding_request_msg,
ADD EVENT sqlserver.hadr_physical_seeding_backup_state_change,
ADD EVENT sqlserver.hadr_physical_seeding_failure,
ADD EVENT sqlserver.hadr_physical_seeding_forwarder_state_change,
ADD EVENT sqlserver.hadr_physical_seeding_forwarder_target_state_change,
ADD EVENT sqlserver.hadr_physical_seeding_progress,
ADD EVENT sqlserver.hadr_physical_seeding_restore_state_change,
ADD EVENT sqlserver.hadr_physical_seeding_submit_callback
ADD TARGET package0.event_file(SET filename=N’autoseed.xel’,max_file_size=(5),max_rollover_files=(4))
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=ON)
GO
ALTER EVENT SESSION AlwaysOn_autoseed ON SERVER STATE=START
GO

Here are a list of useful direct seeding XEvents.

XEvent Comments

• Collect ‘backup’ XEvents on the primary, and collect ‘restore’ XEvents on the secondary.
• The hadr_automatic_seeding_state_transition XEvent occurs on the primary and secondary and report previous and current states to document state transition.
  
• The hadr_physical_seeding_progress XEvent reports performance information related to the seeding.
• When trouble-shooting failures and performance, the DMVs may be a better first step for diagnosis.
• Be careful when collecting XEvents, some may generate numbers of events that can impact SQL Server performance.

FAQs

I used direct seeding mode to start initializing my secondary, when will it be complete?

Query the sys.dm_hadr_physical_seeding_stats DMV for status. One row for each database is listed. Here is a query that returns the data points from the DMV that might be useful when monitoring progress.

select local_database_name, role_desc, internal_state_desc, transfer_rate_bytes_per_second, transferred_size_bytes, database_size_bytes,
start_time_utc, end_time_utc, estimate_time_complete_utc, total_disk_io_wait_time_ms, total_network_wait_time_ms, is_compression_enabled
from sys.dm_hadr_physical_seeding_stats

I setup direct seeding and added a database at the primary replica, but the database does not appear as part of the availability group at the primary or the secondary in SQL Server Management Studio or in the availability group dashboard

If you add a database and it fails to appear as part of the availability group, direct seeding likely failed, which prevents addition of the database at the primary and secondary.

Query sys.dm_hadr_automatic_seeding at the primary and secondary. Here I queried it at the primary to find out why direct seeding failed:

select start_time, completion_time, is_source, current_state, failure_state, failure_state_desc, error_code  from sys.dm_hadr_automatic_seeding

I want to direct seeding to a secondary, but the path to the database data and log files is different, will direct seeding succeed?

No, direct seeding assumes that the path to the data and log files on the primary exist on the secondary. If the paths do not exist, direct seeding will fail.

For more information see the following resources:

CREATE AVAILABILITY GROUP (Transact-SQL)

Overview

SQL Server 2104 and SQL 2012 read-only routing directed traffic to the first available replica in the routing list, unless it was not accessible, and then it would direct the connection to the next replica in the routing list. When you have multiple secondary replicas available for read, it is not possible to spread the read load across those replicas.

In SQL Server 2016 you can configure load-balancing across a set of read-only replicas.

Benefits

This will allow following benefits

• Use advanced configuration to ‘bundle’ read-only routed connections to certain replicas.
• Load balance read-only routed requests across a set of preconfigured available readable secondary replicas.

Configure an Availability Group for Load Balanced Read-Only Routing

First, configure each replica to allow for read access when in the secondary role, and define the connection string to the replica. These two steps are completed just like you would configure read only routing in SQL Server 2012 or SQL Server 2014.

1 Configure read-only access

Enable read access so that the replica can be accessed when it is a secondary:

ALTER AVAILABILITY GROUP [ag]
MODIFY REPLICA ON N’SQL16N2′ WITH (SECONDARY_ROLE(ALLOW_CONNECTIONS = READ_ONLY))
GO

ALTER AVAILABILITY GROUP [ag]
MODIFY REPLICA ON N’SQL16N3′ WITH (SECONDARY_ROLE(ALLOW_CONNECTIONS = READ_ONLY))
GO

2 Configure read-only routing URL

For each secondary replica, define the address to connect to it:

ALTER AVAILABILITY GROUP ag MODIFY REPLICA ON N’SQL16N2′ WITH (SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N’TCP://SQL16N2:1433′));
GO
ALTER AVAILABILITY GROUP ag MODIFY REPLICA ON N’SQL16N3′ WITH (SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N’TCP://SQL16N3:1433′));
GO

3 SQL Server 2016 introduces load balanced lists in the read-only routing list

Create a read-only routing list for SQL16N1 when it is in the primary role. Here is an example:

ALTER AVAILABILITY GROUP ag MODIFY REPLICA ON N’SQL16N1′ WITH (PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=((‘SQL16N3’, ‘SQL16N2’), ‘SQL16N1’)));

This routing list behavior ‘load balances’ read-only connections between SQL16N3 and SQL16N2. Above, we have two embedded lists in the routing list:

List 1: ‘SQL16N3’, ‘SQL16N2’

List 2: ‘SQL16N1’

Here is how the routing plays out:

Route to the replicas in the first list SQL16N3 and SQL16N2 are accessible to read-only connections. The first incoming read-only connection will be routed to SQL16N3, the second read-only connection will be routed to SQL16N2, the third read-only connection will be routed to SQL16N3, the fourth read-only connection will be routed to SQL16N2, and so on, with a ‘round-robin’ distribution of read-only connections between the two replicas in the first list.

If any replicas become unavailable, routing will continue with remaining replicas in the first list If SQL16N3 or SQL16N2 becomes inaccessible to read-only connections, then the read-only connections will only be routed to the accessible read only replicas in the first list. For example, if SQL16N3 is in a not synchronized state or ALLOW_CONNECTIONS is set to NO, then all read-only connections will be routed to SQL16N2. So long as one of the servers is available for read-only connections, NO read-only connections will be routed to SQL16N1.

If all replicas in first list are inaccessible route to replicas in next list If SQL16N3 and SQL16N2 become inaccessible to read-only connections, then all read-only connections will only be routed to the next list of replicas, which in this case is SQL16N1.

Resume routing to first list if any replicas become available As secondary replicas that have higher priority in the list become accessible to read-only connections, future read-only connections will connect to them as appropriate.

Test your read-only routing

SQLCMD is a good tool for testing read-only routing and can be used from the primary replica. Remember to specify the listener name, read intent connection parameter set to read only and an availability database. If these are not all present, SQLCMD  connection will not route to the secondary, sort of a silent failure, the connection may succeed, but to the primary replica.

I have configured my routing list for load balance just as the example above demonstrates. Here is my test. I connect twice using SQLCMD and notice that the first connection attempt results in a connection to SQL16N3, my second connection attempt results in a connection to SQL16N2, just as my load balanced routing list defined.

Troubleshooting Read-Only Routing

Useful Queries for Diagnosing Read-Only Routing Issues

Execute this query against the primary replica to see the settings and state information on your availability replicas that can impact the replica accessibility for read-only routing. See the FAQ section below for how this query can be used to trouble-shoot different scenarios you might encounter.

select ar.replica_server_name, ar.endpoint_url, ar.read_only_routing_url, secondary_role_allow_connections_desc, ars.synchronization_health_desc
from sys.availability_replicas ar join sys.dm_hadr_availability_replica_states ars on ar.replica_id=ars.replica_id

View replica routing list priority

select ar.replica_server_name,arl.routing_priority, ar2.replica_server_name,ar2.read_only_routing_url
from sys.availability_read_only_routing_lists arl join sys.availability_replicas ar
on (arl.replica_id = ar.replica_id) join sys.availability_replicas ar2
on (arl.read_only_replica_id = ar2.replica_id)
order by ar.replica_server_name asc, arl.routing_priority asc

The routing list for availability group ar is defined like this, this is how a SQL Server 2012 or SQL Server 2014 routing list might look.

ALTER AVAILABILITY GROUP ag MODIFY REPLICA ON N’SQL16N1′ WITH (PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=(‘SQL16N3’, ‘SQL16N2’, ‘SQL16N1’)));

Execute the replica routing list priority query. There is a clearly defined routing priority across all three replicas.

Now, adjust the routing list to implement load balancing between SQL16N3 and SQL16N2:

ALTER AVAILABILITY GROUP ag MODIFY REPLICA ON N’SQL16N1′ WITH (PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=((‘SQL16N3’, ‘SQL16N2’), ‘SQL16N1’)));

Query the replica routing list priority again. The priority of the load balanced replicas, SQL16N3 and SQL16N2 are tied at 1, indicating that they are both eligible to route to but SQL16N1 will be a lower priority:

Diagnose Read-Only Routing with Extended Events

There are extended events that can be used to assist in diagnosing read-only routing connection behavior.

sqlserver.hadr_evaluate_readonly_routing_info  — ​Despite what the name implies, this XEvent is not triggered when a read-only connection attempt is made. This event will be triggered on the primary replica only when the routing list is ‘evaluated’ for one of the following reasons:

1. When log scanning of an availability database begins or ceases, which occurs when an availability group goes offline or comes online, or when synchronization of an availability database is suspended or resumed.
2. When the routing list or routing URL is modified.
3. When any other availability group configuration change is made at the primary replica.

Here is what the XEvent looks like captured. The is_routing_replica_not_found is only TRUE when SQL Server cannot return a routing URL to the client because there are no accessible replicas.

sqlserver.read_only_route_complete  This XEvent is triggered on the primary replica only. This XEvent is triggered when SQL Server sends a replica URL to the client, but does not mean the connection will be successful. This indicates that there is an availability replica from the routing list that is accessible for read-only routing. If the URL is bad, or there are connectivity issues to the replica, the connection will fail, but this event will show complete.

Here is what the read_only_route_complete XEvent looks like:

sqlserver.read_only_route_fail  — This XEvent is triggered on the primary replica only, in the event that SQL Server is unable to respond to the client attempting a read-only connection with a readonly URL. This occurs when no secondary is accessible for reasons such as they are not synchronized, offline, not configured for read access, or their readonly URL is incorrectly formatted.

Here is what the XEvent looks  like when triggered. There is no interesting information reported, since we know that the event means that no read-only URL could be returned to the client, there is no report of the circumstances behind the event.

Here is a script to create an XEvent session that captures these events. Execute this on the primary replica of your availability group:

CREATE EVENT SESSION [AlwaysOn_ROO] ON SERVER
ADD EVENT sqlserver.hadr_evaluate_readonly_routing_info,
ADD EVENT sqlserver.read_only_route_complete,
ADD EVENT sqlserver.read_only_route_fail
ADD TARGET package0.event_file(SET filename=N’AlwaysOn_ROO.xel’,max_file_size=(5),max_rollover_files=(4))
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=ON)
GO
ALTER EVENT SESSION AlwaysOn_ROO ON SERVER STATE=START
GO

To monitor activity, locate the XEvent session and ‘Watch Live Data’:

Here is our routing list:

ALTER AVAILABILITY GROUP ag MODIFY REPLICA ON N’SQL16N1′ WITH (PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=(‘SQL16N3’, ‘SQL16N2’)));

First, connect to SQL Server using read-routing, and we will see we are successfully connected to the first replica in our routing list. The read_only_route_complete is triggered because SQL Server was able to return an accessible replica routing URL to the client.

Next, let’s alter the read access of our two replicas listed in the routing list, we will set them both to NO for read access.

ALTER AVAILABILITY GROUP [ag]
MODIFY REPLICA ON N’SQL16N2′ WITH (SECONDARY_ROLE(ALLOW_CONNECTIONS = NO))
GO
ALTER AVAILABILITY GROUP [ag]
MODIFY REPLICA ON N’SQL16N3′ WITH (SECONDARY_ROLE(ALLOW_CONNECTIONS = NO))
GO

The hadr_evaluate_readonly_routing_info XEvent is triggered because we altered the availability group’s routing information. The read_only_route_fail XEvent is triggered because no secondary replica in the routing list could be accessed, and the client fails to connect and receives message ‘Unable to access the ‘agdb’ database because no online secondary replicas are enabled for read only access.’

FAQs

I configured my application to connect with read-only routing but I successfully connect to the primary replica

Reason Your application must properly specify the database name of an availability group database. For example, when attempting to connect using SQLCMD, if we specify all the parameters, but fail to pass the database parameter, set to an availability database, the connection fails to route to the secondary and is routed to the primary. Here we see SQLCMD passing the read-only routing parameter but does not specify the availability database as a parameter, resulting in connection to the primary instead of a read-only secondary.

Reason If you defined the primary replica in your read-only routing list and your secondary replicas cannot be accessed (for example, synchronization is suspended), then the routing will connect the application to the primary replica. For example, the current primary replica is SQL16N1 and we define the routing list like this:

/*Configure a routing list. If SQL16N3 is not available read-only connections connect to SQL16N2*/
ALTER AVAILABILITY GROUP ag MODIFY REPLICA ON N’SQL16N1′ WITH (PRIMARY_ROLE (READ_ONLY_ROUTING_LIST=(‘SQL16N3’, ‘SQL16N2’, ‘SQL16N1’)));
GO

I configured my routing list but I am connecting to a replica I didn’t expect to connect to

Reason The secondary replica you anticipated connecting to is not accessible, here are the reasons:

1. The secondary is not configured for read access. You must set the secondary for read access, see step 1 in the above section ‘Configure an Availability Group for Load Balanced Read-Only Routing.’
2. The secondary is suspended. If the secondary is not in a synchronizing or synchronized state, you cannot read-only connect to that secondary replica.
3. The SQL Server hosting the secondary replica is not running.
4. The read-only routing URL for that secondary is not defined correctly. Compare how that routing URL has been defined for the problematic secondary, see step 2 in the above section ‘Configure an Availability Group for Load Balanced Read-Only Routing.’ To see the existing read-only URL query:

The following query can be executed against the primary replica and it will provide the details needed to trouble-shoot all of the above scenarios.

select ar.replica_server_name, ar.endpoint_url, ar.read_only_routing_url, secondary_role_allow_connections_desc, ars.synchronization_health_desc
from sys.availability_replicas ar join sys.dm_hadr_availability_replica_states ars on ar.replica_id=ars.replica_id

In the following graphic we can see the results of this query, note that SQL16N2 and SQL16N3 are not configured for read-only connections. Also SQL16N2 is not healthy right now, it may be suspended or SQL Server may not be running on SQL16N2. Finally, there is a problem with the read-only routing URL for SQL16N1:

For more information see the Read-Only Routing page:

Configure Read-Only Routing for an Availability Group (SQL Server)

Overview

SQL Server 2104 and SQL Server 2012 can have two synchronous secondary replicas. One synchronous secondary replica can serve the role as automatic failover partner with the primary replica.

In SQL Server 2016 both synchronous secondary replicas can be configured as automatic failover partners with the primary replica.

Benefits

This will allow following benefits

• This increases the chances that high availability can be maintained if one of the automatic failover partners is lost.
• Reduces the need to manually manage failovers or to reconfigure the availability group with another automatic failover partner in the event that one of the automatic failover partners is unavailable.

Configure Multiple Automatic Failover Targets

You can configure three replicas as automatic failover partners using Transact-SQL or SQL Server Management Studio.

USE [master] GO ALTER AVAILABILITY GROUP [AG] MODIFY REPLICA ON N’SQLN1‘ WITH (FAILOVER_MODE = AUTOMATIC) GO ALTER AVAILABILITY GROUP [AG] MODIFY REPLICA ON N’SQLN2‘ WITH (FAILOVER_MODE = AUTOMATIC) GO ALTER AVAILABILITY GROUP [AG] MODIFY REPLICA ON N’SQLN3‘ WITH (FAILOVER_MODE = AUTOMATIC) GO

Troubleshooting Automatic Failover Problems

Troubleshooting multiple automatic failover targets would be similar to the steps of the following article used to troubleshoot automatic failover issues with a single automatic failover target:

Troubleshooting automatic failover problems in SQL Server 2012 AlwaysOn environments

Useful Query

The following query may be used to check replica properties for multiple automatic failover partners and AG database synchronization health:

select replica_server_name, availability_mode_desc,failover_mode_desc, database_id, synchronization_health_desc,synchronization_state_desc,* from sys.availability_replicas as AR inner join sys.dm_hadr_database_replica_states as DRS on DRS.replica_id = AR.replica_id

FAQs

If both the secondary replicas are SYNCHRONIZED and healthy, which replica with Cluster choose to failover to?

Windows Cluster will attempt to failover to the next replica in the preferred owner list of the availability group, which dictates attempted failover order. The preferred owner list of the availability group role in Cluster will list all the availability group replicas defined for automatic failover.

For example replicas SQL16N1 (primary), SQL16N2 (secondary) and SQL16N3 (secondary replica) are all defined for automatic failover and are in that order in the preferred owner list. In the event that a health issue is detected in SQL16N1, Cluster will attempt to failover to SQL16N2 next. In the event that SQL16N2 is not healthy (for example, an availability database was not SYNCHRONIZED) Cluster will attempt to failover to SQL16N3.

Here is a picture of the availability group role properties from Failover Cluster Manager, note the Up and Down buttons for setting priority of these nodes as owner of the primary replica.

Can we set the automatic failover priority, choosing a particular replica for a preferred failover target in SQL Server or by using Failover Cluster Manager?

There is no setting in SQL Server for configuring the preferred owner list settings of the availability group role. In addition, resetting the priority of the nodes in the preferred owner list of the availability group role in Failover Cluster Manager is also not recommended because SQL Server manages Preferred and Possible Owner Properties for AlwaysOn Availability Group/Role. When a failover occurs, SQL Server resets the Preferred Owner list, over-writing any change made through Failover Cluster Manager.

Furthermore, modifying AG resource properties in the cluster manager is not recommended.

Tests have shown you can change the Preferred Owner list in Failover Cluster, and the next automatic failover will abide by the modified preferred owner list. However, once SQL Server resets the preferred owner list, subsequent automatic failovers will proceed with the preferred owner priority set by SQL Server.

Workaround: Configure automatic failover priority by adding replicas to availability group in preferred automatic failover priority

SQL Server will set the preferred owner list priority based on the order in which your availability group replicas are added to the availability group. This can allow you some control over what automatic failover partner Cluster attempts to failover to first.

For example, you have defined an availability group with primary replica SQL16N1 and add SQL16N2 and SQL16N3 as automatic failover partners. If you add SQL16N2 and then SQL16N3 using the Add Replica to Availability Group wizard, SQL Server will add them in the same order to the preferred owner list:

When a health event is detected in SQL16N1 (in this example, I shut down SQL16N1 through SSMS Object Explorer), Cluster checks the next node in the preferred owner list which is SQL16N2. Assuming it is healthy and SYNCHRONIZED, Cluster will successfully failover to SQL16N2.

Now, try adding the availability replicas in the reverse order to the availability group, like this. After adding the replicas, check the properties of the availability group role to confirm the order of the preferred owner list:

When a health issue is detected on SQL16N1 (again shut down SQL Server on SQL16N1) we can see the preferred owner list priority is honored and Cluster attempts to failover to SQL16N3 first and successfully.

For more information see the following link on automatic failover:

Failover and Failover Modes (Always On Availability Groups)

Overview

In SQL Server 2012 and SQL Server 2014, encrypted databases could be added to an AlwaysOn availability group, but they could not be added using the New Availability Group wizard. Additionally, in the event of a failover, the encrypted database data could not be accessed. This is because the database master key in the availability group database was not compatible with the service master key on the other SQL Server instances hosting replicas for that availability group.

For more information on how to add encrypted databases to an availability group and make those databases highly available on all replicas in SQL Server 2012 or SQL Server 2014, see the AlwaysOnPro blog:

How to enable TDE Encryption on a database in an Availability Group

SQL Server 2016 adds support to the availability group wizard for adding encrypted databases and simplifies database access following availability group failover.

NOTE This improvement is part of support for adding the SQL Server Integration Services database (SSISDB) to an AlwaysOn availability group for high availability. For more information see

Always On for SSIS Catalog (SSISDB)

New Availability Group Wizard Adds Support for Encrypted Database

In SQL Server 2016, if you select an encrypted database for inclusion in an availability group, the New Availability Group wizard detects that the database is defined with a database master key and prompts the administrator for the database master key password.

After adding the database with a database master key, the wizard will verify the password during validation. When clicking Finish, credentials are created on each replica using the password of the database master key.

Added Support for Accessing Encrypted Availability Group Database After Failover

On creation, a database’s database master key is encrypted using the SQL Server service master key, which is unique to that SQL Server instance. When a database defined with a database master key is added to an availability group, upon failover to another replica, decryption is not possible for that database when brought online in the primary role on a different SQL Server instance with an incompatible service master key.

When SQL Server needs a database master key to decrypt or encrypt a key, SQL Server tries to decrypt the database master key with the service master key of the instance. If the decryption fails (which in the case following failover, would occur), SQL Server searches the credential store for master key credentials that have the same family GUID as the database for which it needs the master key. SQL Server then tries to decrypt the database master key with each matching credential until the decryption succeeds or there are no more credentials.

Beginning in SQL Server 2016, when you create an availability group containing an database containing a database master key, the New Availability Group Wizard executes procedure sp_control_dbmasterkey_password against each replica to create these credentials associated with the database master key.

EXEC sp_control_dbmasterkey_password @db_name = @dbName, @password = @pwd, @action = N”add”’,N’@dbName nvarchar(9),@password nvarchar(20)’,@dbName=N’encryptDB’,@password=N’1a2b3c4d5e6f7g8h9i0!’

In the event that this replica becomes the primary, and the database master key must be decrypted, SQL Server will first attempt to decrypt the key with the SQL Server instance service master key which will fail. Next, SQL Server will search for and find these credentials that can decrypt the database master key.

You can view the new credentials by querying sys.credentials and sys.master_key_passwords.

More Information About the Relationship between the SQL Server service master key and the database master key

SQL Server has two primary applications for keys: a service master key (SMK) generated on and for a SQL Server instance, and a database master key (DMK) used for a database. The SMK is automatically generated the first time the SQL Server instance is started and is used to encrypt a linked server password, credentials, and the database master key. The database master key is a symmetric key that is used to protect the private keys of certificates and asymmetric keys that are present in the database.

The database master key is a symmetric key used to protect the private keys of certificates and asymmetric keys that are present in the database.  To enable the automatic decryption of the master key, a copy of the key is encrypted by using the service master key and stored in both the database and in master. Typically, the copy stored in master is silently updated whenever the master key is changed.

Database Master Key credentials

When SQL Server needs a database master key to decrypt or encrypt a key, SQL Server tries to decrypt the database master key with the service master key of the instance. If the decryption fails, SQL Server searches the credential store for master key credentials that have the same family GUID as the database for which it needs the master key. SQL Server then tries to decrypt the database master key with each matching credential until the decryption succeeds or there are no more credentials.

In prior versions of SQL Server, if a database contain some encrypted columns protected by database master key and the database is added into an AlwaysOn availability group, the database in the new primary node cannot decrypt the database master key automatically after failover. In order to encrypt/decrypt data on the new primary node, you have to use OPEN MASTER KEY statement explicitly to open the database master key.

As the above picture show, When the database master key is created, the master key is encrypted by using the AES_256 algorithm and a user-supplied password. To enable the automatic decryption of the database master key, a copy of the key is encrypted by using the Service Master Key (SMK) and stored in both the database and in master database.

Since the Service Master Key is generated for a SQL Server instance, SQL Server Instance A and SQL Server Instance B have different service master key.

When failover occurs with the database, the secondary node (Node 2) became the primary node of the availability group, but the copy of database master key in Node 2’s database is encrypted by Service Key A instead of Service Key B. Thus after failover, the database in SQL Server Instance B cannot decrypt the database master key automatically.

For more information see the following resources:

For more information, please see

SQL Server Encryption

SQL Server and Database Encryption Keys (Database Engine)

CREATE MASTER KEY (Transact-SQL)

How to enable TDE Encryption on a database in an Availability Group

sp_control_dbmasterkey_password (Transact-SQL)

Written By:

Grant Carter, Senior Premier Field Engineer

Reviewed By:

Mark Weber – Principal Premier Field Engineer

Norm Eberly – Senior Premier Field Engineer

Charles Allard – Senior Premier Field Engineer

Nick Schueler – Senior Premier Field Engineer

Curt Matthews – Senior Escalation Engineer

Problem

You may discover on a readable secondary database that is part of an AlwaysOn Availability Group the data and transaction log file locations in sp_helpdb and sys.databases_files may display inaccurate results.

Scenario

Results from these DMVs and stored procedures may be unexpected when executing them against the readable secondary if the readable secondary was restored with the move option and placed on a different file path as compared to the file path of the primary replica.   For example, the primary replica may have its files on the E: drive and the secondary replica may have its files on the L: drive. In the example below, you will see that the database is on the E: drive, but it was manually restored using the move command to the L: drive.

On the primary replica:

create database [ag1db1]
on
(
       name = 'ag1db1_data'
       , filename = 'E:\SQL\ag1db1_data.mdf'
       , size = 20MB
       , maxsize = 20MB
       , filegrowth = 0
)
log on
(
       name = 'ag1db1_log'
       , filename = 'E:\SQL\ag1db1_log.ldf'
       , size = 20MB
       , maxsize = 20MB
       , filegrowth = 0
)
go

On the secondary replica:

restore database [ag1db1] from disk = '\\sql2014n1\Backup\ag1db1.bak'
with
       move 'ag1db1_data' to 'L:\SQL\ag1db1_data.mdf'
       , move 'ag1db1_log' to 'L:\SQL\ag1db1_log.ldf'
       , norecovery
       , stats=10
go

Based on the script above, you can see that the data and log paths are different between the primary and readable secondary replica. The primary files were place on the E:\SQL directory and the readable secondary files were placed on the L:\SQL directory.  On the readable secondary replica, the sp_helpdb command returns a file path that is the same as the primary replica and not on the L: drive as the database restore command specified using the move option.

sp_helpdb ag1db1
go

If sys.databases on the secondary replica is queried, the result will show the data and log files on the E: drive just as they did with sp_helpdb on the readable secondary replica which is different than the path they were restored to.

use ag1db1
go
select name, physical_name
from sys.database_files
go

The sys.master_files view will report accurate details about database file locations at the readable secondary replica. Unlike sp_helpdb and sys.databases_files, the sys.master_files view reflects the actual paths specified in the restore with move command as opposed to the file paths for the primary replica.

use master
go
select name, physical_name
from sys.master_files where database_id = db_id('ag1db1')
go

Reason for Behavior

sp_helpdb calls a stored procedure called sp_helpfile. sp_helpfile is a stored procedure that queries a view called sys.sysfiles.   sys.sysfiles gets its data from a system table called sys.sysprufiles. sys.syspruefiles is scoped at the database level. Because the secondary is a mirrored copy of the primary, the data in sys.sysprufiles reflects as it would on the primary, not the secondary, because of its scope. For this reason, the data on a readable secondary will always match the data on the primary replica.

If the availability group were to be failed over and the new readable secondary queried, it would be updated to appear as if its files were on the L: drive to reflect the new primary.

sys.database_files is a view that also uses sys.sysprufiles as its base data. Because of that, the behavior is the same as it is for sp_helpdb or sp_helpfile.

sys.master_files is a view that is scoped at the master database level that calls a system table called sys.sysbrickfiles. The metadata is maintained at the master database level and isn’t mirrored between the replicas like the database scoped system tables. For this reason, sys.master_files demonstrates the correct location of the data and log files for the database on that replica regardless of which replica is primary and which is secondary.

It is important to remember that in order to get the correct locations of where the data and transaction log files are located on a readable secondary replica, sys.master_files will give you the correct location. Relying on sys.database_files or sp_helpdb will give you an inaccurate result if the file locations between the primary and secondary replica are different.

If you have a Service Broker (SSB) application connecting to SQL Server using an AlwaysOn availability group listener, in the event of an unexpected failover, some messages may be lost or stuck in the transmission queue on the old primary (new secondary) after the failover. This could be an automatic failover or manual failover when the SSB application is not stopped.

Incoming (SSB) connections are severed during a availability group failover and those connections use the listener IP address which directs connections to the new primary replica. New incoming SSB connections, established through the listener, will go to the new replica. This is the correct and intended behavior.

The availability group fails over and the new inbound SSB connections reset to the new primary replica:

However, outgoing SSB connections from the old primary replica are left open after a failover, still in session with the initiator, and remain so because the IP addresses are still valid. New outgoing SSB connections are established from the new replica to the initiator, but because the connections from the old primary are still open, incoming SSB messages are still sent to the old primary.

Since Service Broker components are not contained within the availability group, special handling is required to handle the Service Broker connections when the availability group moves to a different replica instance. If you feel the post failover handling for the Service Broker connection is not feasible to be built into your application, please submit a Connect item (http://connect.microsoft.com/sql) so that we are able to understand the community need for this functionality. As an alternative to modifying the code for your application, you could choose to use a SQL Server Failover Cluster Instance which will prevent the issue described above, due to the IP Address remaining the same on failover.

Troubleshooting

If, after the failover, you run the following query on the old primary SSB TARGET and see the value for is_accepted = 0 then you know you have encountered the issue. You will also likely have two or more connections returned here.

–look for is_accept = 0

select is_accept, * from sys.dm_broker_connections with(nolock)

Another side effect are messages stuck in the sys.transmission_queue on the old primary SSB TARGET.

If Profiler trace is run on the old primary SSB TARGET, you would see the following error:

The message has been dropped because the service broker in the target database is unavailable: ‘The database is in read-only mode.’

SQL Server XEvent ‘broker_message_undeliverable’ can also be collected using SQL Server extended events and will also report that messages cannot be delivered. For more information on using SQL Server extended events, see:

Create an Extended Events Session Using the Wizard (Object Explorer)

Workaround

Host the primary replica on a SQL Failover Cluster Instance (SQL FCI). In the event of a failover, SQL Server restarts and the initiator will re-establish connectivity with the availability group listener and the TARGET will re-establish connections with the initiator. There will be no residual broker connections or in-flight transactions that cannot be resolved.

In this scenario, you still have a secondary replica hosted on a standalone instance of SQL Server which provides another means of high availability.

Maintenance

When failing manually over between availability group replicas for rolling upgrade purposes, stop the Service Broker Application before doing the manual availability group failover to prevent any in-flight transactions from getting stuck.

Problem Large Send Queue

You discover that the log send queue for a given availability database has grown very large, which threatens your Recovery Point Objective (RPO) and the transaction log has grown very large on the primary replica, possibly threatening to fill the drive. The cause may have been for various reasons: you discover that the availability database synchronization was inadvertently suspended by an administrator, or even by the system and the problem was not identified until now.

Scenario 1 The database is not large

If it is reasonable to re-initialize the secondary replica copy of the database using a backup and restore of the database whose log file has grown very large, then you can allow synchronization to proceed to drain the log send queue or if the size of the transaction log is much larger than the database itself, remove the database from the availability group, shrink the log file and add it back into the availability group, re-seeding your secondary replicas with the newly added database.

1. In SQL Server Management Studio (SSMS) remove the database from the availability group on the primary replica.
2. Put the database in simple recovery mode on the primary replica. This will allow you to shrink the log file.
3. Shrink the log file down to a reasonable size. To shrink the log file, in SSMS’ Object Explorer right-click the database, choose Tasks and then Shrink and then Files. For File type choose Log and click OK.
4. Once log file has ben shrunk, configure the database for full recovery mode and then right-click the availability group, choose Add Database and use the wizard to add the database back and re-initialize the secondary replica with it.

Scenario 2 The database is very large, making re-initialization prohibitive to the secondary replicas

If the data portion of the database is very large combined with the remoteness of one or more secondary’s, it may not be reasonable to use Scenario 1. Another option is to apply the log backups onto the database at the secondary, to ‘catch up’ the database at the secondary. Following are the steps.

   1. Determine the transaction logs that must be applied. Query the last_hardened_lsn for the database on the secondary. Connect to the secondary and run the following supplying the database name as the only predicate:

select distinct dcs.database_name, ds.last_hardened_lsn from 

sys.dm_hadr_database_replica_states ds join sys.dm_hadr_database_replica_cluster_states dcs

on ds.group_database_id=dcs.group_database_id

where dcs.database_name=’AutoHa-sample’

You will get results like this.

   2. Query the log backups in MSDB to find what log backup the secondary LSN falls in the range of.

select name, backup_set_id, backup_start_date, backup_finish_date, first_lsn, last_lsn from msdb..backupset

where first_lsn<‘93000012832800001′ and last_lsn>’93000012832800001’

We see one row returned, we need to apply all logs starting with this log on the secondary replica:

Alternative to find the first transaction log backup Query sys.dm_hadr_database_replica_states.last_hardened_time for the database (same query as above). Use File Explorer to view the transaction log backup files and order by modified date. Compare the last_hardened_time with the backup modified dates to determine what transaction log backup you should start with, selecting the transaction log backup whose modified date is just beyond the last_hardened_time. Query that transaction log to verify the last_hardened_time falls in that transaction log backup:

restore headeronly from disk= N’f:\backups\AutoHa-sample_backup_2017_01_20_194401_3869314.bak’

   3. Take database out of the availability group on the secondary. Connect to the secondary replica and execute the following to take the database out of the secondary replica.

alter database [AutoHa-sample] set hadr off

On the secondary the database will be in Restoring and ready to apply logs to.

   4. Apply transaction logs to database on the secondary replica Apply the transaction log backups. Here is a view of the log backup files.  We identified the one called ‘AutoHa-sample_backup_2017_01_20_194401_3869314’ as containing our secondary database last_hardened_lsn.

Begin restoring with that log being sure to restore with ‘no recovery’ all the transaction log backups.

restore log [AutoHa-sample] from disk=’\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_194401_3869314.trn’ with norecovery
go
restore log [AutoHa-sample] from disk=’\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_194500_8646900.trn’ with norecovery
go
restore log [AutoHa-sample] from disk=’\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_194600_9809513.trn’ with norecovery
go
restore log [AutoHa-sample] from disk=’\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_200000_6780254.trn’ with norecovery
go

   5. Add the database back into the availability group on the secondary and resume synchronization. Connect to the secondary replica and execute the following to add the database back into the availability group and resume sync.

alter database [AutoHa-sample] set hadr availability group = [contoso-ag]
go
alter database [AutoHa-sample] set hadr resume
go

You can view the availability group database on the secondary to confirm it is added back and synchronized.

Problem Unable to Connect to Azure availability group listener

Creating an availability group in order to make your application highly available when running on Azure virtual machines (IaaS) is very popular. The availability group listener requires special configuration steps in Azure as opposed to running on premise. 

For more information on step by step configuration of Azure availability group listener, see the following article for Azure resource managed availability group.

Configure an internal load balancer for an Always On availability group in Azure

The following article will assist in troubleshooting the failure to connect to the availability group listener.

NOTE This troubleshooter focuses on an availability group listener configured with resource managed internal load balancer, which is the most popular type of listener currently deployed.

Testing the Azure availability group listener

To test an Azure availability group listener, do not test the listener connectivity on the primary replica. Instead, test listener connectivity to the primary replica using the listener, from the secondary replica. We know the secondary replica already has certain connectivity with the primary and that there are SQL Server client tools installed on it, that can be used for testing connectivity to the listener. Use a tool like sqlcmd for connectivity testing.

Troubleshooting Failed Azure listener connectivity

Familiarizing yourself with the article that steps through the Azure listener creation and configuration will help you understand the major pieces of the Azure listener configuration that need to be investigated to determine root cause. The majority of all listener connection issues are a result of incorrect configuration of one of the following:

• Firewall or network security group configuration

• IP Clustered resource creation and configuration (See section of article ‘Configure the cluster to use the load balancer IP address’)

• Internal load balancer creation and configuration (See section of article ‘Create and configure the load balancer in the Azure portal)’

Turn off Windows Firewall on the server hosting the primary replica (Firewall and NSG configuration)

The simplest way to ensure Windows Firewall is not blocking connectivity to the listener is to turn off Windows Firewall on the Windows server hosting the primary replica of the availability group, that which you are trying to connect to with your listener connectivity test.

If you find that connectivity succeeds, turn Firewall back on, and be sure to allow inbound connection through ports 1433 (port SQL Server is listening on) and 59999 (probe port).

Investigate Network Security Group (NSG) for rules that block probe port or SQL Server port connectivity (Firewall and NSG configuration)

It is possible that a network security group rule is preventing connectivity to the virtual machine. For example, the following network security group has an inbound rule allowing SQL connectivity but it is overruled by a rule whose priority value is lower.

For more information review the following section ‘Configure a Network Security Group inbound rule for the VM’ in this article:

Connect to a SQL Server Virtual Machine on Azure (Resource Manager)

Verify RHS.EXE is listening on probe port (59999) (IP Clustered Resource creation and configuration)

On the server hosting the primary replica, verify that RHS.EXE process is listening on the probe port you configured the internal load balancer with.

View the output with notepad and search for a process listening on port 59999 (probe port):

Use Task Manager to verify that the process listening on 59999 is RHS.EXE. If some other process is already listening on port 59999, then it will be necessary to reconfigure both the IP resource’s probe port and the internal load balancer probe port to a different value so there is not a conflict.

Review the internal load balancer configuration (Load Balancer creation and configuration)

Using the article on creating the Azure availability group listener, step through the configuration steps 1-4 listed below and use the Azure portal to visually verify the configuration of each step below is correct:

1. Create the load balancer and configure the IP address
2. Configure the backend pool
3. Create a probe
4. Set the load balancing rules

If a visual of the internal load balancer checks out, you can use network tracing to verify that the load balancer is pinging the probe port on the server hosting the primary replica. On the server hosting the primary replica, run netsh from the command line for 30 seconds:

netsh trace start capture=yes tracefile=d:\nettrace.etl maxsize=200 filemode=circular overwrite=yes report=no

To stop the trace run

netsh trace stop

Open the .ETL file using a tool like Network Monitor and filter for probe port activity. No traffic on the probe port of the virtual machine hosting the primary replica indicates one of two possibilities:

1. The internal load balancer is not configured correctly.
2. There is a network security group, firewall or corporate polcy that is interfering with the probe port traffic.

Here we can see that when we filter on port 59999 we see activity which indicates that the load balancer appears to be performing correctly.

Cluster Quorum File Share Witness Should Not be Part of a DFS Namespace

When deploying AlwaysOn availability groups you may decide to add a vote to Windows Cluster quorum by configuring a File Share Witness. A requirement when configuring that File Share Witness is that it is not part of a Distributed File System (DFS) namespace. Adding a file share witness which is part of a DFS can result in split brain and ultimately data loss. It is documented in the section ‘Requirements and recommendations for clusters using Node and File Share Majority’ of Failover Cluster Step-by-Step Guide: Configuring the Quorum in a Failover Cluster

• Do not use a file share that is part of a Distributed File System (DFS) Namespace.

Avoid Data Loss In Your Availability Group Databases

If a File Share Witness for your Windows Cluster is incorrectly configured as part of a DFS, in the event of a communication between Windows Cluster nodes hosting availability group replicas, quorum may be arbitrated independently, resulting in split brain condition in the Windows Cluster.

The result is two or more availability group replicas will have independent views of the availability group state and may attain the primary role simultaneously, or automatic failover can occur to an availability replica that is not in a truly synchronized state. These are states and behaviors that can lead to the loss of data in your availability group databases.

Microsoft SQL Server support has worked with several customers who have experienced data loss due to split brain or the automatic failover of availability groups that were not synchronized as a result of this configuration.

You may find that one or more availability group databases is reported ‘Not Synchronizing / Recovery Pending’ on the primary replica or ‘Not Synchronizing’ on one of the secondary replicas. Despite this, your availability group replicas report they are in the primary role or secondary role,

This may occur if SQL Server is unable to access the database log files of your availability group database(s). The following blog describes the symptoms, cause and resolution for this problem when encountered on the primary or secondary replicas of your availability group.

Symptoms of Availability Group Database on Primary Replica

Primary Replica Database Reported Not Synchronizing / Recovery Pending

In SQL Server Management Studio’s Object Explorer, on the primary replica, the availability group database may be reported as ‘Not Synchronizing/Recovery Pending:

Primary Replica - Attempts to Access Database or Perform DML Fail

Attempts to access the affected database will fail. For example, an attempt to INSERT into a table in that database reports 9001, ‘The log for database…is not available.’

SQL Server Management Studio will report the database in ‘Not Synchronizing / Recovery Pending’ state.

Cause SQL Server unable to access database log file

This availability group database state could have been caused because SQL Server could not access the log file and or data file(s). In the event the log file is or was inaccessible, SQL Server will offline the database and report the database in Recovery Pending state. In this scenario, the database is defined in an availability group, and therefore will be reported in SQL Server Management Studio as ‘Not Synchronizing / Recovery Pending. Check the SQL Server error log to determine if this is the cause for reporting the Recovery Pending state. You will find a message 9001 reported that the ‘log for database xxx is not available.’

2017-07-13 06:42:44.74 spid1s      Error: 17053, Severity: 16, State: 1.
2017-07-13 06:42:44.74 spid1s      SQLServerLogMgr::LogWriter: Operating system error 21(The device is not ready.) encountered.
2017-07-13 06:42:44.74 spid1s      Write error during log flush.
2017-07-13 06:42:44.75 spid51      Error: 9001, Severity: 21, State: 4.
2017-07-13 06:42:44.75 spid51      The log for database 'testdb' is not available. Check the event log for related error messages. Resolve any errors and restart the database.
2017-07-13 06:42:44.75 spid51      State information for database 'testdb' - Hardened Lsn: '(34:840:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:42:44.75 spid23s     Always On Availability Groups connection with secondary database terminated for primary database 'testdb' on the availability replica 'SN2' with Replica ID: {10b9a12b-afed-48f3-b79c-9ecc5cada754}. This is an informational message only. No user action is required.
2017-07-13 06:42:44.75 spid51      Database testdb was shutdown due to error 9001 in routine 'XdesRMFull::CommitInternal'. Restart for non-snapshot databases will be attempted after all connections to the database are aborted.
2017-07-13 06:42:44.77 spid18s     State information for database 'testdb' - Hardened Lsn: '(34:840:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:42:44.86 spid18s     Error: 17053, Severity: 16, State: 1.
2017-07-13 06:42:44.86 spid18s     fcb::close-flush: Operating system error (null) encountered.

Resolve Primary Replica Database Not Synchronizing / Recovery Pending

A thought might be to suspend and resume the availability group database but this will not solve this issue. Resume does not initiate recovery on the database, which is what is necessary to bring the database back into an online state.

Once you have confirmed the database files are accessible, use the ALTER DATABASE SET ONLINE command to initiate the recovery of the database which should bring the database online.

use master
go
alter database testdb set online
go

Here we online the database and now the database is once again online and in a Synchronized state.

The error log shows SQL Server go through recovery and re-establish connection with secondary replica.

2017-07-13 06:57:41.97 spid58      Setting database option ONLINE to ON for database 'testdb'.
2017-07-13 06:57:41.97 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM
2017-07-13 06:57:41.97 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:57:42.00 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:57:42.00 spid58      Starting up database 'testdb'.
2017-07-13 06:57:42.17 spid58      2 transactions rolled forward in database 'testdb' (9:0). This is an informational message only. No user action is required.
2017-07-13 06:57:42.27 spid30s     0 transactions rolled back in database 'testdb' (9:0). This is an informational message only. No user action is required.
2017-07-13 06:57:46.98 spid36s     Always On Availability Groups connection with secondary database established for primary database 'testdb' on the availability replica 'SN2' with Replica ID: {10b9a12b-afed-48f3-b79c-9ecc5cada754}. This is an informational message only. No user action is required.

Symptoms of Availability Group Database on Secondary Replica

Secondary Replica Database Reported Not Synchronizing

If SQL Server hosting the secondary replica is unable to access availability group database files, database is marked in the Not Synchronizing state. Note that sys.dm_hadr_database_replica_states and sys.databases still reports the database as Online.

Secondary Replica – Attempts to Suspend and Resume synchronization Fail

This might be your first instinct, when seeing an availability group database reported as ‘Not Synchronizing’ in SSMS. Even if the database log file becomes accessible again, synchronization is not resumed. Attempting to Suspend and then Resume will both appear to succeed:

However, querying sys.dm_hadr_database_replica_states will report that the database still is not synchronizing and the database is still reported Not Synchronizing in SSMS Object Explorer.

Cause SQL Server unable to access database log file

Again, confirm if the current availability group issue resulted when SQL Server was unable to access the database files. The error log on the secondary reports the problem during redo (term for availability group ongoing recovery of secondary availability group database) accessing log file and so the database is suspended.

2017-07-13 07:07:20.01 spid1s      Error: 17053, Severity: 16, State: 1.
2017-07-13 07:07:20.01 spid1s      SQLServerLogMgr::LogWriter: Operating system error 21(The device is not ready.) encountered.
2017-07-13 07:07:20.01 spid1s      Write error during log flush.
2017-07-13 07:07:20.01 spid25s     Error: 9001, Severity: 21, State: 5.
2017-07-13 07:07:20.01 spid25s     The log for database 'testdb' is not available. Check the event log for related error messages. Resolve any errors and restart the database.
2017-07-13 07:07:20.19 spid25s     Error: 3313, Severity: 21, State: 2.
2017-07-13 07:07:20.19 spid25s     During redoing of a logged operation in database 'testdb', an error occurred at log record ID (34:896:1). Typically, the specific failure is previously logged as an error in the Windows Event Log service. Restore the database from a full backup, or repair the database.
2017-07-13 07:07:20.38 spid25s     Always On Availability Groups data movement for database 'testdb' has been suspended for the following reason: "system" (Source ID 2; Source string: 'SUSPEND_FROM_REDO'). To resume data movement on the database, you will need to resume the database manually. For information about how to resume an availability database, see SQL Server Books Online.

Resolve Secondary Replica Database Not Synchronizing

You cannot ONLINE a secondary replica availability group database Since a secondary replica availability group database is really in an ongoing restoring state, it cannot be set Online like the primary replica. Remember it is already in the Online role, according to the DMVs.

Solution Restart the SQL Server instance hosting the secondary replica. This will initiate recovery on all databases including availability group databases, and so long as SQL Server can access all the database files, the availability group database affected should recover successfully and resume synchronization with the primary replica.

You may find that one or more availability group databases is reported ‘Not Synchronizing / Recovery Pending’ on the primary replica or ‘Not Synchronizing’ on one of the secondary replicas. Despite this, your availability group replicas report they are in the primary role or secondary role,

This may occur if SQL Server is unable to access the database log files of your availability group database(s). The following blog describes the symptoms, cause and resolution for this problem when encountered on the primary or secondary replicas of your availability group.

Symptoms of Availability Group Database on Primary Replica

Primary Replica Database Reported Not Synchronizing / Recovery Pending

In SQL Server Management Studio’s Object Explorer, on the primary replica, the availability group database may be reported as ‘Not Synchronizing/Recovery Pending:

Primary Replica - Attempts to Access Database or Perform DML Fail

Attempts to access the affected database will fail. For example, an attempt to INSERT into a table in that database reports 9001, ‘The log for database…is not available.’

SQL Server Management Studio will report the database in ‘Not Synchronizing / Recovery Pending’ state.

Cause SQL Server unable to access database log file

This availability group database state could have been caused because SQL Server could not access the log file and or data file(s). In the event the log file is or was inaccessible, SQL Server will offline the database and report the database in Recovery Pending state. In this scenario, the database is defined in an availability group, and therefore will be reported in SQL Server Management Studio as ‘Not Synchronizing / Recovery Pending. Check the SQL Server error log to determine if this is the cause for reporting the Recovery Pending state. You will find a message 9001 reported that the ‘log for database xxx is not available.’

2017-07-13 06:42:44.74 spid1s      Error: 17053, Severity: 16, State: 1.
2017-07-13 06:42:44.74 spid1s      SQLServerLogMgr::LogWriter: Operating system error 21(The device is not ready.) encountered.
2017-07-13 06:42:44.74 spid1s      Write error during log flush.
2017-07-13 06:42:44.75 spid51      Error: 9001, Severity: 21, State: 4.
2017-07-13 06:42:44.75 spid51      The log for database 'testdb' is not available. Check the event log for related error messages. Resolve any errors and restart the database.
2017-07-13 06:42:44.75 spid51      State information for database 'testdb' - Hardened Lsn: '(34:840:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:42:44.75 spid23s     Always On Availability Groups connection with secondary database terminated for primary database 'testdb' on the availability replica 'SN2' with Replica ID: {10b9a12b-afed-48f3-b79c-9ecc5cada754}. This is an informational message only. No user action is required.
2017-07-13 06:42:44.75 spid51      Database testdb was shutdown due to error 9001 in routine 'XdesRMFull::CommitInternal'. Restart for non-snapshot databases will be attempted after all connections to the database are aborted.
2017-07-13 06:42:44.77 spid18s     State information for database 'testdb' - Hardened Lsn: '(34:840:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:42:44.86 spid18s     Error: 17053, Severity: 16, State: 1.
2017-07-13 06:42:44.86 spid18s     fcb::close-flush: Operating system error (null) encountered.

Resolve Primary Replica Database Not Synchronizing / Recovery Pending

A thought might be to suspend and resume the availability group database but this will not solve this issue. Resume does not initiate recovery on the database, which is what is necessary to bring the database back into an online state.

Once you have confirmed the database files are accessible, use the ALTER DATABASE SET ONLINE command to initiate the recovery of the database which should bring the database online.

use master
go
alter database testdb set online
go

Here we online the database and now the database is once again online and in a Synchronized state.

The error log shows SQL Server go through recovery and re-establish connection with secondary replica.

2017-07-13 06:57:41.97 spid58      Setting database option ONLINE to ON for database 'testdb'.
2017-07-13 06:57:41.97 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM
2017-07-13 06:57:41.97 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:57:42.00 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM'
2017-07-13 06:57:42.00 spid58      Starting up database 'testdb'.
2017-07-13 06:57:42.17 spid58      2 transactions rolled forward in database 'testdb' (9:0). This is an informational message only. No user action is required.
2017-07-13 06:57:42.27 spid30s     0 transactions rolled back in database 'testdb' (9:0). This is an informational message only. No user action is required.
2017-07-13 06:57:46.98 spid36s     Always On Availability Groups connection with secondary database established for primary database 'testdb' on the availability replica 'SN2' with Replica ID: {10b9a12b-afed-48f3-b79c-9ecc5cada754}. This is an informational message only. No user action is required.

Symptoms of Availability Group Database on Secondary Replica

Secondary Replica Database Reported Not Synchronizing

If SQL Server hosting the secondary replica is unable to access availability group database files, database is marked in the Not Synchronizing state. Note that sys.dm_hadr_database_replica_states and sys.databases still reports the database as Online.

Secondary Replica – Attempts to Suspend and Resume synchronization Fail

This might be your first instinct, when seeing an availability group database reported as ‘Not Synchronizing’ in SSMS. Even if the database log file becomes accessible again, synchronization is not resumed. Attempting to Suspend and then Resume will both appear to succeed:

However, querying sys.dm_hadr_database_replica_states will report that the database still is not synchronizing and the database is still reported Not Synchronizing in SSMS Object Explorer.

Cause SQL Server unable to access database log file

Again, confirm if the current availability group issue resulted when SQL Server was unable to access the database files. The error log on the secondary reports the problem during redo (term for availability group ongoing recovery of secondary availability group database) accessing log file and so the database is suspended.

2017-07-13 07:07:20.01 spid1s      Error: 17053, Severity: 16, State: 1.
2017-07-13 07:07:20.01 spid1s      SQLServerLogMgr::LogWriter: Operating system error 21(The device is not ready.) encountered.
2017-07-13 07:07:20.01 spid1s      Write error during log flush.
2017-07-13 07:07:20.01 spid25s     Error: 9001, Severity: 21, State: 5.
2017-07-13 07:07:20.01 spid25s     The log for database 'testdb' is not available. Check the event log for related error messages. Resolve any errors and restart the database.
2017-07-13 07:07:20.19 spid25s     Error: 3313, Severity: 21, State: 2.
2017-07-13 07:07:20.19 spid25s     During redoing of a logged operation in database 'testdb', an error occurred at log record ID (34:896:1). Typically, the specific failure is previously logged as an error in the Windows Event Log service. Restore the database from a full backup, or repair the database.
2017-07-13 07:07:20.38 spid25s     Always On Availability Groups data movement for database 'testdb' has been suspended for the following reason: "system" (Source ID 2; Source string: 'SUSPEND_FROM_REDO'). To resume data movement on the database, you will need to resume the database manually. For information about how to resume an availability database, see SQL Server Books Online.

Resolve Secondary Replica Database Not Synchronizing

You cannot ONLINE a secondary replica availability group database Since a secondary replica availability group database is really in an ongoing restoring state, it cannot be set Online like the primary replica. Remember it is already in the Online role, according to the DMVs.

Solution Restart the SQL Server instance hosting the secondary replica. This will initiate recovery on all databases including availability group databases, and so long as SQL Server can access all the database files, the availability group database affected should recover successfully and resume synchronization with the primary replica.

Multisite availability groups allow customers to deploy copies of business data across multiple sites, for disaster recovery and/or for reporting purposes, offering near real-time changes available to the copies of the production data at remote locations.

If secondary replicas are hosted greater distances from the primary replica, network latency can begin to impact availability group synchronization’s ability to keep the secondary up to date with the changes made at the primary replica.

Symptom - Sustained or Growing Log Send Queue

When network latency becomes an issue the most common symptom you will observe is sustained or growing log send queue. Simply defined, log send queue is the changes stored in the transaction log of your availability group database on the primary replica that have not yet arrived and hardened to the transaction log of the availability group on the secondary replica.

Why should you care? The log send queue represents all the data that can be lost from a catastrophic failure. If your primary replica was lost in a sudden disaster, and you were forced to fail over to the secondary replica where these changes have not yet arrived, your business would incur data loss.

Log send queue is a per-database measurement. Here are ways you can actively monitor your availability group database log send queue.

Add the Log Send Queue size (KB) column in AlwaysOn Dashboard In SQL Server Management Studio Object Explorer, right click on your availability group and choose Show Dashboard. The dashboard will appear in the right pane. Right-click the column header above the list of databases in your availability group and choose to add Log Send Queue Size (KB).

The column is added and reports the current queue size in KB.

Add the SQLServer:Database Replica:Log Send Queue Counter On the secondary replica, add the SQLServer:Database Replica:Log Send Queue counter for the database instance whose log send queue you are monitoring.

Measure Network Latency Impact Using Performance Monitor

In the following demo we measure the impact of network latency on log send queue growth. We can use SQL Server performance monitor counters on the primary and secondary replicas during synchronization of a load which results in log send queue growth. Compression is disabled during this load test in order to put more load on the network.

On the secondary replica, launch Performance Monitor and add the following counters:

SQLServer:Database Replica:Log Bytes Received/sec for appropriate database instance

SQLServer:Database Replica:Recovery Queue for appropriate database instance

On the primary replica, launch Performance Monitor and add the following counters:

SQLServer:Databases:Log Bytes Flushed/sec for appropriate database instance

Network Interface:Sent Bytes/sec for appropriate adapter instance

Once you have Performance Monitor running, initiate the load that leads to log send queuing, and monitor the counters.

Review the performance monitor counters on the secondary, we can see Log Bytes Received/sec for our availability group database is averaging 2.1 mb/sec. Note that the Log Send Queue for the database is continually growing meaning that there are a growing number of changes that have occurred at the primary replica that are stored in the transaction log there that have not yet arrived and hardened on the transaction log of the local database on the secondary replica.

Now, on the primary replica, check the availability group database’s Log Bytes Flushed/sec. We see that our load is generating and hardening on average 7.1 mb/sec to the local transaction log.

Clearly, if we are generating over 7 mb/sec of logged changes at the primary but only receiving 2 mb/sec of changes at the secondary, we can expect log send queue growth. These observations can lead us to begin to consider network latency as a possible cause for our growing log send queue.

On the primary, check the Network Interface:Bytes Sent/sec for the correct network adapter instance. On average, the primary is sending 2.2 mb/sec, and we can expect that nearly all of that is to the secondary replica where we observed comparable Log Bytes Received/sec.

Measure your network throughput independently of SQL Server

In order to better understand how fast an application can push changes to the remote server, use a third party network bandwidth performance tool, like iPerf or NTttcp. The following demo is using iPerf on the same operating environment as the prior demo.

These tools use a default port to communicate with each other, so it may be necessary to open that port, in the case of iPerf, it uses 5201 by default.

IMPORTANT It is important to configure the tool to use a single connection to send data, since SQL Server uses a single connection to send synchronization changes from the primary to the secondary replica.

Test network throughput with iPerf

On the remote secondary replica, launch iPerf with the –s parameter and it will begin to listen for a connection over port 5201.

iperf3 –s

On the SQL Server hosting the primary replica, launch iPerf using the following parameters, and it will begin a transmission with the remote SQL Server.

Here are the parameters.

-c ip address of destination server

–P 1 Use a single connection in the test

–t 10 Perform 10 one second transfers

Here is the command line;

iperf –c <<secondaryipaddress>> –t 10 –P 1

iPerf is reporting it can send between 2.12 and 2.50 mb/sec to the remote secondary server. This rate is very comparable to the rates we observed above in our load test using SQL Server.

Testing a server that is closer to your primary replica can provide a useful foil when doing this test. Below, I run the same test from the primary this time to a destination server in the same data center and subnet as the primary server. Here are the test results, noticeably higher throughput when the destination server is in closer proximity and same subnet.

Test network throughput with NTttcp

Below is a demonstration of using NTttcp.exe in the same environment as above.

On the remote secondary replica (the receiver), launch NTttcp as follows, here –m 1 is for single threaded, and the IP address of the local receiver server.

NTttcp.exe –r –m 1,0,10.1.0.4 –rb 2M –a 16

On the primary replica (the sender), initiate the test by executing the following, again supplying the IP address of the receiver. This test is for 10 seconds (-t 10):

NTttcp.exe –s –m 1,0,10.1.0.4 –l 128k –a 16 –t 10

The results report about 2.7 mb/sec which is similar to the testing results obtained from iPerf and from our performance monitor log.

In Conclusion

This testing can help you to determine if network latency is impacting log send queue for remote asynchronous replicas in your availability group. Log send queuing can impact your recovery point objective (RPO) and as described earlier in this blog translate to potential data loss.

For more information on recovery point objective and the potential for data loss see

Monitor Performance for AlwaysOn Availability Groups

If you find that network latency is having an impact, consider using netsh or another network trace tool to check for retransmits, dropped packets, etc.

Workloads that rely on auto-commit (each operation like an INSERT has an implied, corresponding begin and commit transaction) and perform a large number of small transactions may take significantly longer to complete when run against availability group databases configured for synchronous commit. Aside from an issue with resource latency (IO, CPU or network) this performance impact may be expected.

Synchronous Commit Emphasizes Minimizing Data Loss Over Performance

Synchronous commit availability mode is desirable when minimizing data loss is paramount for your high availability solution and prioritizes Recovery Point Objective over performance.

In the following link, it says this about synchronous commit.

Availability Modes (Always On Availability Groups)

Synchronous-commit mode emphasizes high availability over performance, at the cost of increased transaction latency. Under synchronous-commit mode, transactions wait to send the transaction confirmation to the client until the secondary replica has hardened the log to disk.

This blog will demonstrate how to measure the performance impact on a high transaction rate workload that is dependent on auto-commit behavior in SQL Server and how explicitly defining transactions in that same workload can mitigate the impact and provide better performance.

The Transaction Must be Written at the Primary and The Synchronous Secondary To Commit

To illustrate the impact of synchronous commit, take a simple INSERT command initiated on a database at the primary replica. That insert must be hardened on the transaction log file of the primary replica and of the transaction log file on the secondary replica and then acknowledgement must arrive from both replicas before SQL Server can commit the transaction. When performing hundreds of these commands a second in a workload, the impact performance can be significant.

For more internal details on how synchronous commit works, see the following blog:

AlwaysON – HADRON Learning Series:  How does AlwaysON Process a Synchronous Commit Request

Demo – Measure Workload Performance in Synchronous Commit

Let’s demonstrate the impact of auto-commit mode when executing a workload against an availability group database configured for synchronous commit, and how to measure the throughput.

Enable Performance Monitor Counters

Enable the following counters in performance monitor on the primary replica:

For availability group database agdb in our example:
    SQL Server:Databases:Log Bytes Flushed/sec
    SQL Server:Databases:Log Flushes/sec
    SQL Server:Databases:Write Transactions/sec
    SQL Server:Database Replica:Transaction Delay

Run the workload, observe execution time, estimate transaction performance

The following workload inserts 10,000 rows 10 times, and uses auto-commit behavior, which will result in 100,000 transactions. It will report the average time it took to insert 10,000 rows in seconds and the average time it took to insert a single row in milliseconds.

use agdb
go

declare @y int=0
declare @x int=0
declare @starttime datetime
declare @endtime datetime
create table test (col int)
create table results (duration float)

while @x<10
        begin
        Set @starttime =getdate()
                     while @y<10000

                    begin
                            insert test (col)
                            select @y
                            set @y=@y+1
                      end                                      
        set @endtime =getdate()
        insert results (duration)
        select datediff(millisecond,@starttime, @endtime)/1000.0
        set @x=@x+1
        set @y=0
        end

select * from results
select avg(duration) as 'avg duration in sec for 10,000 INSERTs', avg(duration)/10 as 'avg duration in ms for each INSERT' from results

drop table results
drop table test

After we create our availability group and database, and configure it for synchronous commit with a single secondary replica, we run our workload. The report is that our average INSERT took 4.4 ms to complete the round trip, getting hardened on the log file at the primary and secondary replicas. The batch took 7 minutes 50 seconds to complete.

HADR_SYNC_COMMIT waittype reports synchronous commit latency

HADR_SYNC_COMMIT wait type measures the latency when performing transactions in an availability group database when configured for synchronous commit. To reset the instance’s waittype counter values (they accumulate over time), execute the following to clear the data:

DBCC SQLPERF('sys.dm_os_wait_stats' , CLEAR)

Test to demonstrate the severity of this waittype on the instance. There is a large number of these waits reported.

Review the performance monitor counters

Let’s review the performance monitor counters collected during the batch. Log Bytes Flushed/sec for the database is pretty low, less than 1 mb/sec.

Compare Log Flushes/sec to Write Transactions/sec For every write transaction performed, there is a flush to disk. This is what we would expect in auto-commit mode. Here we see 211 transactions are completing every second and for each transaction there is a corresponding flush to disk, on the database log file at the primary replica and on the database log file at the secondary replica.

Confirm estimated transaction time using performance monitor

The workload reported ‘avg duration for each transaction in MS’ as 4.4 ms.  We can use performance monitor to make a second confirmation calculation that this is accurate: divide one second by an average of 211 Write Transactions performed in a second, which is 1/211=.0047 seconds or 4.7 ms. This number is very similar to the number that our workload results reported. This is the time it took to capture the log change, deliver it to the secondary, write it to the log file on the primary and secondary and report back to the client that it is complete.

Do not Depend on Auto-Commit - Define Explicit Transactions

The batch has been modified with an explicit transaction in the workload in such a way that 10,000 INSERT operations will be committed together. This is much more efficient considering the requirements of synchronous commit, that each transaction requires the hardening of the operation to the log files of each replica and the network round trip needed to accomplish that work. Note the BEGIN TRAN and COMMIT TRAN commands added to the batch and commented.

Run the workload using explicit transactions, observe execution time, estimate transaction performance

use agdb
go

declare @y int=0
declare @x int=0
declare @starttime datetime
declare @endtime datetime
create table test (col int)
create table results (duration float)

while @x<10
        begin
        Set @starttime =getdate()
         begin tran                            --BEGIN 10000 row transaction
                     while @y<10000

                    begin
                            insert test (col)
                            select @y
                            set @y=@y+1
                      end                                      
         commit tran                            --COMMIT 10000 row transaction
        set @endtime =getdate()
        insert results (duration)
        select datediff(millisecond,@starttime, @endtime)/1000.0
        set @x=@x+1
        set @y=0
        end

select * from results
select avg(duration) as 'avg duration in sec for 10,000 INSERTs', avg(duration)/10 as 'avg duration in ms for each INSERT' from results

drop table results
drop table test

Our results are almost instantaneous. The results completed in a couple seconds!

HADR_SYNC_COMMIT waittype reports synchronous commit latency

Having reset the SQL Server wait counters using DBCC SQLPERF (described earlier) we see HADR_SYNC_COMMIT waits are significantly lower.

Review the performance monitor counters

Reviewing performance monitor, Log Bytes Flushed/sec spiked to 10.4 mb/sec versus our earlier test in which it averaged less than 1 mb/sec. This shows how much more throughput SQL Server is able to accomplish when not committing a few bytes for individually small transactions.

When comparing Log Flushes/sec to Write Transactions/sec the correlation is not as clear. During batch execution, Log Flushes/sec spike to 181, and Write Transactions are much lower, since the workload itself performed only 10 explicit transactions instead of 100,000.

Conclusion

Synchronous commit can introduce latency to your high transaction workloads, this can be especially impactful in environments that must complete large ETL batches on the weekends or overnight in order to prepare the data for the next business day. Using explicit transactions can optimize workload performance and still take advantage of the strengths synchronous commit offers (eliminate data loss).