12 Tips to Optimize an Exchange 2007 Infrastructure

Posted on by

 

(source : msexchange.org)

Introduction

Let’s face it: it is getting difficult to find tips that can improve overall system performance, when it matters for Exchange Server 2007. This last version of Exchange Server outperforms all its predecessors, since it can now take advantage of the x64 architecture in all its glory.

And if you ran the Exchange Best Practices Analyzer at the end of the installation process, the chances are that you end up with a pretty well tuned system that follows the recommendations from Microsoft and is ready to face the most demanding conditions.

The main goal of the provided tips will be the general optimization of the Exchange infrastructure, which means healthier, more stable and more reliable systems, as opposed to pure performance improvements.

As I did in a similar article for Exchange Server 2003, I would like to start with tip 0: size correctly! There is no way you can optimize a system that has not been correctly sized. I do not call this one a tip, it is a prerequisite, so do not ever, ever go on the adventure of deploying Exchange Server without any prior understanding of the basic hardware needs. A previous article I wrote, Exchange 2007 Sizing Cheat Sheet, can help you on that task.

And now, off with the talking, on with the tips.

1.  Document yourself

You do not have to be a messaging guru, but a deep knowledge of all the technologies involved in an Exchange Server environment will increase the possibilities that this same environment runs smoothly and is well optimized.

The Exchange Team is one of the most active inside Microsoft, documenting its product; there are literally tons of quality technical documents available on TechNet. These are just a few sites available 24×7 that can help you, besides many others all over the World and in so many different languages (I will assume that you are already familiar withMSExchange.org :-)):

Exchange Server TechCenter
Exchange Team Blog (You Had Me at EHLO)
Exchange Server Community
Exchange Server Wiki

2.  Run ExBPA Regularly

Running the Exchange Best Practices Analyzer (ExBPA) should be the first and final step of every Exchange Server setup process. Please, please do not skip it, because it will give you valuable information regarding your environment.

Once a separate tool you could freely download, the ExBPA is now included on the Exchange Server Toolbox, since it is so important for the overall health of Exchange servers. The tool collects data from several repositories, such as Active Directory, Registry and Operating System, and produces an extensive report with all the settings that do not conform to Microsoft best practices.

Run a Health Check regularly and keep ExBPA updated, by downloading the latest definitions from Microsoft.

3.  Optimize Storage

In previous Exchange Server versions, storage was often the root cause of performance bottlenecks. With the advent of Exchange Server 2007 and its ability to take advantage of x64 architectures, the need for large IO throughput is no longer so demanding.

Nevertheless there are a couple of best practices you should definitely follow that can affect Mailbox servers performance:

  • Size storage with the Exchange 2007 Mailbox Server Role Storage Requirements Calculator – sizing the storage for a Mailbox server is not rocket science, but it is far from being an easy process. Unless you are doing it for academic reasons, do not try to do the math by yourself, this is a great tool where some knowledgeable people have invested a lot of hard work and that will do the job for you.
  • Align the partitions – Depending on the storage, aligning partitions to track boundaries can have performance benefits. Your hardware vendor should provide you with pertinent information regarding the use of an alignment tool, but if they do not, use 64 sectors of 32KB. For more information, please read Disk Geometry and How to Align Exchange I/O with Storage Track Boundaries.
  • Partition Allocation Unit size – When formatting the volumes that will host Exchange databases, it is recommended to use a NTFS allocation unit size of 64KB. The recommendation of 64KB is based on the performance improvements seen with large sequential read operations, such as streaming backup and some eseutil tasks.
  • Review ESRP – Practically all the major storage vendors have published some guidance regarding storage configuration for Exchange Server. Most (or all) of these publications can be accessed through the Exchange Solution Reviewed Program (ESRP). There you will find the latest test results and guidelines from storage vendors that can really improve the health, performance and reliability of the storage subsystem.

4.  Tame your Backups

Exchange Server 2007 introduced many choices in the backup process. These are a few tips that can improve the overall backup experience:

  • When possible, always prefer VSS over streaming backups – There will be a plug-in for Windows Server Backup utility (to be released Summer 2008), the replacement for NTBackup, that will allow you to do Exchange VSS backups on Windows Server 2008.
  • If using LCR or CCR, backup from the passive node (SCR does not support backup from the target node), thus offloading the IO demand from the active node. When backing up from the passive, streaming backup cannot be used, only VSS.
  • If you are backing up from the passive node, activate Online Maintenance Checksum on the active node – The copy that is backed up is the copy that is checksummed, meaning that the active copy of the database will not go through the database integrity process. The Online Maintenance Checksum is controlled by the following registry key:
    HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesMSExchangeISParametersSystem
    DWORD Key: Online Maintenance Checksum
    DWORD Value: 1 (enabled), 0 (disabled)
  • Mind the online maintenance window – As in previous Exchange versions, the online maintenance and the backup process should not overlap in time. The event 703 should occur, at least, every 2 days for each mailbox database:
    Event Type: Information
    Event Source: ESE
    Event Category: Online Defragmentation
    Event ID: 703
    For more information please read Exchange 2007 SP1 ESE Changes – Part 2.
  • Use Data Protection Manager (DPM) 2007 – One of the many advantages is that you can offload the backup consistency checking from the Exchange server to the DPM server, thus reducing I/O impact of running eseutil on the Mailbox server during backup.

5.  Memory and Virtual Memory

In a 32-bit World, since the available memory had a 4GB limit, there was not much you could do to improve performance, besides setting the old /3GB switch. Fortunately, with the advent of the x64 architecture we can now address an (almost) unlimited amount of memory, which means the end of bottlenecks.

Exchange Server 2007 SP1 is pretty well tuned for memory utilization, but there are 2 tips I want to share: one regarding virtual memory settings, the other one more specific to Hub transport servers.

Over time, there has been confusing information regarding the correct size of the paging file (Microsoft literature included). At the time of the writing of this article, the official recommendation for Exchange server systems is:

  • If the server has less than 8 GB of RAM: amount of RAM times 1.5.
  • If the server has more than 8 GB of RAM: amount of RAM plus 10 MB.

If you have downloaded the latest update file for ExBPA, this is exactly what the tool will tell you.

Both minimum and maximum memory sizes should be set to the same value and the memory dump setting should be changed to “Kernel Memory Dump” (http://support.microsoft.com/?id=274598). If you have a dedicated drive to host the page file, you should configure a 100 MB page file on the system drive to accommodate the kernel dump.

The tip regarding the Hub Transport role has to do with the database cache size. Exchange 2007 SP1 (and earlier) has a default transport database cache size of 128MB. On Hub servers with high message rates, a higher value of the ESE cache can increase system performance. Since the transport service does not dynamically grow the ESE cache (unlike the Mailbox server role), the DatabaseMaxCacheSize parameter that is specified in theEdgeTransport.exe.config file should be increased to 512MB on Hub servers:

  • Current setting:
  • Recommended setting:

For more information, please read New maximum database cache size guidance for Exchange 2007 Hub Transport Server role.

6.  Use Outlook 2007

There is no doubt that Outlook 2007 and Exchange Server 2007 share a “Better Together” story. The following features are only available when you access an Exchange 2007 mailbox with Outlook 2007:

  • Autodiscover
  • Web-based Offline Address Book (OAB)
  • Availability Service
  • Resource Assistant
  • Unified Messaging
  • Improved Out of Office Assistant
  • Managed Folders
  • Increased Rules Limit
  • Partial Item Change Download
  • Message Classification

Another feature that was introduced with Exchange Server 2007 and works better with Outlook 2007 is RPC Client Throttling (or RPC Client Back-off). Client throttling allows Exchange to identify those users performing an excessive number of RPC operations against a Mailbox server, which could decrease overall system performance. The server will send a “back-off” request to these clients to reduce the performance effect on the server.

Outlook 2007 users receive a ropBackoff request, Outlook 2003 receive a status code of RPC_S_SERVER_TOO_BUSY. This means that the Outlook 2003 client gives up after 1 minute of successive back-offs and disconnects its session to the Exchange Server. For more information, please read Understanding Client Throttling.

As a Messaging Administrator, you have the power to enforce some settings that can improve general performance:

  • Make sure you install Service Pack 1 for Microsoft Office Outlook 2007, since this will fix some OST performance issues.
  • Force the use of Outlook cached mode:
  • Set-CasMailbox MailboxName –MAPIBlockOutlookNonCachedMode:$true
  • Prevent previous versions of Outlook from connecting, by modifying the Registry (KB288894) or by running the following cmdlet:
  • Set-CasMailbox MailboxName -MAPIBlockOutlookVersions:”-11.8010.8036”

Summary

advertisement

This concludes part 1 of this series. As a takeaway, remember that good planning is often the key to good performance. Some of the tips, like good sizing and storage tuning fall into the planning stage, meaning that they should be done before taking those servers into production.

Know your servers, measure them, create a baseline and keep track of your environment, that is the only way of knowing you are heading the right direction.

 Network Tweaks

At the network level, there are a couple of tweaks you can do in order to make things run smoother:

  • Disable Windows 2003 Scalable Network Pack (SNP) ¨C SNP was introduced as a means of accelerating the network stack. Unfortunately it has some issues with Exchange (please read Windows 2003 Scalable Networking pack and its possible effects on Exchange). In order to turn off all SNP features, follow the guidelines fromKB948496. SNP is disabled by default on Windows Server 2008.
  • Disable IPv6 in Windows Server 2008 ¨C Although IPv6 support is one of the great features of Windows Server 2008, it has some known issues with Exchange Server 2007. For instance, it can break Outlook Anywhere. So, if you do not really need IPv6, disable it:
  1. Uncheck Internet Protocol Version 6 (TCP/IPv6) in connection Properties
  2. Remove “::1     localhost¡± from the HOSTS file
  3. Open Registry and navigate to HKLMSystemCCSServicesTcpip6Parameters
  4. Create a DWORD (32-bit) named DisabledComponents and enter 0xFFFFFFFF (this will disable all IPv6 components, except the IPv6 loopback interface)
  5. Reboot
  • Disable TCP auto-tuning in Windows Server 2008 ¨C Auto-tuning is a feature that was introduced with Windows Vista and Windows Server 2008 in order to optimize TCP throughput. The problem is that some network devices do not support these features (most Cisco Firewall devices, Sonicwall Firewall, Check Point Firewall, some NG R55 routers, some Netgear routers), which can make things slower. To disable auto-tuning, run the following command:netsh interface tcp set global autotuninglevel=disabled

8.  Install Exchange on Windows 2008 Servers

The second Better Together story from this article is with Windows Server 2008 (the first one was with Outlook 2007).

There are many improvements in Windows 2008 that directly benefit Exchange 2007, such as:

  • Support for multi-subnet failover clusters ¨C With Windows 2003 you had to stretch the local subnet for geographically dispersed clusters. With Windows Server 2008 you can have CCR and SCC clusters across 2 different subnets.
  • SMBv2 increases file copy performance ¨C This mean faster log shipping. Since CCR and SCR use the SMB protocol to copy files, with Windows Server 2008 you gain up to 40% in file copy throughput.
  • Reduced downtime for hardware maintenance ¨C Windows Server 2008 enables a variety of core hardware components to be added, removed or replaced without requiring system downtime (requires capable hardware).
  • Near-zero downtime when fixing NTFS issues ¨C Windows Server 2008 includes self-healing NTFS, which corrects corruptions of an NTFS file system while the system is online, and without requiring Chkdsk to be run offline.
  • Greater scalability for Outlook Anywhere ¨C The RPC Proxy Service now supports per-IP address connection-limits, up to a maximum of 65,535 connections per IP address. Windows Server 2003 supports a maximum of 65,535 connections, regardless of the number of IP addresses assigned to the server.
  • Integrated Exchange-prerequisites ¨C Faster deployment. Windows Server 2008 includes Exchange 2007 installation dependencies: Microsoft Management Console (MMC) 3.0, Windows PowerShell, and .NET Framework 2.0.

Please remember that the installation of Exchange Server 2007 RTM on Windows Server 2008 is not supported, you will need SP1.

For more information, please read Exchange Server and Windows Server 2008, Part II.

9.  QoS with UM

The use of Quality of Service (QoS) and DiffServ can greatly improve the Unified Messaging experience. If you are experiencing UM problems related with network conditions, you should definitely try QoS. Support for Differentiated Services (DiffServ) QoS was added in SP1.

In order to enable QoS by using DiffServ, follow these steps:

  1. Run GPEdit.msc on the UM server.
  2. Expand Computer Configuration | Administrative Templates | Network | QoS Packet Scheduler | DSCP value of conforming packets.


Figure 1: Local Group Policy MMC

  1. On the right pane, double-click Controlled load service type, click Enabled, set DSCP value to 34, clickOK.


Figure 2: Controlled load service Properties

  1. Double-click Guaranteed service type, click Enabled, set DSCP value to 46, click OK.
  2. Install the QoS Packet Scheduler on the UM server (QoS Packet Scheduler can be installed by adding it as a Network Service on the properties of a network connection in Windows Server 2003. The QoS Packet Scheduler is installed by default on Windows Server 2008).


Figure 3: Network Properties

For more information, read New Unified Messaging Features in Exchange 2007 SP1.

10.  Start PowerShell faster

If you ever clicked the Exchange Management Shell icon (I am sure you did), you know that it can take a while before the window pops up. This pretty long delay is due to the compilation of Common Intermediate Language (CIL) to native code that is happening in the background.

If you want to speed up the launch time a little bit, follow these steps:

  1. Using Notepad, create a .PS1 file (name it Update-GAC.ps1) with the following code (on x86 systems, replaceFramework64 in the second line of this script with Framework):Set-Alias ngen @(
    dir (join-path ${env:windir} “Microsoft.NETFramework64”) ngen.exe -recurse |
    sort -descending lastwritetime)[0].fullName
    [appdomain]::currentdomain.getassemblies() | %{ngen $_.location}
  2. Start the Exchange Management Shell and run the previously created script.
  3. Close the Exchange Management Shell window and start it again. Notice the difference!

For more information, please read Avoiding Jitter: Jumpstarting the Exchange shell.

11.  Proactively Monitor the Exchange Environment

Exchange Server is definitely not a ¡°set it and forget it¡± solution. Like every other mission critical application you need to proactively monitor it, in order to assure the proper operation of the whole Exchange infrastructure and all its dependencies, to identify potential problems and to detect performance degradations before they turn into downtime.

Although you can use any 3rd party management solution, I strongly recommend you use System Center Operations Manager (OpsMgr) 2007, which includes the Exchange Server 2007 Management Pack.

The Exchange Server 2007 Management Pack for Operations Manager 2007 includes an extensive set of rules, views, knowledge, monitors and reports for managing Exchange Server 2007. These resources help you keep your Exchange Server 2007 environment in a healthy state, by detecting, sending alerts, and automatically responding to critical events, thus preventing possible service outages or configuration problems.

The following table provides an overview of the monitoring functionality that is enabled through Operations Manager 2007 (taken from the Exchange Server 2007 Management Pack Guide):

Exchange component Monitoring functionality
Exchange Client Access
  • ActiveSync and OWA connectivity monitoring including synthetic transactions
  • Performance measuring and alerting
Exchange Edge Transport
  • Performance measuring and alerting
Exchange Hub Transport
  • Performance measuring and alerting
Exchange Mailbox
  • Information Store monitoring
  • Mailflow and MAPI connectivity monitoring
  • Performance measuring and alerting
Exchange Unified Messaging
  • Unified Messaging connectivity monitoring including synthetic transactions
  • Performance measuring and alerting
Configuration and Security
  • Exchange best practices
Exchange Event Log monitoring
  • Comprehensive rules for Exchange
  • Detailed product knowledge about events

12.  Active Directory

Exchange Server strongly depends on the Active Directory infrastructure. Although Exchange does a good job of load balancing its requests across the domain controllers, including taking into account how busy the DCs are, it is a demanding application and will present relatively high load across all the GCs available in the site if you are taking full advantage of all the new features and roles in Exchange 2007.

The following table shows the recommended ratio of Exchange 2007 servers to Active Directory global catalog (GC) servers for each site containing Exchange 2007 Mailbox servers or users (assuming that the DCs and the Exchange Mailbox servers are both on roughly equivalent hardware):

32-bit

64-bit
GC:Mailbox server core ratio

1:4

1:8

You must have enough memory installed on the directory server to cache the entire Active Directory database (NTDS.DIT) in memory
1 GC near
XX users

Every 10,000 users

Every 20,000 users

I would like to add a final note regarding Windows Server 2008 Domain Controller/Global Catalogs. Windows 2008 introduced a new role, Read Only Domain Controller (RODC), that does not work with Exchange. Exchange will ignore RODC and will not connect to it, unless an administrator manually sets it, so please make sure you do not do any manual configuration to force this connection.

Conclusion

advertisement

These tips were not exclusively about performance, they were mainly about optimization, which means healthier, more stable, and more reliable systems.

Please bear in mind that some of the present recommendations may change in the future, as new service packs, service releases or new versions of the products see daylight. The best thing to do to make sure you are not using some deprecated recommendations is to follow tips #1 and #2:

Leave a comment