Providence Health

The Challenge: Overlapping systems cause issues

The M&A push began in earnest in the early 2010s, with Providence dramatically increasing the size of the organization. Most notable were the acquisition of Washington-based Swedish Health Services in 2012, and a merger with St. Joseph Health of California in 2016. A happy coincidence of the latter was that St. Joseph and Providence were both running Infoblox NIOS DDI to operate core networking operations. Yet significant differences in the design of the networks, disparate versions of software deployed and varying ages of appliances and equipment posed real challenges. While the network merger and integration of the various Infoblox systems took time and effort, it progressed over the years and ultimately came to a happy resolution. Other technical challenges posed by Providence’s M&A activities, however, weren’t so easily overcome. Specifically, several entities that were running Microsoft Windows and Active Directory to manage DDI were brought into the organization.

“Over the past few years, the biggest challenge we faced was the split responsibilities of DNS and DHCP between Infoblox and Windows,” explained Acker. With Infoblox NIOS, IP Address Management (IPAM) is automated and integrated with DNS and DHCP functionality within the platform. Not so with Microsoft, which raised several serious issues. “Because we were using both platforms for DNS and DHCP services, there was no ‘single source of truth’ for IPAM. We had some IP address data in one platform, other data being spread across a series of Microsoft DHCP servers, and still more stored in spreadsheets. This dispersal of IPAM data resulted in IP overlap, inconsistent or incorrect DNS resolutions, as well as very slow response times as DNS was having to query from platform to platform depending on the data needed. We jokingly referred to this arrangement as ‘DNS Spaghetti,’ as our query path resembled a mess of lines going from platform to platform.”

The Situation: Mapping out a new centralized system

While the Microsoft / DNS Spaghetti situation was a well-recognized problem, Acker and his team had higher priorities that had to be addressed first. Specifically, they needed to move the organization from a distributed environment to one where Providence’s massive network could be managed from a central data center.

“Through all of the M&A activity over the years, our Infoblox systems were arranged as a distributed platform, with appliances spread across multiple ministries and small locations,” explained Acker. “Our research showed us that we could implement a much more resilient architecture if we transitioned away from the distributed design and adopted a more centralized framework. And because Infoblox’s DDI technologies are so advanced, and support standards and capabilities like AnyCast, DHCP Failover and Primary/Secondary DNS Configurations, we knew we could make the transition mostly with the team and the technologies we had in place.”

As the larger centralization effort progressed, Acker and team finally set their sights on resolving the Microsoft conflict in 2022. He explained, “Our Windows Servers were nearing end of support / end of life–that was the final straw leading us to consolidate onto Infoblox and deprecate our remaining DDI operations that were still running on Windows.”

The Solution: Consolidating all DDI operations onto Infoblox NIOS DDI

Acker and his team initiated a two-stage migration, first moving more than 600 DHCP Subnets from Microsoft DHCP onto Infoblox, followed by the migration of just over one million DNS records. “Our migration into Infoblox was an amazing experience,” recounted Acker. “As a critical healthcare provider we can’t have downtime, which meant all migrations had to be done in a live environment. Our engineering team was able to complete these migrations over a two-month period, and the process was nearly transparent to the end users, with very little impact to service offerings for our enterprise environment.”

Infoblox’s Network Insight and the Grid system’s native discovery capabilities proved highly useful in this instance, enabling network administrators to view, inventory and understand all of their network elements. “Network Insight was exactly what we needed to manage the discovery phase,” said Acker. “It gave us excellent visibility and control over our environment, with deep insight into all corners of the legacy networks. This higher level of manageability set the stage for us to clean things up wherever needed, and make fast, solid progress on our vision of having a single fundamental platform to manage all aspects of our critical DNS, DHCP and IPAM operations.

The Result: Much better control over the network environment

Acker and his team couldn’t be happier with the outcomes of the Windows migration, and the larger network centralization effort as well. “The benefits that have been observed have been astonishing,” said Acker “DNS queries are measurably quicker now, resulting in much faster application response times. Also, with all domains and IPAM operations now automated and centralized within Infoblox, we have eliminated cross domain or subdomain queries between different Domain Controllers and realized positive change in several areas:

• Drove down overall queries per second from around 90K QPS to just about 30K QPS.
• Reduced the dynamic registration process for clients from the 10-to-15 minute range in the legacy
  environment to nearly immediately resolvable with Infoblox.
• Consolidated management of DNS and DHCP onto a unified system, with a single source of
  truth for IPAM.
• Achieved much better control and understanding over the DDI environment.

One of their biggest benefits from switching to Infoblox’s NIOS platform has been the ability to standardize configurations across the enterprise. No longer does the team experience issues with local/regional configurations; the system is now standardized and a single configuration can be applied for every device, no matter the location. “With centralized management, we’ve been able to increase our insight into DNS views and what content we’re exposing to partners, so we just feel that we have better control over our environment.”

Looking forward, Acker and team are exploring ways to strengthen their ecosystem with a hybrid approach by tapping into Infoblox’s cloud and security offerings, and increasing NetOps to increase protection for users, devices and data assets. As part of this effort, Acker and team are looking to tap into Infoblox’s growing selection of cloud and security offerings to further automate networking operations and increase protection for users, devices and data assets through APIs. “From initial trials and tests, I believe the API for Infoblox is one of the most mature API’s we have worked with,” said Acker. “There are so many capabilities the API offers that are way beyond what you can do in the GUI.”

Additionally, the team has recently moved many services into cloud platforms and are actively working to automate many networking tasks both in cloud and on-prem. “We are running some proof-of-concept testing on the Cloud Integration and Automation licenses,” related Acker. “We expect that reporting and alerting will be taking a front seat along with automation. We have run through a trial period of Infoblox Reporting and are looking to purchase a permanent license in the months ahead.”