City of Stockton Streamlines Networking, Strengthens Security, and Eliminates Overhead with Infoblox

THE CHALLENGE

Manual Management, Lack of Visibility and Control

A key issue with Stockton’s DDI system involved managing IP addresses. Every device on a network requires a dedicated IP address, and adept management of those addresses is critical for reliable network connectivity. For a city like Stockton, IP address issues could potentially affect fire trucks, patrol cars, and public safety personnel, thereby placing lives and property at risk. When Jamil arrived, the existing system for managing IP addresses had several limitations. For example, it had been assembled in-house using so-called “free” software that was not suited for dynamic networking environments. It also allowed only one person at a time to access the system, which created a single point of failure when it came to performing updates and troubleshooting in a timely manner.

Moreover, the existing system required that IT staff manage the city’s tens of thousands of IP addresses by spreadsheet and without consistent, easy-to-update documentation. IP address conflicts were inevitable and affected service reliability. An underlying obstacle was that Jamil and his coworkers lacked crucial control of IP addresses, a significant impediment for a network as complex as Stockton’s. “There are a lot of intricacies with IP addresses in our network,” Jamil says. Those complexities include the private VPNs and the computer-aided dispatch (CAD) systems that the fire and police departments use, along with the ongoing need to interact seamlessly with VLANs used in other local and regional public safety networks.

The problem for Stockton’s network administrators was that their existing IPAM implementation provided no visibility into device IP and MAC addresses, when they appeared, what subnet/VLAN they were on, device attributes, or DNS and DHCP records. A further complication was that as IP ranges and DNS records were added and dropped and switch ports were updated, all changes had to be made manually, an arduous and error-prone undertaking for any networking team.

In addition, the subnets for Stockton’s fire and police must be segregated from other network segments to remain compliant with Criminal Justice Information Services (CJIS). Without clear visibility into IP address allocations and clear documentation, Stockton’s IT staff had no ready way of knowing when a device was noncompliant. That uncertainty led to connection glitches. In one instance, a city leader was unable to log onto the network. After investigating, IT eventually discovered that the user was on a laptop that was not authorized for the IP address range associated with that subnet.

The IT department had to routinely resolve issues like that one. This effort consumed time and attention. It involved sifting through countless spreadsheets to trace IP address status for individual devices, often leading to miscommunications about which addresses and subnets had been updated.

City leaders were hardly the only ones affected. Jamil’s team was spending as much as 40 hours a month troubleshooting connection issues affecting employees across the city. The consequences rippled beyond IT. “You have a fireman or a police officer that can’t get online at 3 or 4 am. They’re losing time over connectivity issues. They call me, I reach out to the person on call, and if he can’t resolve it, then he has to escalate to a more senior staffer,” Jamil says. “That’s a lot of lost time.”

Risks from the city’s existing IP address management implementation involved more than network reliability and performance. The manual aspect of IP addresses management also increased security risks. Jamil and his colleagues were forced to share and update the city’s IP address spreadsheets via email or online platforms, potentially exposing them to threat actors. From Jamil’s perspective, the city’s IP address management needed to be fully automated and the address database made secure.

THE SOLUTION

Rapid Migration to Robust IP Address Management

When Jamil decided it was time to replace the city’s trouble-plagued DDI solution, Infoblox was already on the IT department’s radar. The city had been exploring Infoblox’s solutions before he’d come on board. However, those discussions always reached the same conclusion: the city’s existing implementation was too entrenched, and replacing it would be too time-consuming and risky.

Despite those objections, Jamil knew that from a technological standpoint, maintaining the status quo with the existing DDI solution was not a viable option. As a result, he and his team began identifying essential requirements for a new DDI system. From his research, he learned that the IP address automation, security, and reporting features of NIOS DDI met his criteria. Moreover, the distribution capabilities of the Infoblox Grid would provide the high availability that the city’s network demanded.

While eager to acquire NIOS, Jamil was also adamant that the migration needed to happen in three months or less, a timeframe deemed overly optimistic according to many people he spoke with, including other vendors. “They all said the journey’s going to take anywhere from six months up to a year because our network was so complex.” In discussions with Infoblox Professional Services and the NIOS project team, he received a very different answer. “They insisted that if we met once a week, we could knock this out in one month. We almost did. The only hold up was scheduling on our part.”

Jamil attributes the smooth rollout to thorough planning and clear communication from Infoblox team members. “They are really dedicated. They saw things that we might not have seen and really buttoned up the details.”

THE RESULT

Eliminating Network Disruptions, Slashing Costs, and Enhancing Security

With the migration complete, Stockton’s network gained the rock-solid connectivity and resiliency it had lacked with the previous DDI system. The city’s NIOS solution includes Infoblox virtual servers, as well as a physical server that will soon be online in Stockton’s data center to further bolster disaster recovery and business continuity.

NIOS fully automates Stockton’s IP address management while enabling complete discovery of all addresses and endpoints. It ensures that IP addresses are always up to date, synchronized across Stockton’s DDI infrastructure, and correctly associated with the right subnets. Moreover, the city’s IP addresses are now maintained in a highly secure, access-controlled database.

The most immediate benefit seen post-migration was the ability for anyone with even the most minimal training to perform DDI tasks through Infoblox’s intuitive web interface. Today, multiple people on the IT team can work simultaneously, eliminating the single point of failure from the prior implementation and greatly accelerating problem resolution.

With the rich dashboards in NIOS, Jamil’s team has gained granular visibility into all IP addresses. No more scanning through thousands of spreadsheet rows. “Everything is in order. We can see every single device that has a managed IP address,” he says. For the city’s IT team, allocating IP addresses or resolving conflicts is now a snap. “That’s the beauty of Infoblox. You don’t have to track people down, ask to see spreadsheets, do IP configs. You just go to the dashboard and boom, it’s done.”

Maintaining CJIS compliance is much simpler now too because team members can quickly drill down to see such details as individual domain controllers and mobile devices in police cars and firetrucks. With NIOS, Stockton has been able to avoid the disruptions caused by noncompliance, as well as VLAN conflicts with other public safety networks. “We have not had any of those issues. None,” Jamil says.

For Stockton, reliable and effective IP address management has generated considerable savings. Since deploying NIOS, there have been no 3 am calls from fire stations. NIOS has eliminated those incidents and the costly overtime associated with them. Moreover, the solution has freed the city’s IT team from having to constantly troubleshoot connectivity disruptions, which could consume as much as 40 hours a month, to instead focus on higher-value activities. “It’s hard to put into numbers but the savings have been limitless. We used to incur significant overtime costs responding to outages. Now, we not only save on those hours but we also eliminate the stress and impact on our team.”

In the security realm, NIOS does more than protect Stockton’s IP address database. By eliminating address duplicates, conflicts, and instances of noncompliance, it also improves the IT department’s security posture. “Our IP address vulnerability has gone down substantially because of this implementation,” says Jamil.

With Infoblox, Jamil and his team can respond faster to security events. For example, when IT receives an alert from an endpoint security solution associated with particular IP addresses, team members don’t have to spend cycles doing investigations or accessing different tools to find the affected devices. “You go to Infoblox and the IP source is right there. You identify it, disable it, and kill the threat right through Infoblox,” Jamil says.

In the months ahead, the City of Stockton plans to enhance its security further still by deploying Infoblox Threat Defense. While his engagement with NIOS is Jamil’s first experience with Infoblox, it’s one he intends to build on. “Infoblox is an incredible value for cybersecurity, productivity, and resiliency. We get bleeding-edge technology and no longer have to worry about getting down in the weeds to build what we need.”