Current State of Monitoring in a Connected World and Challenges We Face
The technological infrastructures of companies have been changing and evolving over the years in this continuous digital transformation we live in. But in this new reality, where COVID-19 has accelerated the pace of change even further, the predominance of the digital is absolute. The monitoring of our infrastructures, organizations, and processes must keep up and go hand in hand with the objectives of our business. In this article, I will explain the challenges we face from a monitoring perspective in a fully connected world.
Discovery, Identification, and Inventory
Network discovery is arguably one of the key points in monitoring any type of infrastructure. It simplifies work (and life in general) by detecting the components that are part of your systems. This is especially important in changing environments, where infrastructure components can appear or disappear. By avoiding manual management as much as possible and automating these processes, we ensure that we cover the entire spectrum of base components that make up the infrastructure supporting your business.
This discovery should be non-intrusive, to facilitate deployment within the various areas of a company. Installing agents on each piece of equipment is cumbersome and may face resistance in certain areas. Moreover, it makes little sense in elastic architectures or for industrial devices. Therefore, discoveries are made through network scans based on protocols such as ICMP, SSH, HTTP, TCP, DNS, etc. Additionally, it is necessary to query information through APIs or CMDBs.
If the discovery is complete and appropriate, a fundamental goal is achieved in any company, which is to have a unified inventory for the entire organization that can be used, consulted, and exploited by any of its departments or areas. At this point, designing a good Rest API is vital for the optimal query and update process of the inventory. It also greatly facilitates integration with other in-house or third-party tools.
Identifying and cataloging the discovered elements is vital for proper monitoring configuration. Linux machines, Windows devices, network devices from different manufacturers, applications, databases, websites; through signatures obtained from scans, as well as analysis of open ports, operating systems, software versions, or information obtained from other tools, it is possible to individualize each of the assets and apply corresponding monitoring automatically, efficiently, and effectively.
Adapting to Changing and Evolving Environments
But obtaining all this information is of no use if it is not kept up to date over time. Infrastructures change within hours, new elements appear, others are decommissioned, or existing ones modify some of their properties. Therefore, it is essential to renew all the information at coherent time intervals according to the frequency of modification or their importance, to always have a “living” inventory.
On the other hand, this update can be automatic or supervised. In certain critical environments, it may be dangerous to remove devices or modify certain properties without supervision. It will depend on the characteristics of the infrastructure and how much we trust our self-discovery process to choose between an instant update or a guided update, to apply changes as a whole or on an element-by-element basis.
Intelligent Asset and Service Management
What do we do once we have such a huge amount of monitored assets?
One of the biggest failures in a monitoring system is notification flooding. At first glance, it may seem appropriate to receive the anomalous status of any component, no matter how small, but in a very short period of time, such event saturation leads to neglect and abandonment in infrastructure monitoring. It is necessary to create intelligent notifications based on logical rules and groupings to alert on affected business processes and quickly and effectively find the root cause of the problem.
Such notifications do not have to end up in an email inbox; sending them through instant messaging platforms to collaborative channels on Slack or Telegram may be more effective. Without neglecting integration with ticketing tools and with SOAR tools to involve support and security departments and thus eliminate information silos.
Cloud Computing
But if we talk about “a connected world,” undoubtedly we have to talk about virtualization and cloud computing, dominant design styles that greatly facilitate the deployment of both infrastructures and tools and components and allow agile development of applications and services.
We must be able to discover and monitor virtualized elements, containers, their orchestration systems, and the most used clouds such as Azure, AWS, and Google Cloud, among others. And for this, it must be understood that these types of architectures are elastic as they can change in a matter of seconds based on system load, events, or external commands, number of connected users, specific business needs in certain time frames, etc.
Application Instrumentation
Due to the ephemeral nature of containers, instrumentation of services and applications makes sense. “Instruments” are added to the software that report metrics through an HTTP endpoint and are collected by a server (Prometheus being the most used) periodically. Latencies, generated traffic, errors, and saturation are usually measured, and alerts can be generated based on rules.
SDN
Another challenge we face is monitoring Software Defined Networks (SDN), in which software-based controllers allow directing network traffic and communicating with the underlying hardware infrastructure through programming APIs, a totally different approach from traditional networks where only dedicated hardware devices (routers and switches) are used. It is a paradigm that is already widely used and to which we must adapt and offer value. Major telecommunications companies like Cisco and Teldat are basing their WAN interconnection solutions on this technology (SD-WAN).
The Value of Data
I wouldn’t want to end this article on a connected world without talking about Internet of Things (IoT) devices, which allow remote and real-time control and management of virtually any object, with applications both industrial and domestic. We’re talking about vehicles, appliances, and even furniture and clothing. The Internet of Things is based on a constant transmission of data, and the monitoring of these devices is done through Machine to Machine (M2M) protocols, especially MQTT, perhaps due to its simplicity and lightness, the most popular protocol for communication between IoT devices. This communication uses a publisher/subscriber pattern where messages are arranged in hierarchically organized topics, which can be consumed by other applications or technological stacks.
