I’ve had the opportunity to be blessed to work with large financial accounts for the last 8 years. A strangely curious problem was presented 18 months ago in evaluating how to interconnect building management systems and newer machine based devices onto the existing network. What was interesting, was that the seemingly simple technical solutions conflicted with real world business problems.
The real world business problems revolved around time to market, organizational competencies, capex vs opex arbitrage, and security (especially in a regulated financial). What initially appeared to be the least cost Capex models can have significant organizational challenges depending on whether competencies to operate and manage virtualized security, at scale, were present in the organization. This includes much more than design, but operations, audit, and ongoing change management support.
At the time this coincided with a need I had to create a capstone project as part of my MSISA program. So I embarked on expansion of the topic. I wanted to share the results of this back into the community to continue the dialogue on what is a very important problem… the attack surface will grow exponentially with the internet of things, and in many cases the virtual to physical divide will become blurred, with an increasing amount of machines interfacing with the physical world having connectivity in the virtual world.
In summary of what is below, I evaluated 5 realistic architectures to evolve modern enterprise infrastructures to support IOT based devices. These include
1) Doing nothing, plug and pray IOT devices onto the existing infrastructure. While this is not the right fit for most organizations it may be suitable for a very few companies, it does have the most risk.
2) Logical segmentation of existing assets and using newer SD-Wan type solutions to overlay existing networks. For many organizations, these models would be a good fit.
3) Airgapped approach, in which there is segmentation at the branch but the enterprise has touchpoints for operations and management, in either a DMZ or third party extranet. For highly regulated environments, or those with sensitive information, this approach is a good fit.
4) Isolated approach : Truly isolated, a good use case for Utilities or highly sensitive environments.
Considerations for which model to adopt includes multiple aspects:
1) Its much more than just IP connectivity. There are new generations of protocols, connectivity models (direct feeds from sensor to third party), and operational concerns (firmware management).
2) Depending on the regulatory and safety requirements, operational costs, and current competencies, different approaches may yield a greater benefit for different organizations.
The below are excerpts from my MSISA capstone report.
—————————————————————————
Introduction
A problem is looming for enterprise IT systems across the globe. As the Internet of Things has developed into a cost effective investment the attack surface is growing exponentially. This Internet of Things are building management systems and sensor based networks, machines connected to the network, rather than human driven machines. The systems being developed offer tangible business value, from automating management to energy efficiency, and the cost points are making them viable business investments. However, with the benefit comes a cost. These systems employ either embedded management systems or at times, stock distributions of operating systems. In some cases these may be centrally managed by a third party. This creates multiple areas of concern as these systems may not only provide a transport mechanism to a third party, but also offer an exploit vector in the firmware itself. In just the past two years this “internet of things” has shown multiple compromises. In one compromise a refrigeration unit was found to be participating in sending spam (Starr, 2014). In another, a well-known breach (Target) was initially able to manifest because of a third party linkage with a refrigeration supplier (Krebs, 2015). The teams responsible for building management systems have been found in one financial customer to be leveraging the internal network for connectivity, against policy. The gap exists that while existing security mechanisms and controls have existed for “fat client” computers and servers for a long time, with considerable research into the exploit vectors, as the internet of things is in its infancy, there hasn’t been as much diligence done at an architectural level to determine adequate security mechanisms and controls an enterprise can take. The purpose of this capstone is to create an architectural analysis of the options to an enterprise for how to connect these building management systems.
There are current programs in flight to help manufacturers and consumers know about Internet of things security, notably the OWASP who has published the top 10 project. The concept of the top 10 project is describes as “The OWASP Internet of Things (IoT) Top 10 is a project designed to help manufacturers, developers, and consumers better understand the security issues associated with the Internet of Things, and to enable users in any context to make better security decisions when building, deploying, or assessing IoT technologies” (OWASP, n.d.). Notably absent in this description is the implementers and architects of large organization make architectural decisions.
The scope of the project will be to discuss the challenges presented by sensor based networks and the internet of things, as well as the business drivers behind why organizations will be compelled to have a level of connectivity in the future. Based on this we will provide context around some security challenges this presents to an existing enterprise network, and provide four different architectural approaches. The scope will be incorporate regulated and non-regulated infrastructure networks running TCP/IP (although some legacy devices may have a converter to connect proprietary systems to an IP based network). The output will be an analysis of the different options with collateral that can be used to steer enterprises, and promote future dialogue on the topic.
Defense of the Solution
This project is to provide a multi-dimensional analysis of architectures that enterprises may take when connecting these new generations of devices to their existing infrastructure. The enterprises in scope are diverse industries with significant differences in regulatory requirements, security requirements, operational size and maturity, as well as financial resources that will affect how they approach the problem. Our “solution” is not to present a single option which may be an optimal fit for any one company. Just as not all organizations have adopted the same form of internet connectivity and enterprise security posture, the next generation of devices which will connect to the network will not adopt a one size fits all connectivity matrix.
The challenge is that while existing thought has gone into security protocols and paradigms within the devices which are connecting to the network, there is a notable absence of architectural analysis of how to zone, segment, and protect existing corporate resources from the new connectivity mesh and protocols which will be required as the devices are connected. To properly secure these networks the industry needs more open dialogue around the positioned security architectures, and this capstone seeks to create normative architectural reference points through which this dialogue may occur.
Problems in the Current State
Prior to evaluating a project design for implementing an internet of things architectural review, an organization should have a firm understanding of its existing security controls and processes in use. In the case of one large financial, it was determined that existing risk management and systems design did not fully account for the various third party connectivity requirements which were being considered as part of their corporate properties groups. For example, newer generation HVAC and security systems are available with remote monitoring and diagnostic capabilities offered via their third party providers, but the risk analysis of all these third party providers needs to be performed, and their management of data was outside of the existing governance model. In addition, there has been strengthened regulatory guidance (FFIEC, n.d.) around third party risk management following the Department of the Treasury’s audit report (Curry, 2014) that documented gaps in third party’s guidance in the existing regulations. This regulatory guidance was in part driven by multiple breaches in 2013 and 2014 which were exploited through third party relationships which were in part outside of the governance model of those companies.
The challenges exposed by this audit were that third party risks are complex to govern and required updates to existing regulatory guidance. These audit results are exacerbated by the next generation of IOT devices which may rely on greater vendor interaction with the devices than has been seen in the past. A proposed benefit of some of these solutions is the remote monitoring and management of devices which may be critical to an organization. The remote management and monitoring, as well as firmware updates, will typically require a level of third party integration. Coupled with the third party risk, is the risk of the expanding scope of firmware and nontraditional operating systems which would seek to connect to the corporate infrastructure. Whereas existing security models typically have internal trusted systems running corporate IT approved images, and potentially a guest or quarantine network for guests not running corporate IT based security software, IOT devices will represent a new generation of devices. These devices will be corporate assets which may be mission critical, and will run software which is outside the governance of the corporate standards. An example of how these new devices (referenced as building management devices below), fit with existing access types, is below.
| Device Type | Software Managed by | Access to Internal Resources | Operationally Critical |
| Corporate Resource | Corporate IT | Yes | Yes |
| Guest | Third Parties | No | No |
| Building Management System | Third Parties and Corporate IT | For some use cases | Yes |
Problem Statement
The security issue present is that the firmware updates as well as system management fall outside of traditional organizational governance models created for user based networks. The challenge this presents against the backdrop of third party risk, is that the attack surface will expand significantly, and the traditional “crunchy on the outside, chewy on the inside” deployment models of many traditional security postures are inadequate. The devices will provide in some cases direct third party access, and the presence of third party firmware which will not run corporate security software creates a significant risk that needs to be addressed. The problem statement as such is to evaluate potential architectures which can account for this expansion of risk while providing guidance on methods to address the connectivity requirements using a risk based approach.
Current State Requirements
Current state requirements are the sum of the requirements an organization is currently subject to, or which are leveraged in the IT organization to meet current business requirements. These include application requirements, the network architecture, security policy and regulatory requirements.
Typical Enterprise Applications
Typical application architectures in current enterprises leverage client-server and multi-tier web application client server architectures. The typical three tier model for a multi-tier web application architecture is shown in the figure below:
Figure 1: 3 Tier Web App Architecture
Of note in the client server and multi-tier architectures, is that the client in reference typically would exist in a campus or branch, or come in via the internet to an externally facing web server. However, the servers and content have been steadily migrating first to centralized data centers, and now also into the cloud. Security and segmentation, where needed, have been built around these architectures. A second traffic type which we find in the enterprise is peer to peer traffic typically used for real time media, such as voice and video.
Authentication and Segmentation Requirements
Current state enterprise IT deployments also use user based machines that are provisioned by corporate IT resources. Enterprises typically adopt a form of certified product and software solution mix which they have validated meets current criteria and current security requirements. The client end stations are typically user based, where a user’s credentials can be leveraged in a corporate security posture assessment for providing authentication and context aware authorization leveraging protocols such as 802.1x, Radius, LDAP and/or Active Directory. With this a users credentials can be leveraged to determine what resources he has access to in the network. This created an ability to provide segmentation for BYOD as well as guest networks. In the case an endpoint was not able to successfully authenticate, it could be placed into a walled garden with a limited subset of connectivity to either internet based content or subsets of internal resources. Availability targets may also differ for these two resource types, with separate RPO and RTO targets established for guest networks as well as corporate networks. Another key difference is in the aforementioned software loads used. In a guest network, it is expected that the OS in use may not be a corporate IT approved image, and appropriate security controls should be in place to ensure the security posture of internal resources when coupled with a guest network. However, for resources provisioned on the internal network there is typically a variety of software controls, from group based policy to anti-virus software solutions, to attempt to reduce risk of infection by a machine within the trusted environment.
Established DMZ and Trust Zones
An example of a typical enterprise architecture is shown in the figure below. Within this architecture are well defined DMZ and trust zones which separate untrusted external traffic from devices within the DMZ using stateful filtering and inspection devices. There is typically a second set of stateful filtering and inspection devices used to separate the semi-trusted DMZ from the internal network. The devices on the internal network are typically considered trusted and conform to an IT organizations information security policy with regards to confidentiality, integrity, and availability. These devices are typically hardened to a trusted state at the edge, and typically not as hardened within the trusted environment (for most enterprises). In a typical architecture additional protections are put at the edge and access layers to prevent intrusion. Additional third party and vendor based architectures are shown in Appendix A.
Current Architecture
An example of a typical enterprise architecture is shown in the figure below. Of note are the segmented trust zones used to protect the resources from untrusted users. Within the internal network devices are usually conformant to an organizations policies and access controls such as 802.1x are positioned at the edge to prevent unauthorized access to the internal resources. Additional third party and vendor based architectures are shown in Appendix A.
Figure 2: Typical Enterprise Deployment Multi Level trust model
Regulatory Requirements
Enterprises are subject to diverse regulatory requirements given the nature of their business. Healthcare organizations will be subject to HIPPA, Governmental organizations will be subject to NIST requirements, retailers will need to conform to PCI, and bank organizations will be subject to numerous bodies, with the FFIEC serving as a centralized examination body. These requirements strongly affect design and placement of security services within an existing architecture, and need to be well understood in designing any target state architectures.
New Requirements for IOT Systems
The requirements which are novel to the IOT use case represent both a problem with the products (protocol risks, firmware), but also products of the problem (horizontal scale pushing addressing architectures). The combination of these creates new requirements for enterprises to focus their architectural effort on. As this is an emerging market, some requirements may change over time and drive an emerging strategy for an enterprise. Any IOT model should account for future change and create logical building blocks which can allow products to be inserted into the architecture which does not change the fundamental scheme or other building blocks within the architecture. Some new requirements an organization should consider are outlined below.
New Connectivity Requirements
A number of new technologies have come to the surface which can enable different devices to be able to connect to the network. Aspects of these protocols include having lower power than traditional WIFI, as well as enabling mesh capabilities in a wireless network, allowing IOT devices to be further from the access gateway than traditionally designed WIFI networks. An organization will need to be well versed in the nuance of these new protocols, as they have come under increased scrutiny in the past years as security researchers have attempted to discover weaknesses in the protocols. Examples of these protocols include LoWPAN, 6LoWPAN, Zigbee, and Z-wave. Enterprises should evaluate and create a strategy around which protocols will be used in the network. Some considerations and organization should take include:
1) Do the forecasted requirements include low power devices which will need to adopt the newer protocols, and if so, which protocols should an organization standardize on?
2) Is Power over Ethernet required for certain subsets of devices? Does infrastructure and facilities groups need to accommodate revised cabling plans to accommodate the plans?
3) Where is the marketplace currently standing with regards to adoption of the protocols? Certain protocols such as Z-Wave and Zigbee have different ecosystems, and there may be room in the marketplace for both. It is also possible that over the coming decade a de facto protocol emerges which IOT manufacturers gravitate towards.
4) What are security ramifications for any protocol considered for use within the IOT plan?
New Protocol Architectures
With the advancements of newer technologies which are converging legacy protocols onto traditional enterprise networks, new breeds of protocols are needed to both standardize, as well as meet new system characteristics. An example of a new characteristic is the requirement for a point to multipoint extremely fast message bus which can be used for device to device, sensor based data. An example would be a self-driving car, where a message retrieved from a sensor may need to be relayed to provide for a reduction in speed and corresponding traction control. For this, standardization work needs to be done to ensure the device to device communications are able to meet current and future needs, as well as receive the same level of security scrutiny as existing protocols, such that systems can be hardened against attack.
A sample taxonomy of the three types of high level protocol architectures is below, and an expansion of some of these protocols will be provided based on the work of Sam Schneider of Electronic Design.
| Device to Device
(D2D) |
Device to Server
(D2S) |
Server to Server
(S2S) |
|
| Example Use case | Factory automation
Connected car sensor response |
Remote monitoring | Transactional Message integrity for Servers (Coherence) |
| Example Protocols | DDS | XMPP
MQTT |
AMQP |
| Forecasted Application Response Requirements | < 10ms | 10ms – 1s | >1s |
MQTT: Message Queue Telemetry Transport is optimized for collecting data from a large number of devices. This is for the orchestration and remote monitoring of server (mostly cloud) based resources. In the enterprises with secure systems that may not use a public cloud infrastructure, an enterprise headend would use this protocol for monitoring devices. The particular protocol stack would run on top of TCP to provide for error control.
XMPP: Extensible Messaging and Presence Protocol provides for a subscribe based service in which a state of presence can be advertised. It has particular strengths in presence, addressing, and security, and with a high amount of scale provides a rich protocol for consumer based IOT applications.
DDS: Data Distribution Service is intended to provide a high performance data bus for device to device interactions. It needs to provide for multipoint connectivity with extremely low latency, as devices measure real time on the order of microseconds.
AMQP: Advanced Message Queue protocol isn’t simply an IOT protocol, with background in high end server transactions processing, but it is sometimes considered an IOT protocol as a bus for cloud based servers to ensure concurrency. (Schneider, 2013)
New Scale requirements
ABI research noted, “The installed base of active wireless connected devices will exceed 16 billion in 2014, about 20% more than in 2013. The numbers of devices will more than double from the current level, with 40.9 billion forecasted for 2020.” (ABI, 2014). The scale of this expands not just the physical architecture, but also the logical addressing of an enterprise will be taxed significantly. As of 2015 many enterprises have not deployed IPv6 (and those that have typically constrain it to their internet facing services), however, the logical addressing requirements should provide a compelling event for organizations to architect what their longer term strategies will be to ensure they are able to meet current and future requirements. IPv6 supports over 340 trillion, trillion addressable devices and has the potential to meet the addressing requirements for the foreseeable future (IPv6, 2015). As organizations review their ability to support to Internet of Things, a review of their addressing strategy will need to take place and an awareness of IPv6 and organizational capabilities, will need to be performed.
Third Party Connectivity Requirements
In some instances, third parties may manage or monitor embedded systems via value added services. This creates a new attack vector for attackers to focus on compromise of the third parties themselves, using the third party as a beachhead into the network. During the past two years this “internet of things” has shown multiple compromises. In one compromise a refrigeration unit was found to be participating in sending spam (Starr, 2014). In another, a well-known breach (Target) was initially able to manifest because of a third party linkage with a refrigeration supplier (Krebs, 2015).
Firmware Management
Typical client computing in an enterprise model has provided for a small number of client based operating systems (typically Windows and OSX), which traditionally have been centrally managed by IT. This shifted somewhat with BYOD, but enterprises typically adopted a security approach to zone these off, or ensure they were compliant with BYOD security posture. There have been exceptions to this rule, for example network attached printers, however the exceptions have traditionally been manageable and represented a minority of devices.
In the coming decade the proliferation of “smart” devices will grossly exceed the amount of humans and IT organizations will see significant scale increase in quantity of devices as well as device type. This will lead to an explosion in the amount of diversity of firmware connected to their network, each with corresponding vendor support policies, and software risk potential. This will shift the security prospectus significantly. In addition to the diversity of firmware, the delivery and support mechanisms will differ by vendor and device type. Some vendors will provide firmware management and update services, whereas others will post new firmware and expect enterprise IT organizations to deploy at will. This significant shift in software risk needs to be addressed with a corresponding architectural solution to constrain the impact of faulty firmware exposing a security risk.
RPO/RTO targets
Within the design of the system to connect these new device requirements, an understanding of the criticality to the system needs to be well understood. For example, a shipping company handling hazardous goods has a requirement to be able to locate all of their shipments in near real time, or stop the shipment in progress to protect human lives. Another example is a manufacturing plant, which may have high availability requirements for their floor machines. This differs from other IOT devices such as HVAC systems or lighting systems, where connectivity failures in the monitoring and management may not require real time or near real time access in order for the business to remain in production. In creating adequate RPO and RTO criteria upon which to create the architecture, enterprises should consider the following questions.
1) What is the impact to the business if the connectivity is unavailable? Will the business be able to perform in the absence of connectivity, or will production be halted?
2) What are the standalone capabilities of the devices? Will they continue to function until connectivity is restored, or do they require constant communication to be able to function?
3) What level of state needs to be maintained by the end devices should a problem or lack of connectivity occur? Does a rollback need to account for near real time state, or what is the timeline in restoral state needed to resume full operation?
Based on these questions, an understanding of restoral point and restoral time can be factored into the system design to support the new systems.
Existing Gaps
Existing gaps are the delta between the current capabilities and the future state requirements. As annotated, there are multiple gaps in how organizations currently manage their systems, from procedural to technological and organizational. Many organizations will have gaps in competencies required to architect and support solutions into the next decade. These gaps will need to be addressed to properly ensure an enterprise class competency around security to meet both organizational risk concerns as well as regulatory risk concerns.
Methodology
The methodology this paper is pursuing is to not evaluate a singular one size fits all solution to apply to any one organization (retail, healthcare, financial, government), but rather evaluate 5 potential architectures and why an organization may choose to deploy it based on regulatory, financial, and security concerns. The reason for this is that different organizations should be aware of the options and considerations for each, and different organizations will have different requirements. An extremely high level view of how to connect these networks is below. Based on this high level view, 5 viable architectures are expanded upon with considerable cost and value differentials. This paper will describe the architectures and the decision making criteria an organization would use in selecting a suitable architecture.
Figure 3 : High Level options
Evaluating the architectures in this manner will allow a multidimensional approach in which we evaluate the project from the perspective of costs and security capabilities. The costs are not just capital, but also operational and are to account for the ability of an organization to meet regulatory requirements and provide for the capability to audit the systems to ensure security posture is enforced and able to be maintained in an ongoing basis.
The reason this approach is taken is there are many factors which could play into an organizational decision on how to connect their infrastructure. Not all organizations will have the same risk prospectus, and there are a finite amount of segmentation architectures available. Each has corresponding value, cost, and risk to apply. Based on an organization’s existing competencies, some architectural options may not be tenable to deploy without significant retrofit to organizational abilities, which could significantly drive costs or affect timelines.
Architectural Options
The architectural options one would evaluate when connecting an IOT system to an enterprise network vary in terms of cost and complexity. The architectures will be evaluated from a multi-dimensional aspect, taking into account cost, security, supportability, and ongoing maintenance costs. There are five potential architectures we will review within this project. The integrated approach is leveraging the existing network, without segmentation, to provide services to the future IOT devices. The virtualized approach leverages the existing network, but uses virtual segmentation technology to enforce a level of segmentation between the existing IT network and the IOT based network. The overlay approach expands on this with the addition of new hardware to provide segmentation via “overlaying” the existing IT network. The air gapped approach creates a parallel network in which the IOT devices are “air gapped” from the existing IT network. And finally, the isolated approach creates a separate network which is both isolated from the IT network, as well as isolated from external connectivity. A sixth option, which is not developed upon in this capstone, is the internet approach in which IOT devices are simply plugged into an unsecured internet connection. This approach is not being developed because there would appear, at the time of this writing, to be limited viability for such an approach. A high level summary of these options is below.
| Option | Description | Potential Use Case |
| 1: Integrated | IOT Devices plugged into existing corporate network without security controls to properly isolate the devices from existing corporate assets. | SMB with Little risk to internal threats, limited intellectual property and minimal customer and collateral damage |
| 2: Virtualized | IOT devices plugged into existing corporate network with security controls via configuration of existing, shared equipment, to isolate the devices from existing corporate assets. | Enterprise with need to protect internal assets from potentially risky devices, and have the organizational capabilities to manage a virtualized environment. |
| 3: Overlaid | IOT devices plugged into a device which provides segmentation from existing corporate assets via securely overlaying the existing network, where the existing network only provides transit for securely tunneled traffic. | Enterprise with need to protect internal assets from potentially risky devices, and may not have the organizational capabilities to manage a virtualized environment. |
| 4: Airgapped | IOT devices plugged into separate, parallel network which converges via the extranet/DMZ environment. | Enterprise with a high level of exposure to corporate data or intellectual property loss, where the cost to leverage a parallel solution will, at scale, be less expensive from a risk perspective, than to leverage the existing infrastructure. An example may be a financial institution. |
| 5: Isolated | IOT devices plugged into a completely separate and parallel solution with complete isolation from semi trusted and untrusted networks. | Explicitly Trusted, critical infrastructure systems where risk of compromise could have significant ramifications on human life or on a nation states operational readiness. |
| 6: Internet | Devices directly exposed to the internet. | This paper will not strongly consider this use case, as there is little to no use case for a majority of enterprise business management systems to have direct exposure to the internet without corresponding security controls to ensure governance around business continuity. Indeed a purpose of this paper is to ensure viable architectures are available to prevent this. |
Figure 4: Summary of Options
Depending on an organizations current deployment, risk profile, regulatory requirements, and organizational competencies, architects would be guided to choose a deployment methodology that fits its requirements best. To do this an organization would want to go through the following steps
1) Determine requirements: As outlined in the previous sections there are numerous factors which would be involved in determining the system requirements for an organization. Some of these requirements would be near term technical requirements. Regulatory requirements would also factor in here. An additional consideration is ongoing line of business requirements that would be needed in the future as the business is developing. Project timeline requirements would also need to be captured during this phase, as they will be a key success criteria.
2) Determine existing architectural requirements and capabilities: This is where an analysis of existing security requirements, addressing requirements, and architectural constraints and guidelines factor in. During this phase a system audit should be done to determine where existing gaps in security policy or implementation are. In addition, the audit should capture any current IOT devices which may have already been installed, and determine any “shadow IT” areas, areas in which lines of business or organizational entities have created systems which are outside of the organizational policies. Existing operational capabilities should also be evaluated so that a gap analysis can be done during the system evaluation phase. During this phase, a “point of departure” understanding of the network is created.
3) Evaluate current industry trends and new capabilities: the IT industry is a constantly evolving space, and there may exist new capabilities which can impact the decision making process when determining a target state architecture. For example, improvements in management and orchestration which has occurred in the prior 5 years with the advent of software defined networking, and the evolution of a software defined WAN offers to significantly shift the operational expenditure component of traditional networks. These trends should be understood so that the target state architecture is best able to accommodate current and future state requirements, given current and future state IT technologies.
4) Creation of a point of arrival architecture: Through taking in the new architectural requirements, existing architectural requirements, and industry trends in technology, a point of arrival architecture can be created in which current and future state requirements are coalesced into the decision making process around selection of a target state architecture.
5) Development of a project plan with milestones: In some instances (especially in larger environments) a phased approach would be required to move from the point of departure architecture to a point of arrival target state. To accomplish this, a project plan with specific assigned resources and deliverables, would need to be designed.
The first architecture we will consider as an IOT option is the simplest to deploy, but also leads to the largest risk to the enterprise. The option is an integrated architecture, where IOT devices are plugged into the existing corporate IT network without significant consideration around long term risk prospectus. This is a credible architecture, and it is likely that enterprises would find small instances of corporate building management systems and IOT systems plugged into the existing IT network. At a small amount of scale with limited devices, which may not have third party management, the attack surface is not extended significantly beyond the current IT deployments of IP attached printers. The risk to this architecture is in the exponential proliferation of IOT devices, causing exponential expansion of the attack surface beyond what IT may have intended. The key benefit of this architecture is in time to market and the ability to leverage the existing infrastructure in place.
Figure 5
The second and third architectures we will consider are the virtualized and overlaid architectures. The security prospectus of each is very similar; it is a segmentation of the existing corporate IT network. The fundamental differences in these architectures are the requirement to procure new equipment to overlay the existing network, versus leveraging virtualization technologies within the existing network. At first glance, leveraging the existing hardware would seem a financially responsible decision, given a similar return (in terms of risk). However it is worth consideration the operational expense of retrofitting the existing network should be considered, as newer devices on the market have a much higher scale operational model which leverages SDN technologies for the campus and WAN, branded under the market term “SD-Wan” technologies. Depending on the scale and level of virtualization in the existing network, there is a cost balancing act to consider between the operational expense of deploying virtualization on the existing network, versus the capital expense of overlaying the existing network with lower cost of ownership devices. An example is an enterprise which currently does not use Layer 3 end to end segmentation in their network today, would not have an organizational competency here, would in some cases need to retrofit existing equipment to ensure hardware support, and would need a fairly significant change to create the capability. In this case it may be more cost effective to use a separate device which overlay’s the existing network without adding complexity. The cost models advertised in the marketplace are typically marketing driven, as the cost of overlaying existing equipment would vary significantly by company. It is advisable that companies perform their own cost of ownership models and organizational capabilities to make internal return on investment decisions prior to embarking on a large scale deployment of either technology.
Figure 6
The fourth architecture we will consider is the Airgapped architecture. This architecture creates a parallel network from the enterprise network, for the purpose of connecting a large amount of these devices. In many cases the organization would still want the ability to remotely manage or monitor the devices, so the connectivity in this design would leverage either an extranet or an internet based connection. In the extranet based connection, connectivity to the parallel network would come in via a connection into the DMZ. In the Internet based model, a lan to lan VPN connection to the DMZ could come in via the internet. In either model it is assumed that the enterprise currently has a security plan developed for DMZ based extranet links, and the value of the Airgapped architecture is in being able to isolate the networks in the branch, and leverage existing security controls in the DMZ. This is in many cases the most expensive option at first glance. There is a need to procure both additional equipment, and additional circuits in many cases. However, for certain organizational profiles, the costs may actually be less. This architecture should be considered by organizations with strong regulatory requirements or high costs ensuring availability and security of existing infrastructure. An example could be a healthcare industry or financial customer. These customers would have regulatory requirements around how they would need to do segmentation, and would have high availability and security requirements of their existing infrastructure. This leads to their IT infrastructure using high cost business class circuits which can meet their business SLA, and investments in appropriate hardware to support it. When taking into account that these customers may also need to deploy stateful security devices at the edge, or IPS/IDS, to meet regulatory requirements should they mix this traffic, it can be less expensive to leverage dedicated commodity circuits and less expensive hardware to meet their IOT needs.
Figure 7
Option 5 takes the isolation of the devices one step further. This option does not allow for any external network connectivity, whether over the untrusted network or over the trusted network. This option is useful for utilities or other IOT based systems, of which the failure could have major impacts to social order or human life. Examples would be the power grid and chemical control systems. In this network deployment, there still may be new circuits required, if communication is needed between two buildings or campuses, but these would be dedicated or point to point circuits which would not have external reachability. As with the Airgapped approach, there would need to be dedicated network hardware to support the IOT devices.
Resources Used
The hardware and resources to be utilized are a criteria which should be a primary focus for the organization. The cost for additional circuits or hardware, versus the operational costs of expansion of a solution, can be significant to an enterprise. All security solutions are balanced against this cost versus risk curve. The cost versus risk curve becomes an asymptote where as you continue to invest money the marginal improvements to security become less. With this in mind, balancing the resources forecasted to be used for a project during the architectural design phase is of critical importance to the firm.
An evaluation of the presented above is in the section below. Of note is the implicit trust level an organization is choosing for their IOT devices. This is derived from the architectural positioning of the devices with respect to an existing network. For example, an organization may not consider their IOT devices as “trusted” when using the integrated architecture, however, the net result is the devices share the same physical infrastructure as the “trusted” corporate devices. This results in an implicit trusting of the devices unless security controls are in place to add segmentation between the IOT devices and the trusted enterprise environment.
| Option | Enterprise Trust Level for IOT devices | Leverages Enterprise Network for Transport | Able to be managed by Third Party | Requires additional equipment for connectivity |
| 1: Integrated | Trusted | Yes | Yes | No |
| 2: Virtualized | Semi-Trusted | Yes | Yes | No |
| 3: Overlaid | Semi-Trusted | Yes | Yes | Yes |
| 4: Airgapped | Semi-Trusted | No | Yes | Yes |
| 5: Isolated | Trusted | Separate Network | No | Yes |
| 6: Internet | Untrusted | No | Yes | Yes |
Figure 8 : Differentiation of Options
In addition the following tables indicates the primary cost drivers behind the architectures. Whereas the Airgapped and isolated deployments have higher circuit and equipment costs, the virtualized and overlaid approaches have heavier infrastructure change implementation costs. In all of the solutions, the scale and existing organizational competencies of an organization are major considerations. As has been noted, an organization which already has an end to end trusted and auditable segmentation solution would have a much lower operational cost to expand the existing solution using the virtualized approach, whereas the operational cost to retrofit an existing organization to support end to end virtualization may make the operational costs of the virtualization solution unattractive compared to an overlaid approach.
| Option | Capital Expense for New Equipment | Operational Expense for new Circuits | Change to Existing Network | Incremental Expense in Maintenance |
| 1: Integrated | No | No | Lowest | None |
| 2: Virtualized | No | No | Heavily dependent on existing organizational capabilities | None |
| 3: Overlaid | Yes | No | Yes, Less than 2 | Yes |
| 4: Airgapped | Yes | Yes | No | Yes |
| 5: Isolated | Yes | In Most Cases | No | Yes |
Quality Assurance
Ensuring quality and availability of the organizations information systems is important in ensuring a viable timeline of provisioning of the new capabilities. Impacts to quality in the beginning phases of the project will greatly lengthen timelines for deployment. In addition, ensuring the security and availability of the information systems is critical to the success of any project.
A proposed method to ensure quality assurance of the solutions would be to:
1) Align with vendor best practices: Understand vendor best practices for the devices you plan to use to accomplish the chosen architecture. Different implementations will have different scale and hardening criteria associated, and prior to designing, testing, and implementing solutions, it is best to design with known best practices and architectural constraints in mind.
2) Solutions level testing will be discussed more exhaustively in the section below. However, it is important that the solution is adequately tested to ensure quality.
3) Engage operations groups early: Critical to the success of any project is the ability for ongoing operations and maintenance of the solution to be well understood and signed off on by operations. Depending on the architecture selected, new devices or configurations will be applied to the network to ensure proper security posture is maintained. The operations teams will need to provide feedback early in the design process to ensure the solution is supportable, and their continual feedback during implementation will be needed to refine and allow for continual service improvement.
4) Training of resources: The resources required to design and support the solution will need to have adequate training. The training should be outlined at the outset of the architecture during the phase in which organizational competency gaps are notated. A plan should be put in place to bridge the organizational gaps through training.
5) Ensure adequate run books have been updated: Where operational run books are in place, they should be updated so that tier 1-4 support resources are able to follow the run books and operate the network.
Prior to rollout of the proposed architecture, system level testing needs to be performed. This testing should include multiple groups which validate the security of the system, as well as ensure the systems being deployed are able to meet the scale and load required of them. The following are an example of testing evolutions an organization should perform prior to implementation.
1) Scale testing: The system should be tested in a testbed proper of replicating a representable production environment. This would likely include virtualized devices or test sets capable of injecting both control plane and data plane traffic possible to replicate the scale and load placed on the system. In smaller organizations, it may be difficult to obtain funding for this level of testing, and vendor best practices and existing testing which has been done to emulate scale can be consulted. However, it is important that a level of diligence in test is done around the required scale of the environment.
2) Application level testing: Applications in use within the environment should be tested to ensure proper functioning prior to any major changes to the underlying architecture.
3) Security penetration testing: Penetration testing of the components should be performed to ensure they are capable of meeting the security posture of the environment. In addition, solution level penetration testing of the architecture should be performed to ensure conformance to existing security requirements.
Post Implementation Support and Issues
Day 2 planning and resources should be forecasted such that the solution is serviceable and able to be maintained. The lack of planning for post implementation support, resource planning, and organizational impact can end up creating a significant drag on the organization and their ability to allocate resources for future solution development. The below sections discuss the considerations an organization should make in post implementation support and maintenance of the solution.
Post Implementation Support
Post implementation support should be adequately prepared for to ensure the delivery roadmap is not subjugated by quality issues which arise as a part of the changes. In order to ensure adequate post implementation support, the following are guidelines which should be followed.
1) During implementation ensure adequate testing for lines of business is done: During the change windows, application level testing should be performed to ensure the network is restored to a functional state such that business can be performed following the change.
2) Prior to implementation: A backup of the known good state should be taken for all components which will undergo change. This will be used in the event a rollback to a prior known state is warranted.
3) Back out scripts should be created for each change: In the event a production issue is arrived at which cannot be worked around, adequate back out scripts should be available such that the network can be restored to a pre change state and business can resume.
4) Fall Forward operational plans should be devised: A level of diligence should be put forth ahead of time to ensure that in the event of a production issue, the operations teams have flexibility to fix on failure within a set of guidelines which does not further degrade the architecture or inject security risk. For example, if a firewall rule is missing, is it viable to add while still upholding the integrity of the architecture? Was a rule just missed in the change? In some situations, a fall forward approach may be warranted. Whereas in others, such as the idea of bypassing a firewall, should be avoided if it cannot be worked through. Ensure operational teams have guidance on what changes can be made to resolve the situation.
5) Following implementation ensure security posture is validated: Following the implementation, or any changes to the architecture, security testing should be performed on the systems to ensure that security posture is not compromised as a result of the change.
6) Ensure day 2 support resources are available: In some organizations, dedicated operations teams are available which are apart from the teams which execute the change. In other organizations, due to resources the same implementation resources may be required to support the change. Adequate preparations should be made so that there is coverage during the business hours such that if an issue arises with the change, there are resources available to address the issues and remediate any quality issues found.
Post Implementation Support Resources
Implementing the architectural change to the network will require ongoing resource availability throughout the lifecycle of the solution. This adds to the cost of the solution and should be analyzed during the analysis phase in choosing the architecture. In all architectural models, these resources will be required. The differentiation between architectures would be the quantity needed to augment an existing IT organization, and that is highly tied into an existing organizations competencies. For example, in supporting a virtualized environment, if an organization already has deployed end to end virtualization, the existing architectural and engineering resources needed to support one additional segment would be less than an enterprise which has not done end to end virtualization.
A summary of some key resources that need to be factored into the planning for long term supportability of any solution is below:
1) Architectural resources: Typically architecture is done upfront in the system design phase. However, architectural resources will need to be trained and understand the architecture and security posture for future architectural changes. An example would be in the overlaid approach, where new hardware may be used for segmentation, depending on scale an architectural resource (or partial resource), may be required for ongoing architectural design.
2) Engineering resources: Engineering resources will be required for low level design and implementation of changes ongoing to the network.
3) IPAM resources: IP address management will be an area of investment with the IOT environment. IOT may push IPv6 in an enterprise, or greatly expand the scope of addressable nodes. This will require ongoing IPAM support to monitor and manage the IP address space within an organization.
4) Change management resources to support maintenance plan: Project managers and resources will be needed to ensure the organization is able to successfully implement the change. Resources from business impacted by the change will also be needed, as well as a resource to manage central PMO functions.
5) Operational resources: Operational resources needed to execute the changes and provide day two support will need to be identified. In some deployments, new skill sets needed to manage the network will be needed, in other situations, more of the existing skill sets may be needed as the scope of the network expands.
6) Audit resources: Resources will be required to validate the network security during the course of the change, and audit resources will be needed for a sustainable ongoing audit function. In some situations, existing audit resources may be sufficient to manage the go forward environment, however in other situations, additional resources may be needed. This is highly subjective to the existing audit capabilities and efficiency within an organization.
Maintenance Plan
A culture of maintenance is an imperative to ensuring system availability. Basic hardware maintenance may need to be performed (air filters, etc). Configuration updates will need to be done. Patches will need to be applied to protect against vulnerabilities found in the future. Considerations for a maintenance plan would include:
1) How often does software on the end systems need to be updated, and how? This is an important consideration as you scope the resources required to be able to maintain the solution long term.
2) How often do configuration changes forecasted to be required to maintain the system? Separate from configuration changes needed to enable the network or that are tied to a project, there are ongoing configuration changes that may need to be made to simply maintain the network, such as applying changes discovered to mitigate new found vulnerabilities of software defects.
3) Vendor supplied maintenance procedures should be followed. Most enterprise class hardware has referenceable guidelines that should be followed to ensure peak performance of the equipment. Typical MTBF (mean time between failure) calculations assume the device is maintained and operating with documented environmental ranges
4) Long term lifecycle management: The equipment or supporting infrastructure (such as circuit type), is all subject to technological obsolescence. An organization should understand their vendors support policies and end of life policies to ensure they have adequate provisions to fund tech refresh, and ensure their vendors support policies align to their depreciation schedule to ensure they can fully depreciate their equipment prior to obsolescence. Provisions for long term care of the equipment, to include ensuring serviceability and refresh, should be done.
Conclusion, Outcomes, and Reflection
The project to create a next generation internet of things structure within an enterprise network, and do so securely, is not a trivial task for an organization. The questions posed have obvious easy answers, the results of which fundamentally undercut an organizations security in the long term. The plug and pray options that have sufficed in providing “connectivity” are no longer sufficient when balanced against the industrialization of hacking which has occurred in the last decade. As every transaction, every logistics system interaction, and every machine based operation are plugged into this global connectivity network, there becomes too much at stake for our existing social models to be willing to tolerate rolling brownouts of critical services. The business value extracted by the transition to the internet will be dwarfed by the social and business changes occurring with the proliferation of connectivity. Whereas most industries previously gained efficiency from the transition to global connectivity, now the very end user processes themselves, including how and where we shop, where we bank, and how we protect our environment are being evolved. The security of the institutions that are making the transition to digital enterprises in the coming decade will be critical to an organizations survival.
It may be found one day, as it is expected that the machine based networks dwarfs the existing user based networks, that we are in the phase of creating the central nervous system to be used by society in the orchestration of many of life’s processes as we know it. Healthcare and the delivery of such, with remotely monitored equipment and increasing complexity, provides a telemetry network for health data. Self driving cars will increasingly take advantage of the real time systems feedback from sensors within itself, as well as sensors external to itself. The IOT based networks and their communications systems will become foundational to how we live our lives. Protecting this will be critical to ensuring the safety of society.
References
ABI: The Internet of Things Will Drive Wireless Connected Devices to 40.9 Billion in 2020. (2014, August 20). Retrieved August 10, 2015, from https://www.abiresearch.com/press/the-internet-of-things-will-drive-wireless-connect/
Building management system. (2015, August 11). In Wikipedia, The Free Encyclopedia. Retrieved 04:17, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=Building_management_system&oldid=675543325
Cisco SAFE Reference Guide – SAFE Overview [Design Zone for Security]. Cisco. Cisco Systems, 8 July 2010. Web. 11 Aug. 2015. <http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Security/SAFE_RG/SAFE_rg/chap1.html>.
Curry, T. (2014, April 21). OCC’s Review of Banks’ Use of Third Party Service Providers Is Not Sufficiently Documented. Retrieved November 9, 2015, from http://www.treasury.gov/about/organizational-structure/ig/Audit Reports and Testimonies/OIG-14-034.pdf
Cyber Tip: Be Vigilant with Your Internet of Things (IoT) Devices. (2015, October 13). Retrieved November 6, 2015, from https://www.fbi.gov/news/news_blog/cyber-tip-be-vigilant-with-your-internet-of-things-iot-devices
DMZ (computing). (2015, June 17). In Wikipedia, The Free Encyclopedia. Retrieved 04:06, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=DMZ_(computing)&oldid=667348170
FFIEC Press Releases. (n.d.). Retrieved November 9, 2015, from http://www.ffiec.gov/press.htm
Firewall (computing). (2015, August 5). In Wikipedia, The Free Encyclopedia. Retrieved 04:14, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=Firewall_(computing)&oldid=674727382
Internet of Things. (2015, September 7). In Wikipedia, The Free Encyclopedia. Retrieved 04:26, September 8, 2015, from https://en.wikipedia.org/w/index.php?title=Internet_of_Things&oldid=679945685
IPv6. (2015, August 11). In Wikipedia, The Free Encyclopedia. Retrieved 04:18, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=IPv6&oldid=675672216
ITIL. (2015, October 26). In Wikipedia, The Free Encyclopedia. Retrieved 03:51, November 6, 2015, from https://en.wikipedia.org/w/index.php?title=ITIL&oldid=687607369
Krebs, B. (2014, February 14). Krebs on Security. Retrieved August 10, 2015, from http://krebsonsecurity.com/2014/02/email-attack-on-vendor-set-up-breach-at-target/
Local area network. (2015, July 21). In Wikipedia, The Free Encyclopedia. Retrieved 04:08, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=Local_area_network&oldid=672359119
Payment Card Industry Data Security Standard. (2015, April 1). Retrieved November 2, 2015, from https://www.pcisecuritystandards.org/documents/PCI_DSS_v3-1.pdf
Point of presence. (2015, July 1). In Wikipedia, The Free Encyclopedia. Retrieved 04:10, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=Point_of_presence&oldid=669415980
Ponemon Institute’s 2015 Global Cost of Data Breach Study Reveals Average Cost of Data Breach Reaches Record Levels. (2014, May 27). Retrieved November 9, 2015, from http://www.prnewswire.com/news-releases/ponemon-institutes-2015-global-cost-of-data-breach-study-reveals-average-cost-of-data-breach-reaches-record-levels-300089057.html
OWASP Internet of Things Top Ten Project. (2014). Retrieved August 10, 2015, from https://www.owasp.org/index.php/OWASP_Internet_of_Things_Top_Ten_Project
Router (computing). (2015, August 11). In Wikipedia, The Free Encyclopedia. Retrieved 04:14, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=Router_(computing)&oldid=675543288
Security Architecture. Checkpoint. Checkpoint. Web. 11 Aug. 2015. <http://www.checkpoint.com/services/education/training/courses/samples/PoNS_C09_Security_Architecture.pdf>.
Schneider, S. (2013, October 9). Understanding The Protocols Behind The Internet Of Things. Retrieved November 2, 2015, from http://electronicdesign.com/iot/understanding-protocols-behind-internet-things
Starr, M. (2014, January 19). Fridge caught sending spam emails in botnet attack – CNET. Retrieved August 10, 2015, from http://www.cnet.com/news/fridge-caught-sending-spam-emails-in-botnet-attack/
The Ultra-Secure Network Architecture. The Ultra-Secure Network Architecture. McGladrey. Web. 11 Aug. 2015. <http://mcgladrey.com/content/mcgladrey/en_US/what-we-do/services/risk-advisory/the-ultra-secure-network-architecture.html>.
Wide area network. (2015, August 10). In Wikipedia, The Free Encyclopedia. Retrieved 04:09, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=Wide_area_network&oldid=675370533
Z-Wave. (2015, August 9). In Wikipedia, The Free Encyclopedia. Retrieved 04:04, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=Z-Wave&oldid=675239280
ZigBee. (2015, August 7). In Wikipedia, The Free Encyclopedia. Retrieved 04:08, August 12, 2015, from https://en.wikipedia.org/w/index.php?title=ZigBee&oldid=674977066
Strongedge.io 