The miniaturization of switching elements and large-scale integration of nanophotonics gives rise to a critical challenge to the internal processing of the optical data packets while maintaining a memory-free switching fabric. We have developed a new routing approach specifically designed to accommodate the unique nano-scale integrated optical interconnection network, called SPINet (Scalable Photonic Integrated Network). SPINet is a novel optical packet switching architecture that does not employ optical buffering of any kind within its switching nodes; consequently, messages are dropped upon contention. A novel physical layer acknowledgement protocol allows for a dropped-message detection mechanism in which an optical acknowledgement (ack) pulse is sent from the receiving port upon successful transmission. SPINet leverages broadband WDM to offer high transmission bandwidths through a wavelength-striped packet format. A 4×4 experimental network test-bed has been implemented with complex programmable logic devices (CPLDs) and discrete components, demonstrating critical network concepts such as address encoding and decoding, correct routing and switching, and error-free transmission of high bandwidth messages in the presence of ack pulses.
The SPINet network test-bed is comprised of 2×2 SOA-based photonic switching nodes, offering wideband transmission, data transparency, and packet-rate granularity. An experimental prototype switching node verifies its performance, demonstrating an average bandwidth exceeding 40 Gb/s per port and error-free routing of 160 Gb/s peak bandwidth.
Within the scope of the SPINet network simulations and test-bed, we have introduced the concept of path diversity as a means of increasing the utilization of optical packet switched networks. The network test-bed also demonstrates a high level of flexibility and programmability, straightforwardly supporting both synchronous and asynchronous traffic as well as priority-encoded optical packets. The interoperability between the optical network test-bed and an interface buffer has demonstrated dynamic queue management and cross-layer signal communication. The SPINet OPS network test-bed has also been adapted to support a programmable high bandwidth multicasting operation with mixed data rate packets [Lai OFC 2009].
The silicon material system endows novel optical devices with exceptional performance for diverse applications ranging from short-haul optical communication links to on-chip interconnection networks. Its complementary metal-oxide-semiconductor (CMOS)-process compatibility enables low-cost, high-yield fabrication of monolithically integrated circuits that can combine the best of optical and electrical functionalities.
The large index contrast in silicon photonic devices enables waveguides with low bending losses and engineered dispersions, empowering a broad and flexible design space complemented with immense dispersion tunability. Leveraging this tuning capability, silicon waveguides have recently become a promising platform for ultrafast all-optical parametric processing based on four-wave mixing (FWM), supporting future transparent optical networks with data rates approaching 1 Tb/s per wavelength channel.
We have demonstrated ultra-broadband wavelength conversion in these silicon photonic waveguides at the data rates of 10 and 40 Gb/s. Using critical quantitative system-level performance metrics, we quantitatively characterized these devices. In order to further demonstrate data rate transparency and scalability of this method, we have subsequently demonstrated all-optical wavelength conversion approaching hundreds of gigabits-per-second.
Broadcasting and multicasting of optical messages, critical network processes associated with the selective dispersing of information across many nodes, have traditionally been performed in the electronic domain using power-hungry optical-electronic-optical (O-E-O) conversion—as the data rate increases, the energy consumed by the serializers and deserializers alone quickly becomes the energy dissipation bottleneck in these systems. We have presented for the first time the use of FWM for all-optical wavelength multicasting on-chip in the silicon platform using the same dispersion-engineered silicon photonic waveguides. The all-optical processes using this inherently optically-transparent method are scalable both in multicast number and data rate.
Keren Bergman, Columbia University
David Keezer, Scott Wills, Georgia Institute of Technology
Keren Bergman, Columbia University
Gary Carter, University of Maryland Baltimore County
This project aims to develop a scalable architecture for an optical packet-switching (OPS) fabric for use in massively parallel high-performance computing systems. The goal is to provide large-bandwidth, low-latency connectivity between each of the users (e.g. processing nodes and memory banks) in such a system. The data vortex is a distributed deflection-routing interconnection network architecture designed to fully exploit the properties of fiber-optic technology in order to achieve ultrahigh bandwidth, low latency, and a high degree of port-count scalability. Utilizing photonic switching elements which facilitate transparent, broadband, self-routed optical packets, we have constructed a complete 36-node 12×12 optical switch with terabit-per-second scale routing capacity (see images below). This is the first complete implementation of a truly OPS network where no central arbitration is required. The system is capable of routing packets with immense bandwidths—160 Gb/s (10 Gb/s per channel × 16 wavelength channels) has been demonstrated but much higher rates are possible—from any one of the 12 input ports to any one of the 12 output ports, providing a median (and average) latency of 110 ns.
A number of experimental investigations of the system behavior have been performed, demonstrating for instance the flexibility of the system to power and timing variations. Additionally, injecting packets with varying duration and varying numbers of wavelength channels into the network simultaneously has verified the system’s robustness to variable message sizes.
Although the implemented switch fabric accommodates 12 ports, the intended application requires extensive scalability to harbor multi-thousands of users. The precise photonic implementation of the physical layer in such scaled systems may limit the physical size and usable bandwidth. Because SOAs are employed as the key switching element, understanding their performance characteristics is important to optimizing the system. A recirculating testbed environment has been implemented to experimentally study the physical limitations of OPS networks induced by the individual switching elements. This testbed has enabled unique physical layer analysis in the context of system scalability investigations, the effects of different modulation formats, and the first investigation of polarization dependent gain in OPS networks. Modeling and simulation of the physical-layer behavior are used in conjunction with the experiments to optimize the information carrying capacity and efficiency in future OPS network implementations.
For a full overview of our body of work related to the data vortex OPS interconnection network systems and subsystems, as well as many improvements that we have made to the original fabric, see our recent invited paper in the Journal of Lightwave Technology.
To address the need for a practical solution for buffering optical packets, we have developed a novel optical packet buffer architecture. The transparent buffering design is comprised of identical SOA-based building-block modules, yielding straightforward scalability and extensibility. In a time-slotted manner, the buffer supports independent read and write processes without packet rejection or misordering. Both first-in first-out (FIFO or queue) and last-in first-out (LIFO or stack) prioritization schemes have been experimentally realized. Further, active queue management (AQM) can be implemented on the buffer architecture to allow for network congestion control. Simulations have verified the improved buffer performance with latency, and experiments have demonstrated the functional verification of the optical packet buffer modeling AQM. The basic optical packet buffer architecture has also been adapted to realize network interface packet injection control for an optical packet-switched network, accepting backpressure and controlling the traffic injected into the network.
Global Environment for Network Innovations (GENI) is a NSF research agenda for clean-slate Internet design, experimentation, and collaboration to support experimental research in network science and engineering. The scope of our GENI project Embedded Real-Time Measurements (ERM) specifically ensures that the future GENI network infrastructure includes the appropriate technology to support cross-layer communications. By collaborating with other control frameworks, we endeavor to realize the ability within GENI to incorporate a diverse set of real-time measurements in networking protocols. The project addresses the GENI challenge of architectural experimentations across diverse heterogeneous technologies by supporting real-time cross-layer communications and measurements. Our objective is to develop networking capabilities within the future GENI infrastructure that enable deeper exposure of cross-layer information and user access to real-time measurements.
Within this project, we have developed a set of GENI requirements for real-time measurements and defined specifications for GENI networking protocols, recommending the use of an integrated, unified measurement framework. In addition, our project involves performing discrete-event simulations of cross-layer based networks in ns-2 and OPNET. Current work involves the experimental verification of these concepts.
This work continues along similar lines of the previous sections covering SPINet and POPS with the focus of using integrated optical systems to provide connectivity between many users. Here, though, the users envisioned are board-level elements that would be found in a typical computer system, such as processors, memory banks, or I/O devices. The goal of the project is to develop an integrated photonic switching system that delivers high-bandwidths to a large number of these users, and can be switched at sufficiently high speeds (nanosecond-scale) to facilitate message exchange in a very dynamic system where communication patterns are constantly changing.
Utilizing the functionality of many of the silicon photonic components explored through other research by our group, a wide variety of complex switching systems can be envisioned, which can each be integrated on a single chip providing connectivity to the local computational elements. Due to the large design space for such a system, it is important to compare both network and physical layer performances among the many alternatives. Therefore, we have developed a simulation environment which can monitor network performance (e.g. latency and throughput) for varying traffic patterns and user configurations, while also aggregating the physical-layer metrics (e.g. insertion loss, crosstalk, and power consumption) of the system in order to obtain the full view of how the system behaves. Such a simulation environment provides (1) a pre-fabrication performance comparison between competing designs, (2) perspective on how physical device improvements enhance system performance or similarly what device performance is required in order to achieve the desired level of system performance, and (3) an abstraction (similar to electronic CAD tools) of rigorous physical-layer modeling algorithms, which enables the investigations of more complex systems.
The image above represents a snapshot of the video to the left. In the image, a number of the 1×2 switches, 2×2 switches, waveguide crossings, and bends are arranged in the simulator to form the 4×4 non-blocking photonic router. The video depicts a brief demonstration of the simulator, which was implemented on the OMNeT++ platform. We develop the photonic device models in a hierarchical manner that allows us to describe in high detail several basic building blocks, and from these to construct larger functional components and even entire large-scale topologies. The animation begins by demonstrating the message transmission protocol on a photonic torus network. The left window shows the photonic plane and the right window shows the electronic control plane. We first see a message being transmitted along an electronic path which constitutes a path-setup, and subsequently data transmission occurs on the photonic plane. Near 45 seconds into the animation, the multi-level design methodology that our simulator leverages is illustrated when a user opens a single tile in the photonic network. Next, one of the 4×4 switches found within the tile is opened, and finally the viewer gets to peer into one of the basic building blocks that is explicitly programmed within the simulation. It is at this level that the physical device parameters may be altered.
The Future of Microprocessors
Recent developments in the microprocessor industry have indicated the end of the uniprocessor design and the emergence of multi-processor architectures. Chip designers have reached the limit of instruction-level parallelism that can be exploited with better pipeline design, as well as the improvements from clock scaling due to packaging and technology constraints. However, the emerging class of multi-processor systems raises a new problem in the interconnection network, the medium that allows all the processors, memory, and IO devices to communicate.
Typical modern day on-chip interconnection networks are implemented as electronic packet switched topologies. In some multi-core architectures, the network accounts for over 50% of the total power consumption of the chip. This proportion will be exacerbated as workload demand on these on-chip networks increases. Also, off-chip bandwidth is limited to the number of metal wires that can be traced off the limited peripheral area, which results in a large mismatch in on- and off-chip bandwidth. In order to alleviate these issues of power and bandwidth, one proposed solution that we and the research community are actively pursuing is the use of a photonic interconnection network.
Bringing photonics to the chip-level can potentially bring increased bandwidth, reduced latency, and improved power efficiency. The main advantage of on-chip nanophotonics lies in the decoupling of distance with power consumption. For chip-scale distances, signals can be modulated once and carried completely to their destination regardless of how far away it is. Conversely, electronics requires buffering every few millimeters, which consumes more power as data travels further. In addition to this key advantage for on-chip communications, photonics brings the added advantage that this concept also applies for chip IO, while maintaining the same bandwidth delivered on-chip! Speed of light in silicon end-to-end propagation can also yield significant improvements in latency. Because optics is not constrained to the common electrical restrictions that prevent high signaling rates across interfaces, it effectively solves the bandwidth mismatch.
Comprehensive Photonic Network Simulator
We are currently developing Photonic and Electronic Network Integration and Execution Simulator (PhoenixSim), a cycle accurate photonic interconnection network simulator suite on top of the OMNeT++ environment. The package is primarily composed of four simulation planes: the photonic plane, the electronic plane, the processing plane, and the IO plane. The photonic plane is a hierarchical structure with very detailed physical models of a number of photonic building blocks which can be used to create more complicated photonic structures, such as a switch or a network. The electronic plane acts as both a way to control the photonic network as well as an independent conventional electronic network, which typically shadows the photonic layout. The processing plane contains the modules that generate traffic, whether it be random, trace-based, or from cycle accurate instruction execution. The IO plane contains DRAM and other storage modeling, as well as bridges to other systems.
This simulation environment is being developed in an extremely dynamic and customizable way to enable it to be used in a variety of ways. The suite can analyze photonic networks at both the physical level, looking at parameters such as insertion loss and crosstalk, as well as at the system level in order to examine network and application performance. This functional diversity is enabled by the variety of other powerful open-source modeling packages we incorporate to increase accuracy. This includes, among others, the cycle accurate micro-architectural simulator SESC, the electronic router power model package ORION, and the DRAM functional and power model DRAMsim.
Photonics for Board-Level Communications
Currently we are investigating ways to bring photonics to both the microprocessor chip and the motherboard. Current computing systems suffer from a bandwidth bottleneck between the processor and memory. Typically processor designs use caching to help hide the latency, but as the number of cores on chip grows, as will the network complexity, careful deliberation of the communication architecture will be extremely important.
Nanophotonic CAD Environment
We are also developing Visual Automated Nanophotonic Design And Layout (VANDAL), a CAD environment for nanophotonic interconnects.
Recent advances in silicon photonic micro-fabrication techniques—the same highly-developed techniques used in the commercial development of the ubiquitous complementary metal–oxide–semiconductor (CMOS) platform—have led to the expansion of a photonic toolbox consisting of key elements which enable the realization of chip-scale photonic interconnection networks in the context of networks-on-chip (NoCs) interconnecting chip multi-processors. Collaborating with leading research groups in this area, we have developed a network-driven design approach for implementing and characterizing these building blocks. Our vision includes a three-dimensional integrated (3DI) monolithically-integrated silicon stack, comprising traditional CMOS and dynamic random access memory (DRAM) technology as well as optoelectronic and other photonic integrated circuit (PIC) devices.
Among the components that we have investigated to support our vision is an electro-optic form-factor translator, which converts space-parallel data from an electronic bus into wavelength-parallel photonic data coinciding on a single waveguide link. We then investigated the ultra-broadband, low-loss photonic link capable of carrying terabits-per-second of information encoded in this wavelength-parallel manner from one node to another.
Providing a seamless gateway from the compute nodes to the photonic network represents a key enabler for the realization of an end-to-end optical interconnection fabric. Format and data rate mismatches at the electronic/photonic edge necessitate the development of a unique network interface to address these critical bottlenecks. The design and development of an Optical Network Interface Card (ONIC) that will address these challenges to latency and bandwidth scalability is currently underway.
The ONIC manages the encoding, packetization, and multiplexing/demultiplexing of serial electronic data to a wavelength-parallel optical format in a low-latency manner that is transparent to the interconnected compute nodes. By abstracting away the interconnection network, the attached compute nodes can exploit the capacity of WDM â€“ supporting high-bandwidth multi-wavelength striped message exchange â€“ while being completely agnostic to the details of the underlying photonic interconnect.
The advancement of various standard protocols, such as PCI Express and InfiniBand, has enabled interconnectivity among diverse communicating modules and processors. By leveraging field programmable gate array (FPGA) technology, we have flexibility in the definition of the data exchange protocol, optical message structure, timing, and synchronization. This flexibility thus allows for the architectural exploration, experimental testing, and design validation of a variety of computing systems supporting different communications protocols.
PCI Express (PCIe) has emerged as the preeminent protocol standard for high-bandwidth chip-to-chip communication in current and future generation computing systems. PCIe is based on packetized serial point-to-point links and, in its current iteration, can support up to 16 data lanes per link signaling at 8 Gb/s per lane.
We have developed an all-optical photonic interface capable of transparently formatting serial data streams, such as PCIe, into high-bandwidth wavelength parallel photonic packets. Employing the aforementioned photonic interface, we have also demonstrated the end-to-end generation of a PCIe link originating from a remote endpoint across the interface to a host computer. The remote endpoint is implemented on a customized FPGA-based device. PCIe traffic originating from the endpoint and logically routed through the established PCIe link is demonstrated via direct memory accesses (DMA) initiated by an application running on an x86-based PC. Successful transmission of PCIe data at 2.5 Gb/s and maintenance of the logical PCIe link was experimentally confirmed across eight wavelengths.
InfiniBand is a switched-fabric interconnection network prevalent in today's high-performance computing systems. Infiniband routes variable-sized packets from source to destination at high data rates and low latencies. And like 10 Gigabit Ethernet, the individual copper cables linking the network together have been replaced with WDM fiber optics. However, the Infiniband switches are still electronic, necessitating an optic to electronic conversion at every switch in the network. Our research aim in this arena is to transparently send Infiniband packets over our optical packet-switched networks. To achieve this goal, we are adapting an off-the-shelf FPGA board to decode the Infiniband packet from a standard Infiniband HCA (essentially a network interface card) and transport it out over our optical network. The picture at left shows some of our test equipment, notably the FPGA board and physical adapters. We are working to expand our test setup to create a point-to-point link between two Infiniband HCAs using our optical packet format.
In order to evaluate critical architectural design considerations a macro-scale test-bed environment consisting of high-performance blade-level CMP compute nodes interfaced to an optical interconnection network via the aforementioned ONIC is currently under development. This interdisciplinary effort will leverage ongoing research in optical interconnection networks, low-latency photonic network interfaces, and communication intensive applications to create a coherent end-to-end prototyping environment. The implemented test-bed will facilitate the experimental validation of end-to-end optical message exchange in various hardware platforms and topological configurations, enabling realistic investigation of critical architectural concepts and real-world performance characterizations. Specifically, the test-bed will provide a platform for investigation of optical address encoding/decoding methods, demonstration of photonic end-to-end payload routing and transmission, and evaluation of interface/routing latencies and throughput scaling in a practical environment. Furthermore, by executing realistic application driven traffic on high-bandwidth inter-node optical communications platforms, this demonstrator will provide a means for evaluating high-performance WDM infrastructures as full network solutions for CMP-based advanced computing systems. It is our vision that the successful implementation this interconnect fabric system test-bed will help bridge the gap toward practical use of optical interconnects in future high-performance computing systems.
In an effort to bridge the gap between academic research and real-world application, we are collaborating closely with the engineers at Intel Research towards a path for viable commercialization of optical interconnects supporting multi-wavelength striped message exchange for high performance cluster computing. The focus of our collaboration includes the design and development of a unified experimental platform consisting of the ONIC and the interconnect fabric system test-bed. Investigations on various real-world performance metrics and design considerations will be conducted with continual interactions with Intel engineers to enable a cohesive partnership exploiting the resources and expertise of each group.
It is becoming increasingly clear that a clean-slate architectural design of the network protocol stack is an essential target for next-generation IP networks and network routing applications. A major challenge in realizing the required enhanced capabilities in next-generation networks lies in overcoming the current limitations due to the rigid notion of network layering. To address the research agenda of future networking infrastructures, our goal is to achieve a cross-layer communication infrastructure to provide bidirectional information exchange between layers. Enabling programmable interaction with the optical substrate will allow the higher layers dynamic access to the full optical bandwidth. Our goal is to create an intelligent, dynamic, programmable, network and application layer aware optical substrate, where data introspection and optical performance monitoring measurement data can be leveraged for cross-layer communications to impact network routing and performance.