New examples

.devcontainer/devcontainer.json

@@ -14,6 +14,6 @@
   },
   "onCreateCommand": "./.devcontainer/onCreate.sh",
   "remoteEnv": {
-    "PYTHONPATH": "${containerEnv:PYTHONPATH}:/workspaces/simbricks-examples/corundum"
+    "PYTHONPATH": "${containerEnv:PYTHONPATH}:/workspaces/simbricks-examples/corundum:/workspaces/simbricks-examples/prog-api-demo"
   }
 }

README.md

@@ -88,6 +88,27 @@ mybinder](https://mybinder.org/v2/gh/simbricks/labs/main?urlpath=git-pull%3Frepo
 Within the `netwroking-case-study` folder you will find SimBricks examples that demonstrates how to leverage SimBricks orchestration framework to create increasingly complex virtual prototypes of network topologies.
 It makes use of the concepts shown in the FirstSteps example and shows a simple setup in which multiple components of a virtual prototype are simulated by the same simulator. 
 
+## Corundum
+
+The Corundum examples showcases the integration of the open-source, high-performance FPGA-based Corundum NIC.
+
+## Data Center
+
+Example that showcases how SimBricks can be used to consisting of one spine switch and a number of racks.
+Each rack contains a TOR switch and a mix of abstractly simulated hosts for background traffic that are simulated in ns-3 and hosts
+simulated in QEMU that each are connected to a dedicated NIC simulator. The QEMU hosts run a full Linux stack and execute Iperf.
+The topology created looks as depicted [here](https://github.com/simbricks/simbricks-examples/blob/main/data-center/topology.png).
+
+## Programmatic API Demo
+
+SimBricks orchestration framework is written in Python. This makes it easy to write convenience functions yourself to share orchestration bits across experiments.
+Besides, SimBricks allows you to run your virtual prototype through python directly. This is useful in order to retrieve the simulation ourput line by line in python for parsing and testing.
+In the Api demo you can see simple examples for this.
+
+# Questions
+
+In case you have any questions regarding one of the examples, SimBricks in general or how you can use SimBricks in your workflow feel free to [reach out directly to us](https://www.simbricks.io/about.html) or open a issue in the [simbricks main repository](https://github.com/simbricks/simbricks/issues). 
+
 <!-- ## Example: Custom Images
 Any non-trivial project will typically require you to set up custom disk images
 for your simulated hosts to install necessary software dependencies and tools.

data-center/simple_data_center.py

@@ -0,0 +1,197 @@
+import ipaddress
+import random
+from simbricks.orchestration import system
+from simbricks.orchestration import simulation as sim
+from simbricks.orchestration.simulation.net import ns3_components
+from simbricks.orchestration import instantiation as inst
+from simbricks.orchestration.helpers import instantiation as inst_helpers
+from simbricks.orchestration.helpers import system as sys_helpers
+
+"""
+Simple Data Center Example:
+
+Simulate a topology consisting of one spine switch and N_RACKS many racks. Each rack contains a TOR
+switch and a mix of abstractly simulated hosts for background traffic (simulated in ns-3) and hosts
+(+ a separate NIC) running a full Linux stack and the desired workload (simulated in QEMU and
+dedicated NIC simulator).
+"""
+
+N_RACKS = 2
+N_DETAILED_HOSTS_PER_RACK = 1
+N_NS3_HOSTS_PER_RACK = 3
+
+if N_RACKS * N_DETAILED_HOSTS_PER_RACK % 2 != 0 or N_RACKS * N_NS3_HOSTS_PER_RACK % 2 != 0:
+    print("Number of detailed hosts and ns-3 hosts must each be even")
+    exit(1)
+
+LINK_LATENCY = 100000 # in nanoseconds
+LINK_BANDWIDTH_SPINE = "10Gbps"
+LINK_BANDWIDTH_TOR = "1Gbps"
+
+class Host:
+    def __init__(self, host: system.Host):
+        self.host: system.Host = host
+        self.nic: system.SimplePCIeNIC | None = None
+        self.channel: system.EthChannel | None = None
+        self.ip: str | None = None
+        self.ip_prefix: str | None = None
+
+instantiations: list[inst.Instantiation] = []
+
+# ============ SYSTEM ============
+
+sys = system.System()
+
+# IP network for assigning IP addresses to hosts
+ip_network = ipaddress.ip_network("10.0.0.0/16")
+ips = ip_network.hosts()
+ip_prefix = f"/{ip_network.prefixlen}"
+
+# Create spine switch
+spine_switch = system.EthSwitch(sys)
+spine_switch.name = "Spine Switch"
+
+# Create TOR switches
+tor_switches: list[tuple[system.EthSwitch, system.EthChannel]] = []
+for i_tor in range(N_RACKS):
+    tor_switch = system.EthSwitch(sys)
+    tor_switch.name = f"TOR Switch-{i_tor}"
+
+    channel = sys_helpers.connect_eth_devices(spine_switch, tor_switch)
+    channel.set_latency(LINK_LATENCY)
+    channel.parameters['data_rate'] = LINK_BANDWIDTH_SPINE
+    tor_switches.append((tor_switch, channel))
+
+# Create ns-3 hosts and connect them to TOR
+ns3_hosts: list[list[Host]] = []
+for i_tor_switch in range(N_RACKS):
+    rack_hosts = []
+    for i_host in range(N_NS3_HOSTS_PER_RACK):
+        sys_host = system.Host(sys)
+        sys_host.name = f"Ns-3 Host-{i_tor_switch}-{i_host}"
+        host = Host(sys_host)
+
+        channel = sys_helpers.connect_eth_devices(tor_switches[i_tor_switch][0], sys_host)
+        channel.set_latency(LINK_LATENCY)
+        channel.parameters['data_rate'] = LINK_BANDWIDTH_TOR
+        host.channel = channel
+
+        host.ip = str(next(ips))
+        host.ip_prefix = ip_prefix
+        sys_host.parameters["ip"] = host.ip + ip_prefix
+
+        rack_hosts.append(host)
+    ns3_hosts.append(rack_hosts)
+
+# Create disk images for hosts running Linux
+distro_disk_image = system.DistroDiskImage(sys, "base")
+
+# Create detailed hosts and connect them to TOR
+detailed_hosts: list[list[Host]] = []
+for i_tor_switch in range(N_RACKS):
+    rack_hosts = []
+    for i_host in range(N_DETAILED_HOSTS_PER_RACK):
+        # create a host instance and a NIC instance then install the NIC on the host
+        sys_host = system.I40ELinuxHost(sys)
+        sys_host.name = f"Detailed Host-{i_tor_switch}-{i_host}"
+        host = Host(sys_host)
+        sys_host.add_disk(distro_disk_image)
+        sys_host.add_disk(system.LinuxConfigDiskImage(sys, sys_host))
+
+        nic = system.IntelI40eNIC(sys)
+        sys_host.connect_pcie_dev(nic)
+        host.nic = nic
+
+        channel = tor_switches[i_tor_switch][0].connect_eth_peer_if(nic._eth_if)
+        channel.set_latency(LINK_LATENCY)
+        channel.parameters['data_rate'] = LINK_BANDWIDTH_TOR
+        host.channel = channel
+
+        host.ip = str(next(ips))
+        host.ip_prefix = ip_prefix
+        nic.add_ipv4(host.ip)
+
+        rack_hosts.append(host)
+    detailed_hosts.append(rack_hosts)
+
+# Set applications of ns-3 hosts for background traffic
+ns3_pairs = [i for i in range(N_RACKS * N_NS3_HOSTS_PER_RACK)]
+random.shuffle(ns3_pairs)
+for i in range(N_RACKS * N_NS3_HOSTS_PER_RACK // 2):
+    host0_rack_id = ns3_pairs[2*i] // N_NS3_HOSTS_PER_RACK
+    host0_host_id = ns3_pairs[2*i] % N_NS3_HOSTS_PER_RACK
+    host0 = ns3_hosts[host0_rack_id][host0_host_id]
+
+    packet_sink = system.Application(host0.host)
+    packet_sink.parameters["type_id"] = "ns3::PacketSink"
+    packet_sink.parameters["ns3_params"] = {
+        "Protocol": "ns3::TcpSocketFactory",
+        "Local(InetSocketAddress)": host0.ip + ":5001",
+    }
+    host0.host.add_app(packet_sink)
+
+    host1_rack_id = ns3_pairs[2*i+1] // N_NS3_HOSTS_PER_RACK
+    host1_host_id = ns3_pairs[2*i+1] % N_NS3_HOSTS_PER_RACK
+    host1 = ns3_hosts[host1_rack_id][host1_host_id]
+
+    on_off_app = system.Application(host1.host)
+    on_off_app.parameters["type_id"] = "ns3::OnOffApplication"
+    on_off_time = ns3_components.E2ENS3UniformRandomVariable()
+    on_off_time.min = 1
+    on_off_time.max = 5
+    on_off_app.parameters["ns3_params"] = {
+        "Protocol": "ns3::TcpSocketFactory",
+        "Remote(InetSocketAddress)": host0.ip + ":5001",
+        "DataRate": "100Mb/s",
+        "OnTime": on_off_time.get_config(),
+        "OffTime": on_off_time.get_config(),
+    }
+    host1.host.add_app(on_off_app)
+
+# Set applications for detailed hosts (Iperf clients and servers)
+detailed_pairs = [i for i in range(N_RACKS * N_DETAILED_HOSTS_PER_RACK)]
+random.shuffle(detailed_pairs)
+for i in range(N_RACKS * N_DETAILED_HOSTS_PER_RACK // 2):
+    host0_rack_id = detailed_pairs[2*i] // N_DETAILED_HOSTS_PER_RACK
+    host0_host_id = detailed_pairs[2*i] % N_DETAILED_HOSTS_PER_RACK
+    host0 = detailed_hosts[host0_rack_id][host0_host_id]
+    iperf_server_app = system.IperfTCPServer(host0.host)
+    host0.host.add_app(iperf_server_app)
+
+    host1_rack_id = detailed_pairs[2*i+1] // N_DETAILED_HOSTS_PER_RACK
+    host1_host_id = detailed_pairs[2*i+1] % N_DETAILED_HOSTS_PER_RACK
+    host1 = detailed_hosts[host1_rack_id][host1_host_id]
+    iperf_client_app = system.IperfTCPClient(host1.host, host0.ip)
+    iperf_client_app.wait = True
+    host1.host.add_app(iperf_client_app)
+
+# ============ SIMULATION ============
+
+simulation = sim.Simulation(name="simple-data-center", system=sys)
+
+# Simulate detailed hosts with QEMU and their NICs with an I40e simulator
+for i_tor_switch in range(N_RACKS):
+    for i_host in range(N_DETAILED_HOSTS_PER_RACK):
+        host = detailed_hosts[i_tor_switch][i_host]
+        host_inst = sim.QemuSim(simulation)
+        host_inst.add(host.host)
+        host_inst.name = f"Qemu-Host-{i_tor_switch}-{i_host}"
+
+        nic_inst = sim.I40eNicSim(simulation=simulation)
+        nic_inst.add(host.nic)
+
+# Simulate the rest of the system with ns-3
+net_inst = sim.NS3Net(simulation)
+net_inst.add(spine_switch)
+for i_tor_switch in range(N_RACKS):
+    net_inst.add(tor_switches[i_tor_switch][0])
+    for i_host in range(N_NS3_HOSTS_PER_RACK):
+        net_inst.add(ns3_hosts[i_tor_switch][i_host].host)
+
+#simulation.enable_synchronization()
+
+# ============ INSTANTIATION ============
+
+instantiation = inst_helpers.simple_instantiation(simulation)
+
+instantiations.append(instantiation)

prog-api-demo/helpers.py

@@ -0,0 +1,53 @@
+# Copyright 2021 Max Planck Institute for Software Systems, and
+# National University of Singapore
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+from simbricks.orchestration import system
+import re
+
+"""
+Helper functions that are used within the other files in this folder.
+"""
+
+# Custom defined helper function to create an 'I40ELinuxHost' attached to an 'IntelI40eNIC'.
+def sys_host_nic(sys, image, ip, hn=None, nn=None):
+    host = system.I40ELinuxHost(sys)
+    host.add_disk(image)
+    host.add_disk(system.LinuxConfigDiskImage(sys, host))
+    if hn:
+        host.name = hn
+
+    nic = system.IntelI40eNIC(sys)
+    nic.add_ipv4(ip)
+    host.connect_pcie_dev(nic)
+    if nn:
+        nic.name = nn
+
+    return host, nic
+
+
+def parse_Iperf_line_bytes(line: str) -> float | None:
+    pattern = r"(\d+\.?\d*)\s*MBytes"
+    match = re.search(pattern, line)
+    if not match:
+        return None
+    m_bytes = float(match.group(1))
+    return m_bytes

prog-api-demo/loop.py

@@ -0,0 +1,95 @@
+# Copyright 2021 Max Planck Institute for Software Systems, and
+# National University of Singapore
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+from simbricks.orchestration import simulation as sim
+from simbricks.orchestration import system
+from simbricks.orchestration.helpers import instantiation as inst_helpers
+from simbricks.orchestration.helpers import simulation as sim_helpers
+from helpers import sys_host_nic
+
+"""
+Simple example of a simulation: All components are executed in the same fragment.
+ _________________________________________________________________________
+|                                                                        |
+|  Iperf-Server -- Server-NIC -- Switch ----- Client-NIC -- Iperf-CLient |
+|________________________________________________________________________|
+
+We define different rates at which the Iperf UDP client tries to send packets 
+and create 3 instantiations (one for each rate).
+"""
+
+instantiations = []
+
+iperf_rates = ["150m", "430m", "600m"]
+for rate in iperf_rates:
+
+    """
+    System configuration
+    """
+    sys = system.System()
+
+    # We create a Linux disk image instance that will be used by the hosts we create.
+    # The image used here is provided by SimBricks, user can however also provide custom images.
+    distro_disk_image = system.DistroDiskImage(sys, "base")
+
+    # Configure the server to start an Iperf server by adding an application to the server object.
+    server_host, server_nic = sys_host_nic(
+        sys, distro_disk_image, "10.0.0.1", "Iperf-Server", "Server-NIC"
+    )
+    server_host.add_app(system.IperfUDPServer(server_host))
+
+    # Configure the client to start an Iperf client by adding an application to the client object.
+    # Besides, we set the wait flag on the application to tell SimBricks to run until this application is completed.
+    client_host, client_nic = sys_host_nic(
+        sys, distro_disk_image, "10.0.0.2", "Iperf-Client", "Client-NIC"
+    )
+    ping_client_app = system.IperfUDPClient(client_host, server_nic._ip, rate)
+    ping_client_app.wait = True
+    client_host.add_app(ping_client_app)
+
+    # Create a network switch that connects the server and client NICs with each other.
+    switch0 = system.EthSwitch(sys)
+    for nic in [server_nic, client_nic]:
+        switch0.connect_eth_peer_if(nic._eth_if)
+
+    """
+    Simulation configuration
+    """
+
+    # We make a simulator choice by simply mapping component types to simulators.
+    simulation = sim_helpers.simple_simulation(
+        sys,
+        compmap={
+            system.FullSystemHost: sim.QemuSim,
+            system.IntelI40eNIC: sim.I40eNicSim,
+            system.EthSwitch: sim.SwitchNet,
+        },
+    )
+    # simulation.enable_synchronization()
+
+    """
+    Instantiation configuration
+    """
+
+    # Instantiate the virtual prototype
+    instantiation = inst_helpers.simple_instantiation(simulation)
+    instantiations.append(instantiation)