Difference between revisions of "Interconnection Network"

From gem5
Jump to: navigation, search
m (Missing '\' after '--network=garnet2.0')
 
(12 intermediate revisions by one other user not shown)
Line 20: Line 20:
 
                       --num-cpus=16 \
 
                       --num-cpus=16 \
 
                       --num-dirs=16  \
 
                       --num-dirs=16  \
−
                       --network=garnet2.0
+
                       --network=garnet2.0 \
 
                       --topology=Mesh_XY \
 
                       --topology=Mesh_XY \
 
                       --mesh-rows=4  </pre>
 
                       --mesh-rows=4  </pre>
−
 
−
 
  
  
Line 31: Line 29:
  
 
The connection between the various controllers are specified via python files.
 
The connection between the various controllers are specified via python files.
 +
All external links (between the controllers and routers) are bi-directional. All internal links (between routers) are uni-directional -- this allows a per-direction weight on each link to bias routing decisions.
 +
 
* '''Related Files''':
 
* '''Related Files''':
 
**  '''src/mem/ruby/network/topologies/Crossbar.py'''
 
**  '''src/mem/ruby/network/topologies/Crossbar.py'''
Line 40: Line 40:
 
** '''src/mem/ruby/network/Network.py'''
 
** '''src/mem/ruby/network/Network.py'''
 
** '''src/mem/ruby/network/BasicLink.py'''
 
** '''src/mem/ruby/network/BasicLink.py'''
 +
** '''src/mem/ruby/network/BasicRouter.py'''
  
 
* '''Topology Descriptions''':
 
* '''Topology Descriptions''':
 
** '''Crossbar''': Each controller (L1/L2/Directory) is connected to a simple switch. Each switch is connected to a central switch (modeling the crossbar). This can be invoked from command line by '''--topology=Crossbar'''.
 
** '''Crossbar''': Each controller (L1/L2/Directory) is connected to a simple switch. Each switch is connected to a central switch (modeling the crossbar). This can be invoked from command line by '''--topology=Crossbar'''.
 
** '''CrossbarGarnet''': Each controller (L1/L2/Directory) is connected to every other controller via one garnet router (which internally models the crossbar and allocator). This can be invoked from command line by '''--topology=CrossbarGarnet'''.
 
** '''CrossbarGarnet''': Each controller (L1/L2/Directory) is connected to every other controller via one garnet router (which internally models the crossbar and allocator). This can be invoked from command line by '''--topology=CrossbarGarnet'''.
−
** '''Mesh_*''': This topology requires the number of directories to be equal to the number of cpus. The number of routers/switches is equal to the number of cpus in the system. Each router/switch is connected to one L1, one L2 (if present), and one Directory. It can be invoked from command line by '''--topology=Mesh_XY'''. The number of rows in the mesh '''has to be specified''' by '''--mesh-rows'''. This parameter enables the creation of non-symmetrical meshes too.
+
** '''Mesh_*''': This topology requires the number of directories to be equal to the number of cpus. The number of routers/switches is equal to the number of cpus in the system. Each router/switch is connected to one L1, one L2 (if present), and one Directory. The number of rows in the mesh '''has to be specified''' by '''--mesh-rows'''. This parameter enables the creation of non-symmetrical meshes too.
−
*** '''Mesh_XY''': Mesh with XY routing/
+
*** '''Mesh_XY''': Mesh with XY routing. All x-directional links are biased with  a weight of 1, while all y-directional links are biased with a weight of 2. This forces all messages to use X-links first, before using Y-links. It can be invoked from command line by '''--topology=Mesh_XY'''
−
*** '''Mesh_westfirst''': Mesh with west-first routing.
+
*** '''Mesh_westfirst''': Mesh with west-first routing. All west-directional links are biased with  a weight of 1, al other links are biased with a weight of 2. This forces all messages to use west-directional links first, before using other links. It can be invoked from command line by '''--topology=Mesh_westfirst'''
−
** '''MeshDirCorners''': This topology requires the number of directories to be equal to 4.  number of routers/switches is equal to the number of cpus in the system. Each router/switch is connected to one L1, one L2 (if present). Each corner router/switch is connected to one Directory. It can be invoked from command line by '''--topology=MeshDirCorners'''. The number of rows in the mesh '''has to be specified''' by '''--mesh-rows'''.
+
** '''MeshDirCorners_XY''': This topology requires the number of directories to be equal to 4.  number of routers/switches is equal to the number of cpus in the system. Each router/switch is connected to one L1, one L2 (if present). Each corner router/switch is connected to one Directory. It can be invoked from command line by '''--topology=MeshDirCorners_XY'''. The number of rows in the mesh '''has to be specified''' by '''--mesh-rows'''. The XY routing algorithm is used.
 
** '''Pt2Pt''': Each controller (L1/L2/Directory) is connected to every other controller via a direct link. This can be invoked from command line by '''--topology=Pt2Pt'''.
 
** '''Pt2Pt''': Each controller (L1/L2/Directory) is connected to every other controller via a direct link. This can be invoked from command line by '''--topology=Pt2Pt'''.
−
** '''Torus''': This topology requires the number of directories to be equal to the number of cpus. The number of routers/switches is equal to the number of cpus in the system. Each router/switch is connected to one L1, one L2 (if present), and one Directory. It can be invoked from command line by '''--topology=Torus'''. The number of rows in the Torus '''has to be specified''' by '''--mesh-rows'''. This parameter enables the creation of non-symmetrical tori too. By default, this file models a '''folded torus''' topology, where the length of each link (including the wrap-around ones) is the same, and approx. equal to twice that of a mesh. The default link latency is thus assumed to be 2-cycles.
 
  
−
http://www.gem5.org/graphics/ruby/figures/Topology_overview.jpg
+
<!-- http://www.gem5.org/graphics/ruby/figures/Topology_overview.jpg -->
 +
 
 +
http://pwp.gatech.edu/ece-synergy/wp-content/uploads/sites/332/2016/10/topologies.jpg
 +
 
 +
'''In each topology, each link and each router can independently be passed a parameter that overrides the defaults (in BasicLink.py and BasicRouter.py)''':
  
−
* '''Optional parameters specified by the topology files (defaults in Basic_Link.py)''':
+
* '''Link Parameters:'''
 
** '''latency''': latency of traversal within the link.
 
** '''latency''': latency of traversal within the link.
 
** '''weight''': weight associated with this link. This parameter is used by the routing table while deciding routes, as explained next in [[Interconnection_Network#Routing|Routing]].
 
** '''weight''': weight associated with this link. This parameter is used by the routing table while deciding routes, as explained next in [[Interconnection_Network#Routing|Routing]].
−
** '''bandwidth_factor''': For garnet networks, this is equal to the channel width in bytes. For simple networks, the bandwidth_factor translates to the bandwidth multiplier (simple/SimpleLink.cc) and the individual link bandwidth becomes bandwidth multiplier x endpoint_bandwidth (specified in SimpleNetwork.py).
+
** '''bandwidth_factor''': Only used by simple network to specify width of the link in bytes. This translates to a bandwidth multiplier (simple/SimpleLink.cc) and the individual link bandwidth becomes bandwidth multiplier x endpoint_bandwidth (specified in SimpleNetwork.py). In garnet, the bandwidth is specified by ni_flit_size in GarnetNetwork.py)
 +
 
 +
** '''Internal Link Parameters:'''
 +
** '''src_outport''': String with name for output port from source router.
 +
** '''dst_inport''': String with name for input port at destination router.
 +
These two parameters can be used by routers to implement custom routing algorithms in garnet2.0 (see [[Interconnection_Network#Routing|Routing]].
 +
 
 +
* '''Router Parameters:'''
 +
** '''latency''': latency of each router. Only supported by garnet2.0.
  
 
== Routing ==
 
== Routing ==
  
−
Based on the topology, shortest path graph traversals are used to populate ''routing tables'' at each router/switch. The default routing algorithm tries to choose the route with minimum number of link traversals. Links can be given weights in the topology files to model different routing algorithms. For example, in Mesh.py, MeshDirCorners.py and Torus.py, Y-direction links are given weights of 2, while X-direction links are given weights of 1, resulting in XY traversals. '''adaptive_routing''' (in src/mem/ruby/network/simple/SimpleNetwork.py) can be enabled to make the simple network choose routes based on occupancy of queues at each output port.
+
'''Table-based Routing (Default): '''
 +
Based on the topology, shortest path graph traversals are used to populate ''routing tables'' at each router/switch. This is done in src/mem/ruby/network/Topology.cc
 +
The default routing algorithm is table-based and tries to choose the route with minimum number of link traversals. Links can be given weights in the topology files to model different routing algorithms. For example, in Mesh_XY.py and MeshDirCorners_XY.py Y-direction links are given weights of 2, while X-direction links are given weights of 1, resulting in XY traversals.  
 +
In Mesh_westfirst.py, the west-links are given weights of 1, and all other links are given weights of 2. In garnet2.0, the routing algorithm randomly chooses between links with equal weights. In simple network, it statically chooses between links with equal weights.
 +
 
 +
'''Custom Routing algorithms: '''
 +
In garnet2.0, we provide additional support to implement custom (including adaptive) routing algorithms (See outportComputeXY() in src/mem/ruby/network/garnet2.0/RoutingUnit.cc).
 +
The src_outport and dst_inport fields of the links can be used to give custom names to each link (e.g., directions if a mesh), and these can be used inside garnet to implement any routing algorithm.
 +
A custom routing algorithm can be selected from the command line by setting --routing-algorithm=2. See configs/network/Network.py and src/mem/ruby/network/garnet2.0/GarnetNetwork.py
  
 
== Flow-Control and Router Microarchitecture ==
 
== Flow-Control and Router Microarchitecture ==
  
 
Ruby supports two network models, Simple and Garnet, which trade-off detailed modeling versus simulation speed respectively.
 
Ruby supports two network models, Simple and Garnet, which trade-off detailed modeling versus simulation speed respectively.
−
 
−
* '''Related Files''':
 
−
** '''src/mem/ruby/network/Network.py'''
 
−
** '''src/mem/ruby/network/simple'''
 
−
** '''src/mem/ruby/network/simple/SimpleNetwork.py'''
 
−
** '''src/mem/ruby/network/garnet/BaseGarnetNetwork.py'''
 
−
** '''src/mem/ruby/network/garnet/fixed-pipeline'''
 
−
** '''src/mem/ruby/network/garnet/GarnetNetwork_d.py'''
 
−
** '''src/mem/ruby/network/garnet/flexible-pipeline'''
 
−
** '''src/mem/ruby/network/garnet/flexible-pipeline/GarnetNetwork.py'''
 
−
 
−
=== Configuration and Setup ===
 
−
 
−
The default network model in Ruby is the [[Interconnection_Network#Simple_Network|simple]] network.
 
−
[[Interconnection_Network#Garnet_Networks|Garnet]] fixed-pipeline or flexible-pipeline networks can be enabled by adding '''--garnet-network=fixed''', or '''--garnet-network=flexible''' on the command line, respectively.
 
−
 
−
* '''Configuration''':
 
−
Some of the network parameters specified in Network.py are:
 
−
** '''number_of_virtual_networks''': This is the maximum number of virtual networks. The actual number of active virtual networks is determined by the protocol.
 
−
** '''control_msg_size''': The size of control messages in bytes. Default is 8. '''m_data_msg_size''' in Network.cc is set to the block size in bytes + control_msg_size.
 
−
 
  
 
=== Simple Network ===
 
=== Simple Network ===
Line 104: Line 103:
  
 
The interconnection networks can be run in a standalone manner and fed with synthetic traffic.
 
The interconnection networks can be run in a standalone manner and fed with synthetic traffic.
−
We recommend doing this with garnet2.0
+
We recommend doing this with garnet2.0.
  
−
'''[[Garnet_standalone | Running Garnet Standalone]]'''
+
'''[[Garnet_Synthetic_Traffic | Running Garnet Standalone with Synthetic Traffic]]'''

Latest revision as of 08:56, 7 November 2017

The various components of the interconnection network model inside gem5's ruby memory system are described here.

How to invoke the network

Simple Network: 

./build/ALPHA/gem5.debug \
                  configs/example/ruby_random_test.py \
                  --num-cpus=16  \
                  --num-dirs=16  \
                  --network=simple
                  --topology=Mesh_XY  \
                  --mesh-rows=4

The default network is simple, and the default topology is crossbar.


Garnet network: 

./build/ALPHA/gem5.debug \
                      configs/example/ruby_random_test.py  \
                      --num-cpus=16 \
                      --num-dirs=16  \
                      --network=garnet2.0 \
                      --topology=Mesh_XY \
                      --mesh-rows=4


Topology

The connection between the various controllers are specified via python files. All external links (between the controllers and routers) are bi-directional. All internal links (between routers) are uni-directional -- this allows a per-direction weight on each link to bias routing decisions.

  • Related Files:
    • src/mem/ruby/network/topologies/Crossbar.py
    • src/mem/ruby/network/topologies/CrossbarGarnet.py
    • src/mem/ruby/network/topologies/Mesh_XY.py
    • src/mem/ruby/network/topologies/Mesh_westfirst.py
    • src/mem/ruby/network/topologies/MeshDirCorners_XY.py
    • src/mem/ruby/network/topologies/Pt2Pt.py
    • src/mem/ruby/network/Network.py
    • src/mem/ruby/network/BasicLink.py
    • src/mem/ruby/network/BasicRouter.py
  • Topology Descriptions:
    • Crossbar: Each controller (L1/L2/Directory) is connected to a simple switch. Each switch is connected to a central switch (modeling the crossbar). This can be invoked from command line by --topology=Crossbar.
    • CrossbarGarnet: Each controller (L1/L2/Directory) is connected to every other controller via one garnet router (which internally models the crossbar and allocator). This can be invoked from command line by --topology=CrossbarGarnet.
    • Mesh_*: This topology requires the number of directories to be equal to the number of cpus. The number of routers/switches is equal to the number of cpus in the system. Each router/switch is connected to one L1, one L2 (if present), and one Directory. The number of rows in the mesh has to be specified by --mesh-rows. This parameter enables the creation of non-symmetrical meshes too.
      • Mesh_XY: Mesh with XY routing. All x-directional links are biased with a weight of 1, while all y-directional links are biased with a weight of 2. This forces all messages to use X-links first, before using Y-links. It can be invoked from command line by --topology=Mesh_XY
      • Mesh_westfirst: Mesh with west-first routing. All west-directional links are biased with a weight of 1, al other links are biased with a weight of 2. This forces all messages to use west-directional links first, before using other links. It can be invoked from command line by --topology=Mesh_westfirst
    • MeshDirCorners_XY: This topology requires the number of directories to be equal to 4. number of routers/switches is equal to the number of cpus in the system. Each router/switch is connected to one L1, one L2 (if present). Each corner router/switch is connected to one Directory. It can be invoked from command line by --topology=MeshDirCorners_XY. The number of rows in the mesh has to be specified by --mesh-rows. The XY routing algorithm is used.
    • Pt2Pt: Each controller (L1/L2/Directory) is connected to every other controller via a direct link. This can be invoked from command line by --topology=Pt2Pt.


topologies.jpg

In each topology, each link and each router can independently be passed a parameter that overrides the defaults (in BasicLink.py and BasicRouter.py):

  • Link Parameters:
    • latency: latency of traversal within the link.
    • weight: weight associated with this link. This parameter is used by the routing table while deciding routes, as explained next in Routing.
    • bandwidth_factor: Only used by simple network to specify width of the link in bytes. This translates to a bandwidth multiplier (simple/SimpleLink.cc) and the individual link bandwidth becomes bandwidth multiplier x endpoint_bandwidth (specified in SimpleNetwork.py). In garnet, the bandwidth is specified by ni_flit_size in GarnetNetwork.py)
    • Internal Link Parameters:
    • src_outport: String with name for output port from source router.
    • dst_inport: String with name for input port at destination router.

These two parameters can be used by routers to implement custom routing algorithms in garnet2.0 (see Routing.

  • Router Parameters:
    • latency: latency of each router. Only supported by garnet2.0.

Routing

Table-based Routing (Default): Based on the topology, shortest path graph traversals are used to populate routing tables at each router/switch. This is done in src/mem/ruby/network/Topology.cc The default routing algorithm is table-based and tries to choose the route with minimum number of link traversals. Links can be given weights in the topology files to model different routing algorithms. For example, in Mesh_XY.py and MeshDirCorners_XY.py Y-direction links are given weights of 2, while X-direction links are given weights of 1, resulting in XY traversals. In Mesh_westfirst.py, the west-links are given weights of 1, and all other links are given weights of 2. In garnet2.0, the routing algorithm randomly chooses between links with equal weights. In simple network, it statically chooses between links with equal weights.

Custom Routing algorithms: In garnet2.0, we provide additional support to implement custom (including adaptive) routing algorithms (See outportComputeXY() in src/mem/ruby/network/garnet2.0/RoutingUnit.cc). The src_outport and dst_inport fields of the links can be used to give custom names to each link (e.g., directions if a mesh), and these can be used inside garnet to implement any routing algorithm. A custom routing algorithm can be selected from the command line by setting --routing-algorithm=2. See configs/network/Network.py and src/mem/ruby/network/garnet2.0/GarnetNetwork.py

Flow-Control and Router Microarchitecture

Ruby supports two network models, Simple and Garnet, which trade-off detailed modeling versus simulation speed respectively.

Simple Network

Details of the Simple Network are here.

Garnet

Details of the original (2009) Garnet network are here. This design is no longer supported in the codebase.

Garnet2.0

Details of the new (2016) Garnet2.0 network are here.


Running the Network with Synthetic Traffic

The interconnection networks can be run in a standalone manner and fed with synthetic traffic. We recommend doing this with garnet2.0.

Running Garnet Standalone with Synthetic Traffic

Retrieved from "http://daystrom.m5sim.org/wiki/index.php?title=Interconnection_Network&oldid=8287"